Webhook Delivery Architecture
A webhook is a reverse API call: instead of clients polling for changes, the provider POSTs an event to a customer-registered URL when something happens. The architecture problem is delivering to untrusted, unreliable endpoints you do not control — they go down, respond slowly, return errors, and may receive a message more than once.
The provider side is an at-least-once delivery system over HTTP to arbitrary receivers. Because the receiver's availability is out of your hands, naive synchronous "POST and hope" loses events whenever the endpoint is briefly down. The resilient design decouples event production from delivery via a durable queue. Source: compiled from webhook-delivery best practices
Provider-side architecture
- Durable queue — events are persisted, not delivered inline, so a slow/down receiver never drops them (log/queue).
- Delivery workers — POST with a strict timeout; treat non-2xx / timeout as failure.
- Retries with exponential backoff + jitter — re-attempt over an increasing window (minutes → hours), capped; see Resilience Patterns (Circuit Breaker, Bulkhead, Retry).
- Dead-letter + visibility — after max attempts, park the event and surface it (a dashboard + manual replay) so customers can recover.
- Signing — sign each payload (HMAC over body + timestamp) so receivers can verify authenticity and the timestamp guards against replay.
- Ordering — usually not guaranteed; include sequence numbers/timestamps so receivers can order or detect gaps.
Receiver-side contract
- Verify the signature on the raw body before parsing (parsing first breaks HMAC) — the core of Frontend and Design Skills.
- Respond fast (2xx), process async. Acknowledge immediately and enqueue the work; slow handlers cause provider retries and duplicates.
- Be idempotent. Retries mean the same event id arrives more than once; dedup on event id (see Idempotency and Exactly-Once Processing).
- Handle out-of-order and unknown event types gracefully.
Design implications
- At-least-once is the only honest contract — so signing + idempotency are mandatory on both sides, not optional.
- Decouple delivery from the event — inline delivery couples your system's health to the customer's endpoint.
- Replay is a feature — a manual/auto replay from the dead-letter queue turns a customer outage into a recoverable backlog.
Kevin's Frontend and Design Skills skill encodes the receiver half of this (raw-body verification, negative-lifecycle handling, idempotent side effects), and these deliveries flow through the edge the gateway exposes.
Architecture Position
| Axis | Value |
|---|---|
| Family | Distributed systems primitives |
| Boundary owned | Reverse API event delivery with signing, retries, dedupe, and observability. |
| Read with | Idempotency and Exactly-Once Processing, Saga Pattern and Distributed Transactions, Observability Architecture |
| Use this page when | delivering external events reliably |
Timeline
-
2026-07-01 | Architecture category refresh added this page to the Distributed systems primitives family, linked it to Architecture System Map, and kept it standalone because it owns this boundary: Reverse API event delivery with signing, retries, dedupe, and observability. Source: User request, 2026-07-01
-
2026-06-02 | Page created. Captured webhooks as delivery to untrusted endpoints, the provider-side durable-queue + worker + backoff + dead-letter + signing design, the receiver contract (verify raw body, ack-fast/process-async, idempotent), and the at-least-once implications. Source: compiled from webhook-delivery best practices (Stripe, Svix)