Sending a webhook is one HTTP POST. Delivering a webhook is a distributed systems problem with retries, ordering, duplicate suppression, signature verification and a dead-letter path. Most implementations we review have built the first thing and named it the second, which works fine until the receiver has its first bad afternoon.
01Decide your guarantee and say it out loud
At-most-once, at-least-once and effectively-once are genuinely different products, and consumers build differently depending on which one you offer. Almost every practical webhook system is at-least-once, which means duplicates are not a bug you will eventually fix - they are a permanent property the receiver must handle.
Document it prominently. A receiver that assumes exactly-once will eventually double-process a payment, and the argument about whose fault that is goes better when the guarantee was written down before the incident rather than after.
| Guarantee | How it fails | Receiver must |
|---|---|---|
| At-most-once | Events silently lost on receiver error | Reconcile via API polling |
| At-least-once | Duplicates on retry | Deduplicate by event id |
| Effectively-once | Complex, expensive to build | Still deduplicate defensively |
02Retry schedules that respect a struggling receiver
Immediate retries hurt. A receiver returning 500 because its database is overloaded does not benefit from three more requests in the next second. Exponential backoff with jitter, spread over hours rather than minutes, gives the receiver room to recover and gives you a far better delivery rate.
Cap the total attempt window somewhere between twelve and seventy-two hours, then move the event to a dead-letter store the receiver can query and replay. Retrying forever fills your queues; giving up after five minutes loses events for any outage longer than a coffee break.
| Attempt | Delay after previous | Cumulative |
|---|---|---|
| 1 | immediate | 0 |
| 2 | 30s ± jitter | ~30s |
| 3 | 5m ± jitter | ~5m |
| 4 | 30m ± jitter | ~35m |
| 5 | 2h ± jitter | ~3h |
| 6 | 6h ± jitter | ~9h |
| 7 | 12h ± jitter | ~21h then dead-letter |
03Sign the payload, and verify it properly
Sign with an HMAC over the raw request body plus a timestamp, and send both in headers. The receiver recomputes the signature and rejects anything that does not match or is older than a few minutes, which stops both forgery and replay.
Two implementation details cause most of the bugs. The receiver must verify against the raw bytes, not a re-serialised object, because JSON key order and whitespace will differ. And the comparison must be constant-time, because a naive string equality check leaks the signature one byte at a time to anyone patient enough to measure.
04The receiver's side of the contract
A receiver should acknowledge fast and process asynchronously. Doing real work inside the webhook handler means your processing time counts against the sender's timeout, and a slow database write turns into a retry storm and a pile of duplicates.
Return 2xx as soon as the event is durably queued. Deduplicate on the event id, not on the payload contents. And be careful about ordering: most senders make no ordering guarantee at all, so a receiver that assumes 'created' always arrives before 'updated' will eventually process them backwards.
05Always give consumers a way to catch up
However good the delivery system is, a receiver will eventually be down longer than your retry window, or will process an event incorrectly and need to reprocess it. Without a replay path, their only recovery is to ask you to manually resend events, which becomes a support burden for you and an outage for them.
Two things solve this cheaply: an events endpoint they can page through by time range, and a replay control in your dashboard. Both turn a support ticket into something the consumer resolves themselves at three in the morning without needing you.
Topics
Priya Iyer
Staff Engineer · SyncTrix
Writes about the engineering decisions behind production systems - architecture, delivery and the trade-offs that only show up at scale.
Building something like this?
SyncTrix engineers AI, SaaS, platform and cloud systems for enterprises and high-growth teams. Tell us what you're shipping and we'll scope it with you.
Talk to an engineer