Receiving webhooks looks like the easy half. You expose an endpoint, the provider posts to it, you update your database. Then a deploy causes a thirty-second gap, the provider retries into a duplicate, events arrive out of order, and your subscription states are wrong in ways that take a week to unpick.
01Acknowledge fast, process later
Doing real work inside the handler means your processing time counts against the sender's timeout. A slow database write or a downstream call turns into a timeout, a retry, and now two copies of the same event racing each other through your system.
Validate the signature, write the raw event to a queue or table, return 200. Process asynchronously. This one change eliminates the majority of duplicate-processing problems, because the acknowledgement no longer depends on how long your business logic takes.
| Step | Inside the handler | Why |
|---|---|---|
| Verify signature | Yes | Reject forgeries before anything else |
| Check timestamp freshness | Yes | Reject replayed old events |
| Deduplicate on event id | Yes | Cheap check, avoids queueing twice |
| Persist raw payload | Yes | Durable before acknowledging |
| Return 200 | Yes | Sender stops retrying |
| Business logic | No, async | Slow work must not affect the ack |
02Deduplicate on the event id, not the contents
Every sender worth using includes a unique event identifier. Store the ones you have processed and skip repeats. Do not deduplicate on payload contents, because two genuinely distinct events can be byte-identical - two identical charges a second apart are two charges.
Keep the identifier record for longer than the sender's maximum retry window, and be generous. A sender that retries for three days against a receiver that remembers for one hour is a duplicate-processing arrangement with extra steps.
03Ordering is not guaranteed and you will feel it
Most providers make no ordering guarantee. A subscription updated event can arrive before the created event, and a cancellation can arrive before the update it supersedes. Processing them in arrival order produces a state that is wrong and looks correct.
Handle it by comparing a version or timestamp on the object and ignoring events older than the state you already hold. Where the provider offers no such field, fetching the current state from their API on receipt is the reliable fallback - treat the webhook as a notification that something changed rather than as the change itself.
| Strategy | Requires | Trade-off |
|---|---|---|
| Version comparison | Version field on the object | Simple and reliable when available |
| Timestamp comparison | Event timestamp | Clock skew between sender systems |
| Fetch current state on receipt | An API to call | Extra request, always correct |
| Process in arrival order | Nothing | Silently wrong state |
04Plan for the gap you will eventually have
Your endpoint will be down or broken at some point - a deploy, a certificate expiry, a bug in the handler that returns 500 for a particular event shape. The sender will retry for its window and then give up, and you will be missing events with no notification.
Reconcile periodically. Poll the provider's API for objects changed since your last known good timestamp, and compare against what you processed. Doing this nightly turns a silent data-divergence problem into a self-healing one.
05Make failures visible
Events that fail processing must land somewhere visible with the payload and the error, not disappear into a log. A dead-letter table with a count on a dashboard is enough, and it means a handler bug becomes a number someone notices rather than a discrepancy a customer reports.
Alert on the rate rather than on individual failures. One failed event is often a genuinely bad payload; a rising rate is a bug you shipped, and the difference in urgency is significant.
Topics
Priya Iyer
Staff Engineer · SyncTrix
Writes about the engineering decisions behind production systems - architecture, delivery and the trade-offs that only show up at scale.
Building something like this?
SyncTrix engineers AI, SaaS, platform and cloud systems for enterprises and high-growth teams. Tell us what you're shipping and we'll scope it with you.
Talk to an engineer