Integration partners report missing events. Your logs show the requests were sent. Both are true: sending an HTTP request inline when something happens and discarding the result means every timeout, every brief outage on their side and every deployment of yours loses events permanently, with no record that anything was lost.
01Never send webhooks inline
Dispatching a webhook synchronously within the request that triggered it couples your latency to the recipient's response time. A partner whose endpoint takes thirty seconds makes your API take thirty seconds, and a partner who is down can exhaust your connection pool.
Write the event to a queue and deliver from a separate worker. Your transaction commits, your response returns immediately, and delivery proceeds independently with its own retry behaviour. This one change resolves most reliability complaints.
| Inline | Queued | |
|---|---|---|
| Caller latency | Bound to recipient | Unaffected |
| Recipient outage | Event lost | Retried later |
| Your deployment | In-flight events lost | Resumed by workers |
| Retry capability | None practically | Full control |
| Delivery visibility | Log line at best | Queryable per event |
02Retry with backoff and a ceiling
Retrying immediately and repeatedly turns a recipient's brief problem into a sustained flood arriving exactly when they are least able to handle it. Exponential backoff with jitter spreads the load and gives them room to recover.
Continue for long enough to survive a realistic outage - retries spanning roughly twenty-four hours are a reasonable norm - then move the event to a failed state that the customer can inspect and replay. Give up too early and a routine maintenance window on their side loses a day of events.
03Sign payloads so recipients can verify them
A webhook endpoint is a publicly reachable URL accepting instructions about a customer's account. Without verification, anyone who learns the URL can send forged events. Sign the payload with a shared secret and include a timestamp so replays can be rejected.
Document the verification process precisely, including exactly which bytes are signed. Ambiguity here produces integrations that skip verification because they could not make it work, which is worse than not offering it.
| Property | What to tell integrators |
|---|---|
| Ordering | Not guaranteed - use timestamps |
| Duplicates | Possible - use the event id for idempotency |
| Retry schedule | Exact intervals and total duration |
| Timeout | How long you wait for a response |
| Success criteria | Which status codes count as delivered |
| Failure handling | When delivery stops and how to replay |
04At-least-once means duplicates happen
A recipient that processes an event and then fails to respond before your timeout will receive it again. This is unavoidable in any reliable delivery system, so include a stable event identifier and state clearly that consumers must be idempotent.
Ordering is similarly not guaranteed once retries exist - a retried earlier event can arrive after a later one. Include a timestamp and a sequence number where ordering matters so recipients can detect and handle out-of-order arrival themselves.
05Give customers visibility and control
The most common support burden is a customer asking whether an event was sent. A dashboard showing recent deliveries with status, response code and payload removes that entirely and lets them diagnose their own endpoint problems.
Add manual replay. When a customer's endpoint was misconfigured for a day, replaying that period is a self-service action rather than an engineering request, and it turns your most frequent support ticket into a button.
Topics
Aarav Patel
Principal Engineer · SyncTrix
Writes about the engineering decisions behind production systems - architecture, delivery and the trade-offs that only show up at scale.
Building something like this?
SyncTrix engineers AI, SaaS, platform and cloud systems for enterprises and high-growth teams. Tell us what you're shipping and we'll scope it with you.
Talk to an engineer