The failure that costs real money is not an error response. It is a timeout, where the request may have succeeded, may have failed, and your client has no way to tell. Retry it and you might charge twice. Do not retry it and you might drop a payment the customer believes they made. Idempotency keys are how you make that choice safely, and they are absent from most integrations we are asked to review.
01A timeout is not a failure
When a call times out, three things could have happened: the request never arrived, it arrived and failed, or it arrived and succeeded but the response was lost. Only the third case is dangerous, and from the client's position all three look identical. Treating a timeout as a failure and retrying is exactly what causes duplicate charges, duplicate orders and duplicate emails.
This is not an edge case that appears once a year. It happens every time a load balancer recycles a connection, a deployment rolls, or a vendor has a slow minute. In a system doing thousands of transactions a day, timeouts are routine, which means unsafe retries are routine too.
| Outcome | Server state | Safe to retry? | Typical handling |
|---|---|---|---|
| 200 OK | Applied | No need | Record and move on |
| 4xx | Not applied | No - fix the request | Surface to caller |
| 5xx | Unknown | Only with idempotency key | Retry with backoff |
| Timeout | Unknown | Only with idempotency key | Retry with same key |
02The key belongs to the operation, not the request
The common mistake is generating a fresh key on each attempt, which defeats the entire mechanism. The key identifies the business operation - this specific payment for this specific order - and every retry of that operation carries the same key. Generate it once, at the point the user acts, and persist it alongside the work you are about to do.
Choosing what the key represents matters more than the format. A UUID per checkout attempt is usually right. Hashing the request body is tempting but wrong: two genuinely separate payments of the same amount to the same merchant would collide and the second would be silently swallowed.
03Server side: store the result, not just the key
A correct implementation records the key, the request fingerprint and the response. On a repeat, it returns the stored response rather than re-executing. Storing only the key and returning a bare 'already processed' forces the client into a second lookup to find out what actually happened, which is another call that can time out.
Guard the record with a unique constraint and insert it in the same transaction as the work. Checking for an existing key and then inserting is a race: two concurrent retries both check, both find nothing, and both proceed. The database constraint is what makes it correct, not the application logic around it.
| Column | Purpose | Notes |
|---|---|---|
| key | Client-supplied identifier | Unique constraint, indexed |
| request_hash | Detect key reuse with different body | Return 422 on mismatch |
| status | in_progress / completed | Lets concurrent retries wait or fail fast |
| response_body | Stored result | Returned verbatim on replay |
| expires_at | Retention window | 24-72h is typical; longer for payments |
04Expiry, and the trap inside it
Keys cannot live forever, so most implementations expire them after a day or two. That creates a subtle hazard: a client that retries after the window has closed gets a fresh execution and a duplicate. Rare, but it is precisely the scenario that occurs when a queue backs up and drains hours later.
Match the retention window to the maximum age of a retry your system can produce, then make it longer. If a job can sit in a dead-letter queue for a week before someone replays it, a 24-hour window is not protecting you. For payments we default to 30 days and treat storage as the cheaper side of the trade.
05What to check in your own integrations
Look for retry logic without an accompanying key - that is the combination that generates duplicates. Look for keys generated inside the retry loop. Look for check-then-insert without a unique constraint. Those three patterns cover the overwhelming majority of duplicate-charge incidents we have been called in to diagnose.
Then test it properly. Fire the same key concurrently from several threads and assert exactly one execution. Kill the process between the write and the response and assert the retry returns the stored result. These are cheap tests that fail loudly on a broken implementation, and almost nobody writes them until after the first incident.
Topics
Marcus Hale
Lead Architect · SyncTrix
Writes about the engineering decisions behind production systems - architecture, delivery and the trade-offs that only show up at scale.
Building something like this?
SyncTrix engineers AI, SaaS, platform and cloud systems for enterprises and high-growth teams. Tell us what you're shipping and we'll scope it with you.
Talk to an engineer