A customer presses “Pay”. The request reaches the server, the payment succeeds and the response disappears with the train's network connection. The button stops spinning and shows an error.
What should happen when they press it again?
If the answer is “we hope the first request failed”, the interface is exposing a distributed-systems ambiguity directly to the customer.
Give the intent an identity
The client creates an idempotency key before the first attempt and keeps it for every retry of the same business intent:
POST /payments
Idempotency-Key: pay_01JABC…
Content-Type: application/json
{
"orderId": "ord_01JXYZ…",
"amountMinor": 4200,
"currency": "GBP"
}The key is not a random request ID generated by the network layer on every attempt. It identifies “pay £42 for this order” across those attempts and is scoped to the authenticated actor and operation.
Amazon's Builders' Library defines the property clearly: “a request can be retransmitted or retried with no additional side effects”.
Reserve the key with the operation
On the first valid request, persist the key and a fingerprint of the material parameters in durable storage. The operation moves through explicit states:
not_seen → in_progress → succeeded
↘ failed_terminalA retry behaves according to state:
- In progress: return a response that lets the client wait or poll.
- Succeeded: return the original successful result.
- Terminal failure: return the same failure where the contract requires it.
- Same key, different parameters: reject as a conflict.
Stripe documents a concrete version of this contract: it stores the status code and body from the first request for a key, then returns the same result to later requests. It also compares parameters to prevent accidental reuse.
Validation failures usually should not consume the key because the operation never began. The exact boundary must be documented so clients know which responses are safe to retry.
The database row is not the whole effect
Suppose the payment record is inserted once, but the application publishes two payment_succeeded messages after a retry. The customer receives two receipts and fulfilment runs twice. The API endpoint was idempotent; the product was not.
Carry identity into downstream work:
- write the business change and an outbox record in one transaction;
- give emitted events stable IDs;
- make consumers record processed IDs at their own boundary;
- pass idempotency keys to external providers that support them;
- design compensating action for providers that do not.
Exactly-once delivery is rarely the transport guarantee. An effectively-once business outcome is built from durable identity and deduplication at each side effect.
Treat unknown as a first-class response
After a timeout, the client should not immediately invite a fresh payment. Keep the idempotency key, show “Checking payment status” and query the operation by its stable identifier.
Support needs the same identifier. A colleague should be able to answer:
- Did the server receive the intent?
- Did execution begin?
- What result was stored?
- Which downstream effects were emitted?
- Is recovery still running?
“Unknown” is a temporary system state to resolve, not a reason to guess success or failure in copy.
Retain keys for the real retry window
Set retention around offline clients, delayed queues and support investigation, then document it. Expiring too early reopens duplication risk; retaining forever creates unbounded storage and may preserve identifiers longer than necessary.
Keys need enough entropy to resist guessing and must never permit access across accounts. Reusing an old key after expiry should be treated as a new operation only under a clear contract.
This matters wherever one action affects money, inventory or a message outside the system. Our backend engineering and regulated product experience such as ClearBank put the same question at the boundary: how does one user intent remain one outcome when every network call can be repeated?
When idempotency is designed well, users never notice it. They press once, reconnect and see the truth. Invisible reliability is still a product feature.
Sources
Filed under
- Idempotency
- API design
- Distributed systems
- Payments