Idempotency, retries, and timeouts

01Idempotency, retry, timeout, and deadline: definitions

Idempotency means that repeating the same logical operation has the same intended business effect as performing it once. A retry is another attempt at that operation. A timeout is a maximum waiting duration: if no response arrives, the caller does not know whether the server received the request, committed it, or lost its reply. A deadline is an absolute point in time by which a call should finish. Propagating one deadline bounds the total waiting budget across a call chain; it does not prove that remote work stopped or undo committed effects.

Idempotency does not require every low-level network message to occur once. Without a stable operation identity, another attempt can accidentally create a second business intent. Our target is one order O17 for checkout attempt buy-204, even if three HTTP attempts arrive.

This distinction appears in payments, file uploads, job queues, webhooks, and agent tool calls. First identify the business operation and which transaction or external service durably records its result. Then decide how a retry finds that result.

02Lost-response retry: one order from two attempts

Bind an idempotency key, a stable identifier for one logical operation, to the authenticated caller and a normalized representation of the request. For example, POST /orders uses Idempotency-Key: buy-204, user U9, item B2, quantity 1, and quote Q8. The following trace isolates the failure between committing O17 and returning its response.

The request identity must come from a stable retryable intent, not a fresh random key on every network attempt. A distinct second purchase should use a new key. Reusing a key with a different item should be rejected rather than silently returning a result for the wrong request.

03Idempotency key, payload fingerprint, and atomic result storage

Store RequestResult(callerId, key, payloadHash, state, resourceId, response) with a unique (callerId, key) constraint. A payload hash is a fingerprint computed from the fields that define the operation, such as item, quantity, and quote. Canonical means these fields are normalized consistently before hashing, so equivalent inputs produce the same representation. It detects reuse of the same key for a different intent; it is not authorization.

When work cannot finish in one short database transaction, persist its progress and give a worker temporary ownership, often through a lease. Expiry lets a replacement take over, but recovery must still account for requests the previous worker may already have sent. This is why a long operation needs more states than simply “key absent” or “completed.”

Specify the key namespace: the group within which an idempotency key must be unique, such as all requests by one caller. The example uses caller-wide keys, so the fingerprint includes the operation and target as well as item fields. A tenant or service that uses separate namespaces must include that scope in the unique identity. Replaying a saved response still requires current permission; an old idempotency key must not expose a resource after access is revoked.

A lease lets a replacement worker take over after a deadline. The old worker may resume later, so the store must atomically check the current ownership version (epoch) and expected state before saving a result. That check cannot undo an external request already sent. The receiving service still needs duplicate protection, or a way to check and resolve the uncertain result.

04External effects and uncertain payment outcomes

Suppose checkout calls a payment provider after creating an order. The provider charges successfully, but its reply is lost before local state records success. Repeating a new provider request can double-charge even if the local order insert was idempotent.

Use one stable provider attempt key for the payment, record it durably before or as part of scheduling the attempt, and reconcile the provider's status after uncertainty. A webhook may report the result, but duplicate and reordered webhooks need their own identity/state checks. Only finalize the local purchase once the confirmed result satisfies the state machine.

If the provider has no safe retry or status lookup, an uncertain payment may need manual investigation. Explain that limit. To claim duplicate protection, identify the exact action protected, how long its request ID is remembered and which failures are covered. “Exactly once” alone explains none of those.

Read the provider's actual contract rather than copying a generic retry recipe. For example, Stripe documents replaying the first saved status and body for an idempotency key, including a saved 500. Reusing that key can therefore replay an error without proving that no effect occurred; using a fresh key simply to escape the saved error can duplicate work. Keep the operation pending and use the supported recovery path.

05End-to-end deadlines and retry amplification

A deadline is an absolute point in time by which a call should finish; a timeout is a maximum waiting duration, often for one step. If the user allows two seconds for checkout, giving three nested services independent two-second timeouts can exceed that budget. Propagate the deadline or its remaining time budget through the call chain and stop work that is no longer useful when safe to do so.

Retry transient transport failures or documented retryable responses when the operation is safe and time remains. Do not repeatedly retry invalid input, denied permission, or a business condition that is no longer satisfied, such as an expired reservation. Respect server retry guidance.

Budget connection setup, queueing, processing, backoff and response transfer within the same end-to-end limit. If Retry-After asks for a wait beyond the remaining interactive budget, return a retryable/pending result instead of sleeping and then starting an already-expired attempt. HTTP and RPC clients may have their own automatic retries, so inventory them before multiplying attempts. gRPC clients also need an explicit realistic deadline; deadline propagation and cancellation handling vary by language and application code.

06Exponential backoff, jitter, circuit breakers, and bulkheads

Bounding the number of retries still leaves two problems: many clients may retry together, and slow calls may occupy every available resource. The controls below address different parts of that load: when to retry, whether to call a failing dependency, and which workloads share a resource pool.

Exponential backoff increases the waiting interval between retries. For a base interval of 100 ms, caps might be 100, 200, 400, and 800 ms. Jitter randomizes each wait, for example choosing a value between zero and the current cap. Ten thousand clients then avoid retrying at exactly the same instant.

A circuit breaker stops calls temporarily after sufficient failure evidence and later allows limited probes. It reduces repeated futile work; it does not repair the dependency or authorize dropping important writes. A bulkhead gives workloads separate concurrency/resource pools so a slow image export cannot consume every checkout connection.

Queues also need limits. If work arrives faster than it can complete indefinitely, an ever-growing queue delays the failure while consuming memory or storage; it does not add processing capacity.

07Interview walkthrough: safe checkout retries

Interviewer: “The customer presses Buy twice because the first request timed out. How do you prevent two orders?”

Candidate: “Both attempts use buy-204 for the same user and request. I save the unique request-result record and order in one transaction. If the reply is lost after commit, the retry returns O17. For an external payment, I reuse the provider’s request key and check uncertain results. I stop retries at the user’s deadline and spread them with backoff and jitter so an outage does not trigger a flood.”

The answer is grounded because it names the durable state before and after the lost response, rather than assuming the network delivers exactly once.

Recall the key ideas

Answer from memory before opening each card. Explain why the choice works and what it costs. Revisit missed cards tomorrow.