API Idempotency Retry

API Idempotency and Retry

You design the two halves of one guarantee: server-side idempotency keys that make a retried write safe, and client-side retry behavior that uses them responsibly. Shipping either half alone is the trap - Resilience4j's own docs state it as a hard constraint, not a suggestion: retried operations must be idempotent, so retry guidance without key support (or the reverse) leaves the guarantee open exactly where a dropped connection tests it.

Two facts frame everything below:

• Exactly-once delivery is impossible over an unreliable network (Tyler Treat's canonical argument, grounded in the FLP result). What a key actually buys is _effectively-once_: at-least-once delivery plus duplicate collapse, bounded to the scope where idempotency is actually implemented.
• There is no ratified standard behind the Idempotency-Key header. The IETF draft (draft-ietf-httpapi-idempotency-key-header) expired in April 2026 without ratification. The header name is consistent across the industry because Stripe popularized it, but the semantics (TTL, mismatch behavior, concurrency handling) are whatever each provider decided, so yours must be designed and documented, never assumed.

Memory (advised): When memory lives in a file, consider using `developer-platform-context.md`; if a different memory system is in use, rely on that instead. The file is an advisory reference, not a mandatory requirement. Separate task info in different sections. Remove finished tasks. Add a date to a task; no date for general project context. Some interview responses may differ between 2 tasks.

Clarifying questions

Ask these before designing anything; each answer changes a later step. Batch them - this is a tactical design task, not a strategy interview.

1. Paradigm: REST, gRPC, or both? (gRPC carries deadline propagation natively; REST needs the header convention and explicit deadline plumbing)
2. Which endpoints are non-idempotent? Request the endpoint list or OpenAPI spec and enumerate every state-changing operation.
3. What storage backs the key store - a relational DB with unique constraints and transactions, a conditional-write KV store, or only a cache? (rules atomic-claim options in or out - see step 4)
4. Do any requests call an external, non-rollbackable system mid-flight (a charge, an outbound email)? (drives recovery points - see step 5)
5. Which SDK languages do you ship, and do integrators also call raw HTTP? (drives how much of the retry mechanics you can default for them - see step 6)
6. What is the longest possible retry chain - dead-letter-queue replay windows, dispute windows, scheduled re-runs? (drives TTL - see step 3)
7. By when must the guarantee hold: a one-off fix (stop the duplicate charges one client is filing tickets about), or a compounding contract every future endpoint inherits? What is the effort ceiling on the claim mechanism - a ship date that rules out a schema or isolation-level migration, or existing fluency with SERIALIZABLE workloads? (re-ranks the claim menu - see step 4: a hard date promotes whatever the current store already supports, a compounding mandate promotes the richer claim lifecycle, and a ceiling that forbids a schema migration deletes the rungs it cannot reach rather than ranking them last)

Who controls the retry loop

Every retry decision is made by a developer or a program, whoever bought the product. The design splits by who controls the retry loop:

• First-party developers - you control both ends; internal convention plus code review covers most of it. The cheapest audience to serve.
• Third-party integrators calling raw HTTP - they write their own retry loop from your docs alone. They need the full contract published: key semantics, TTL, mismatch and in-flight behavior, and a worked backoff example, because no standard fills the gaps for them.
• SDK-mediated callers - your SDK _is_ their retry loop. Its defaults (backoff, jitter, attempt caps, key auto-generation) are the contract most callers will never read past, so wrong defaults ship duplicates at fleet scale.

Design for raw-HTTP integrators first - a contract complete enough for them is what your own SDKs then automate - and treat SDK defaults as the highest-leverage artifact, since they decide behavior for the majority who never read the docs.

Workflow

1. Identify retry-unsafe endpoints.
2. Design the key contract.
3. Set storage and TTL.
4. Claim the key atomically.
5. Design the response semantics.
6. Write client retry guidance and SDK defaults.
7. Document the contract for integrators.

Each step has a section below, in order.

1. Identify retry-unsafe endpoints

• RFC 9110 - the one ratified document relevant here - defines GET, PUT, and DELETE as idempotent methods and POST as not. It says nothing about the Idempotency-Key retrofit pattern.
• Enumerate every state-changing endpoint. Each must either accept an idempotency key or be explicitly documented as unsafe to retry - the review-level check samber/developer-platform-skills@public-api-design-review performs; this skill builds what makes the first option true.
• Verify claimed idempotency, don't trust the method: a PUT that appends to a log or a DELETE that decrements a counter is not idempotent whatever the method promises.

2. Design the key contract

• The client mints the key, the server never derives it. AWS's Builders' Library names the rejected alternative: a server-side hash of request parameters breaks when two genuinely identical requests are both intended (launching two identical instances on purpose) - a caller-provided key expresses intent an inferred one cannot. Stripe, AWS, Twilio, and Resend all put the client in charge.
• Scope keys per caller - (account_id, idempotency_key) - never globally. Stripe, AWS, and the standard Postgres reference implementation all key on the pair, so the same key string reused by different accounts never collides.
• Recommend a derivation to clients. Three work; rank them for the client rather than listing them (Resend's own recommendation order):
• value: event-derived > request-scoped > random UUID reused across retries
• effort: event-derived > request-scoped > random UUID reused across retries
• efficiency: event-derived > request-scoped > random UUID reused across retries - value and effort rank identically here, so their spread decides the ratio: minutes wide on effort (name the key after an identifier the caller already holds) against the whole guarantee on value.
• Default rung: event-derived (order-confirm-${orderId}) - deterministic by construction, so a retry cannot fail to reuse the key. Move down to request-scoped when no stable business identifier exists at call time, and to a reused random UUID when the caller is a stateless script. Nothing is starved here: every rung costs one line of client code.
• Never rank a fourth rung below these: a fresh value per attempt (Date.now(), randomUUID() regenerated on retry) defeats the mechanism entirely, so document it as the anti-pattern instead.
• Guard the payload: the same key with a different request body is a client bug - reject it (409 is the field convention; Resend names it 409 invalid_idempotent_request), never silently replay. The expired IETF draft generalizes this into an optional payload _fingerprint_ stored beside the key; a checksum of the payload is the simplest form.
• Store both outcomes. Stripe saves failures as well as successes - including 500s - so a retry replays the exact original result. Decide explicitly whether a stored failure replays verbatim (Stripe v1) or the operation may be re-attempted without side effects (Stripe v2); leaving it undecided is a design gap.

3. Set storage and TTL

• Retention is two separate concerns, not one TTL (Brandur Leach's reference implementation):