Idempotency Keys: Making Retries Safe

Understand why naive request retries can duplicate side effects like payments or order creation, and implement idempotency keys in PHP so a repeated request with the same key returns the original result instead of re-executing it.

Networks are unreliable in a very specific, very annoying way: a client can send a request, the server can fully process it and even commit the result to a database, and then the response can get lost on the way back — a proxy times out, a mobile connection drops, a load balancer restarts mid-flight. From the client's point of view this looks identical to a request that never reached the server at all, so the client does the only reasonable thing it knows how to do: it retries. If that original request was 'create an order' or 'charge this card', the retry doesn't just resend a question, it repeats an action that already happened. Idempotency keys are the standard fix, and once you've been burned by a double-charged customer or a duplicate shipment, you stop treating them as optional.

The Duplicate-Request Problem

Consider a checkout endpoint: POST /orders with a cart payload. The client sends the request, the server validates the cart, charges the customer's card through a payment provider, inserts an order row, and starts building the response. Right as the server finishes and tries to send the 200 response back, the client's connection drops — maybe the user's phone lost signal in an elevator, maybe a corporate proxy enforced an aggressive timeout. The client never receives confirmation. All it knows is that the request is now in an ambiguous state: did it fail before reaching the server, did it fail during processing, or did it succeed and only the response got lost?

A reasonable client — especially one built with retry logic for reliability, which is good practice in general — resends the exact same POST. If the server has no way to recognize this as 'the same logical operation as before', it will happily charge the card a second time and create a second order row. The bug here isn't in any single line of code; it's a structural gap between 'at-least-once delivery' (which is what most retry-capable HTTP clients actually provide) and 'exactly-once execution' (which is what a payment charge needs). This gap exists at every layer of distributed systems — message queues, webhooks, mobile apps with spotty connectivity — and PHP web APIs sit squarely in the blast radius.

It's worth being precise about which operations are dangerous here. A GET request is naturally idempotent — reading the same resource twice causes no harm, so retries are always safe. A PUT that fully replaces a resource is also naturally idempotent in the HTTP sense — sending it twice with the same body produces the same end state. The danger is concentrated in POST requests that create something new or trigger an external side effect with each execution: creating an order, sending an email, charging a card, incrementing a counter, sending an SMS. These operations are 'non-idempotent by nature' — calling the underlying logic twice produces two of whatever it does, and no amount of careful HTTP-verb choice fixes that on its own.

• At-least-once delivery (what real networks give you) does not imply exactly-once execution (what a payment/order needs). • A lost response looks identical, to the client, to a request that never arrived — so a well-behaved client retries. • GET and full-replacement PUT are naturally idempotent; POST that creates/charges/emails is not. • The risk is concentrated exactly where the business impact is highest: payments, orders, notifications.

Idempotency Keys as the Solution

An idempotency key is a unique identifier, generated by the client, that represents one specific logical operation — 'this particular checkout attempt', not 'the checkout endpoint in general'. The client generates it once (typically a UUID) before the first attempt and sends it in every retry of that same logical operation, usually as a header like Idempotency-Key: 6b1f8f1e-.... The server's job is to remember which keys it has already processed and what result it produced, so that a repeated key returns the stored result instead of running the operation again.

The server-side mechanism is straightforward in shape even though the details matter a lot: before executing the operation, look up the incoming key in a store (a database table or a fast cache like Redis). If it's not there, insert a row marking the key as 'in progress', execute the actual business logic, then update that row with the final result (success payload, or a recorded failure) and return it. If the key IS already there, skip execution entirely and return whatever was stored — the same order ID, the same charge confirmation — as if the operation had just now succeeded, even though it ran once, earlier.

The single trickiest part is the race between 'check' and 'store': if two requests with the same key arrive close together (a client's original request and its own retry, overlapping because the response was merely slow rather than lost), a naive check-then-insert has a race window where both could pass the check before either has stored a result, letting both execute. The fix is to make the initial insert atomic and use it as the lock: attempt an INSERT of the key with an unique constraint on the key column, and treat a constraint-violation error as 'someone else already claimed this key' — the second request then waits for (or polls for) the first one's result rather than re-executing.

• Client generates one key per logical operation attempt, not one key per HTTP call — the retry reuses the same key. • Server stores key → result the first time; a repeated key returns the stored result without re-executing. • A unique constraint on the key column, and treating its violation as 'already claimed', closes the check-then-insert race. • Keys should expire eventually (hours to days) — they exist to cover a retry window, not forever.

Why This Matters Most for Non-Idempotent-by-Nature Operations

It's tempting to add idempotency keys everywhere 'just in case', but the cost (extra storage, extra lookups, extra complexity in the request path) is only clearly worth it where a duplicate execution causes real harm. Charging a card twice means a refund process, a support ticket, and a customer who no longer trusts your checkout flow. Creating a duplicate order means split shipments, inventory discrepancies, and confused fulfillment staff. Sending a duplicate 'your account was created' email is merely annoying; sending a duplicate 'you were charged $499' email is alarming even if the charge itself was deduplicated properly. The severity of the outcome, not the mere fact that an operation is a POST, should drive where you invest in this pattern.

This is exactly why major payment APIs (Stripe being the most widely copied example) make idempotency keys a first-class, documented part of their API surface rather than an internal implementation detail — they know their integrators will have retry logic somewhere in the stack (a load balancer, an SDK, a queue consumer) and they'd rather give clients an explicit, supported tool for safe retries than have every integrator invent their own ad hoc deduplication, badly, under pressure after an incident.

• Prioritize idempotency keys where a duplicate execution has real cost: payments, order creation, external notifications with financial implications. • Following well-known API design (Stripe, PayPal) — accepting a client-supplied Idempotency-Key header — is a proven, integrator-friendly pattern. • Don't idempotency-key everything reflexively; weigh the storage/complexity cost against the actual blast radius of a duplicate.

Retry-Safe Design in General: Prefer Naturally Idempotent Operations

Idempotency keys are a general-purpose patch for operations that are inherently unsafe to repeat. A complementary, often cheaper strategy is to redesign the operation itself so repeating it is harmless, which avoids needing key-tracking infrastructure at all for that particular case. The classic example is INSERT versus UPSERT: a plain INSERT INTO orders (external_id, ...) VALUES (...) executed twice with the same external_id creates two rows (or fails on a unique constraint, which is at least safe, just ungraceful). An INSERT ... ON DUPLICATE KEY UPDATE (MySQL) or INSERT ... ON CONFLICT (...) DO UPDATE (Postgres) executed twice with the same natural key converges to exactly one row either way — the second execution is a no-op update rather than a duplicate insert.

Similarly, 'set the user's status to active' is naturally idempotent (setting it to active a second time changes nothing), whereas 'increment the user's login count by one' is not (running it twice adds two, not one) — the fix there might be recording the specific login event with its own unique ID and deriving the count from a count of distinct events, rather than blindly incrementing a counter on every call. The general principle: wherever the business logic allows it, design the write to be a convergent, replayable operation (set X to Y, ensure record with key K exists with these fields) rather than a cumulative one (add N to X, insert a new row unconditionally) — it eliminates an entire category of duplicate-execution bugs without needing any key-tracking machinery.

• UPSERT (INSERT ... ON CONFLICT DO UPDATE / ON DUPLICATE KEY UPDATE) converges to one row no matter how many times it runs with the same key. • 'Set to a value' operations are naturally idempotent; 'increment/add' operations are not. • Prefer recording discrete, uniquely-identified events over blindly incrementing counters, when duplicates are a risk. • Idempotency keys and naturally-idempotent design are complementary, not exclusive — use both where it makes sense.

Code Examples

The unique-constraint INSERT is what makes this safe under real concurrency: only one request can ever successfully claim a given key, so a genuinely simultaneous retry is forced down the 'someone already owns this' path instead of racing past a plain SELECT-then-INSERT check. The result is stored once and replayed verbatim on every subsequent request carrying the same key.

INSERT INTO idempotency_keys (...) VALUES (...) — the UNIQUE constraint is the actual concurrency control, not application-level logic.

$existing = $pdo->query("SELECT * FROM idempotency_keys WHERE idempotency_key = '$key'")->fetch(); if (!$existing) { /* insert then process */ }

Attempt an INSERT with a UNIQUE constraint on idempotency_key first, and treat a duplicate-key error as 'already claimed'.

A SELECT-then-INSERT check has a race window: two near-simultaneous requests with the same key can both pass the SELECT before either INSERTs, letting both execute the non-idempotent work. Only an atomic, constraint-backed insert closes that window.

Why can't a client reliably tell the difference between 'my request never reached the server' and 'the server processed it but the response was lost'? A. It can always tell, because HTTP status codes disambiguate this B. Both situations look identical from the client's side: no response arrives either way C. PHP automatically retries lost responses internally D. This only happens with GET requests, never POST

B. Both situations look identical from the client's side: no response arrives either way — From the client's perspective, a dropped connection before the server even receives the request and a dropped connection after the server fully processed it and started responding are indistinguishable — no response arrives in either case, which is exactly why naive retries are dangerous for non-idempotent operations.

What role does the UNIQUE constraint on the idempotency key column play? A. It's purely for query performance, unrelated to correctness B. It provides the atomic 'claim' operation that prevents two concurrent requests with the same key from both executing the underlying operation C. It automatically expires old keys D. It encrypts the key for security

B. It provides the atomic 'claim' operation that prevents two concurrent requests with the same key from both executing the underlying operation — A plain SELECT-then-INSERT check has a race window under real concurrency; relying on the database's UNIQUE constraint to reject a duplicate INSERT is what actually guarantees only one request can claim a given key and proceed to execute the operation.

Which of these is the best example of redesigning an operation to be naturally idempotent, rather than relying on an idempotency key? A. Using INSERT ... ON CONFLICT DO UPDATE (UPSERT) instead of a plain INSERT keyed on a natural unique identifier B. Adding more server RAM C. Removing the UNIQUE constraint from the orders table D. Increasing the client's request timeout

A. Using INSERT ... ON CONFLICT DO UPDATE (UPSERT) instead of a plain INSERT keyed on a natural unique identifier — An UPSERT converges to the same end state (one row) no matter how many times it is executed with the same key, eliminating the duplicate-row problem at the operation-design level instead of patching it with key-tracking infrastructure.

<?php
declare(strict_types=1);

function handleCheckout(PDO $pdo, string $idempotencyKey, array $cartPayload): array
{
    // 1. Try to atomically claim this key. The UNIQUE constraint on
    //    idempotency_key is what actually prevents a race between
    //    two near-simultaneous requests carrying the same key.
    try {
        $pdo->prepare(
            'INSERT INTO idempotency_keys (idempotency_key, status, created_at)
             VALUES (:key, :status, NOW())'
        )->execute(['key' => $idempotencyKey, 'status' => 'in_progress']);
    } catch (PDOException $e) {
        // 23000 = integrity constraint violation (duplicate key)
        if ($e->getCode() === '23000') {
            return waitForOrFetchResult($pdo, $idempotencyKey);
        }
        throw $e;
    }

    // 2. We own this key — do the actual, non-idempotent work exactly once.
    try {
        $order = createOrderAndChargeCard($pdo, $cartPayload);

        $pdo->prepare(
            'UPDATE idempotency_keys
             SET status = :status, response_body = :body, completed_at = NOW()
             WHERE idempotency_key = :key'
        )->execute([
            'status' => 'completed',
            'body' =
…