Cancellation, Timeouts, Retries and Resilience

Things go wrong all the time in concurrent and networked systems: a server is slow, a user clicks "cancel", a deadline passes.

Learning Objectives

Implement cooperative cancellation with tokens and exceptions; track cancellation state; propagate it; clean up; handle cancellation races and cancellation-safe acquisition.

Apply timeouts, deadline propagation, per-attempt versus overall timeouts.

Design retry policies with exponential backoff, jitter and retry budgets; require idempotency; prevent retry storms.

Use circuit breakers, bulkheads, rate limiting, hedged requests, graceful degradation and partial results; clean up after timeouts.

Compare cancellation in threads and coroutines; explain why killing threads is dangerous; handle shutdown signals; test resilience.

Plain-Language Explanation

Things go wrong all the time in concurrent and networked systems: a server is slow, a user clicks "cancel", a deadline passes. Good systems *ask* work to stop rather than ripping it out mid-step (cancellation is cooperative), give up waiting at sensible times (timeouts and deadlines), try again carefully when it is safe (retries with backoff), and stop hammering something that is clearly broken (circuit breakers). The goal is to fail quickly, cleanly and without making the problem worse.

Precise Technical Explanation

The canceller *requests*; the cancelled code *checks* and stops at a safe point, releasing resources. Every mainstream runtime works this way because only the running code knows where its invariants hold.

A token is a read-only view of "has cancellation been requested?" plus a way to register callbacks: .NET CancellationToken (source: CancellationTokenSource), JavaScript AbortSignal (AbortController), Go context.Context (Done() channel), C++20 std::stop_token, Java Thread.interrupt() + isInterrupted(), Python threading.Event (threads) or Task.cancel() (asyncio).

Many runtimes surface cancellation as an exception that unwinds the stack and runs finally: OperationCanceledException, asyncio.CancelledError (a BaseException since 3.8), InterruptedException, Kotlin CancellationException, AbortError DOMException.

Requested → acknowledged (code noticed) → completed (task ended as cancelled). A task may also complete normally despite a request if it was already done.

Pass the token/context to every callee (Go's convention: ctx is the first parameter); link tokens (CancellationTokenSource.CreateLinkedTokenSource, AbortSignal.any, child contexts) so a parent cancels children.

Release locks, close files and connections, roll back partial work in finally/defer/RAII; re-raise the cancellation afterwards.

The operation may complete at the same moment cancellation is requested. Code must tolerate both orders: completion APIs settle once (Module 9), cleanup must be idempotent, and "cancelled" must not be reported for work that actually committed.

Cancellation-safe resource acquisition If cancellation strikes between *acquiring* a resource and *registering* its cleanup, the resource leaks. Acquire inside try/with/using constructs, or use APIs that are atomic with respect to cancellation (async with on a pool connection).

Bound how long to wait. A timeout requests cancellation of the wait (and, if the API supports it, of the operation) — it does not guarantee that external work (a database query, a remote server) has stopped.

Convert a timeout into an absolute deadline at the edge (request arrives at 10:00:00.000, deadline 10:00:02.000) and pass the *remaining* budget downstream (gRPC deadlines, Go context.WithDeadline), so inner calls never wait longer than the caller is willing to.

Per-attempt versus overall timeout With retries, use both: each attempt gets a short timeout; the whole operation gets an overall deadline that caps attempts plus backoff.

Retry only *transient* failures (timeouts, 503, connection reset), never permanent ones (400, authentication failure); cap attempts; respect Retry-After.

Wait base × 2^attempt (capped) between attempts so a struggling service gets breathing room.

Randomise the delay ("full jitter": uniform in [0, backoff]) so clients that failed together do not retry together.

Limit retries to a fraction of normal traffic (e.g. retries ≤ 10% of requests) so that during an outage retries cannot multiply load.

An operation is idempotent if performing it twice has the same effect as once (PUT, DELETE, "set balance to X"). Non-idempotent operations ("charge €10") must carry an idempotency key so the server can detect duplicates before you retry them.

When every layer retries (client ×3, gateway ×3, service ×3), one failure becomes 27 calls to the database — amplification that turns a blip into an outage. Retry at one layer, use budgets, and fail fast when overloaded.

A state machine around a dependency: closed (calls flow, failures counted) → open after too many failures (calls fail immediately without touching the dependency) → half-open after a cool-down (a few trial calls decide whether to close again). Protects both caller and callee.

Isolate resources per dependency or tenant (separate pools, semaphores, connection limits) so one slow dependency cannot consume every thread or connection — like watertight compartments in a ship.

Cap request rate (token bucket, leaky bucket — Module 18) to protect yourself or respect a provider's limits.

Hedged requests and tail latency (advanced) If a request has not answered by the p95 latency, send a duplicate to another replica and take whichever answers first — cuts tail latency at the cost of extra load; only for idempotent reads; cancel the loser.

Serve a reduced but useful response when a dependency fails (cached recommendations, "prices temporarily unavailable") instead of failing the whole request.

Return what completed by the deadline, clearly marked as partial (search results from 9 of 10 shards).

Resource cleanup on timeout After a timeout, cancel the underlying operation if possible, close connections that may be in an unknown protocol state, and release pool slots — or the next request inherits a broken resource.

Cancellation in threads versus coroutines Threads: check a flag/event or interrupt() and handle InterruptedException; blocking calls must be interruptible or have timeouts. Coroutines: cancellation is injected at the next suspension point; CPU loops need explicit yield/check points.

Why forcibly killing a thread is dangerous Killing a thread (Thread.stop — deprecated and removed in Java 20+ as an operation, TerminateThread, asynchronous pthread_cancel) can stop it while it holds a lock, half-way through updating a data structure, or inside the memory allocator, leaving the process permanently inconsistent. Demonstrated below with pthread_cancel: the mutex stays locked forever. Only processes are safely killable — the OS reclaims their resources.

SIGTERM/SIGINT (POSIX), console control events and service stop requests (Windows), container stop timeouts (Kubernetes sends SIGTERM, then SIGKILL after the grace period). Convert the signal into a cancellation of the root scope.

Inject latency, errors and dropped connections (fault-injection proxies such as Toxiproxy, chaos engineering), test timeouts with fake clocks, verify that retries respect budgets and that circuit breakers open and recover.

Why It Matters in Real Systems

Most large outages involve timeouts that were too long or missing, retries that amplified load, or work that kept running after it was no longer wanted. Resilience patterns are what keep one failing component from taking down everything around it.

Key Terms

Request + voluntary stop at safe points Cancellation token / context

Object carrying the cancellation request Absolute time by which work must finish Exponential backoff + jitter Growing, randomised retry delays

Identifier that lets a server detect duplicate requests Load amplification by layered retries

Fails fast while a dependency is unhealthy Isolated resource pool per dependency Duplicate request to cut tail latency

Diagram or Execution Timeline

flowchart LR
  C[closed: calls flow] -- failures exceed threshold --> O[open: fail fast]
  O -- cool-down elapses --> H[half-open: trial calls]
  H -- trial succeeds --> C
  H -- trial fails --> O

Syntax and API Overview

The specification's timeout example (CPython 3.13.1):

In Python 3.11+ asyncio.TimeoutError is an alias of the built-in TimeoutError, so either name catches it (in 3.10 and earlier they were different classes). wait_for cancels operation() when the timeout expires and waits for the cancellation to complete; if operation were talking to a remote system, the remote side might still finish the work.

import asyncio

async def operation():
    await asyncio.sleep(5)
    return "done"

async def main():
    try:
        result = await asyncio.wait_for(operation(), timeout=1.0)
        print(result)
    except asyncio.TimeoutError:
        print("operation timed out")

asyncio.run(main())

operation timed out

Complete Code Example

Retries with exponential backoff and full jitter, an idempotency key, a retry budget and a circuit breaker — a deterministic simulation with a fake clock and a seeded flaky service (CPython 3.13.1):

Deadline propagation with Go's context (Go 1.23.4): the handler gets 300 ms; each downstream call sees only the *remaining* budget:

Why threads must not be killed — asynchronous pthread_cancel of a thread holding a mutex (C, Linux). The victim is cancelled at a cancellation point *inside* its critical section; the mutex is never unlocked:

Cancellation tokens with registered cleanup and linked sources in .NET 10:

import random

rng = random.Random(42)


class FlakyService:
    """Fails transiently for the first `failures` calls; deduplicates by idempotency key."""
    def __init__(self, failures):
        self.failures, self.calls, self.applied = failures, 0, {}

    def charge(self, key, amount):
        self.calls += 1
        if key in self.applied:                       # duplicate retry of a request that already succeeded
            return self.applied[key]
        if self.calls <= self.failures:
            raise TimeoutError("transient")
        self.applied[key] = f"charged {amount}"
        return self.applied[key]


def call_with_retry(fn, *, attempts=5, base=0.1, cap=2.0, deadline=3.0):
    clock, history = 0.0, []
    for attempt in range(attempts):
        try:
            result = fn()
            history.append(f"t={clock:.2f}s attempt {attempt + 1}: ok")
            return result, history
        except TimeoutError:
            backoff = min(cap, base * 2 ** attempt)
            sleep = rng.uniform(0, backoff)            # full jitter
            history.append(f"t={clock:.2f}s attempt {attempt + 1}: timeout, sleep {sleep:.2f}s (backoff cap {backoff:.2f}s)")
            if
…

t=0.00s attempt 1: timeout, sleep 0.06s (backoff cap 0.10s)
t=0.06s attempt 2: timeout, sleep 0.01s (backoff cap 0.20s)
t=0.07s attempt 3: timeout, sleep 0.11s (backoff cap 0.40s)
t=0.18s attempt 4: ok
result: charged 25 | server calls: 4 | charges applied: 1
retrying the same idempotency key again: charged 25 | charges applied: 1
retry budget: 100 failed requests -> 10 retries allowed (not 100-300)
t=0s: failed (closed)
t=1s: failed (closed)
t=2s: failed (open)
t=3s: rejected fast (open)
t=4s: rejected fast (open)
t=6s: rejected fast (open)
t=8s: ok (closed)
t=9s: ok (closed)

package main

import (
	"context"
	"fmt"
	"time"
)

func call(ctx context.Context, name string, work time.Duration) error {
	deadline, _ := ctx.Deadline()
	remaining := time.Until(deadline).Round(50 * time.Millisecond)
	select {
	case <-time.After(work):
		fmt.Printf("%-8s finished (it had ~%v left)\n", name, remaining)
		return nil
	case <-ctx.Done():
		fmt.Printf("%-8s abandoned: %v (it had ~%v left)\n", name, ctx.Err(), remaining)
		return ctx.Err()
	}
}

func main() {
	ctx, cancel := context.WithTimeout(context.Background(), 300*time.Millisecond)
	defer cancel()
	_ = call(ctx, "auth", 100*time.Millisecond)
	_ = call(ctx, "catalog", 100*time.Millisecond)
	_ = call(ctx, "pricing", 500*time.Millisecond) // exceeds what is left of the budget
}

auth     finished (it had ~300ms left)
catalog  finished (it had ~200ms left)
pricing  abandoned: context deadline exceeded (it had ~100ms left)

#include <errno.h>
#include <pthread.h>
#include <stdio.h>
#include <unistd.h>

static pthread_mutex_t m = PTHREAD_MUTEX_INITIALIZER;
static int balance = 100;

static void *victim(void *arg)
{
    (void)arg;
    pthread_mutex_lock(&m);
    balance -= 30;            /* half of a two-step update ... */
    sleep(1);                 /* sleep() is a cancellation point: the thread dies here */
    balance += 30;            /* ... the second half never runs */
    pthread_mutex_unlock(&m);
    return NULL;
}

int main(void)
{
    pthread_t t;
    pthread_create(&t, NULL, victim, NULL);
    usleep(100 * 1000);
    pthread_cancel(t);        /* "kill" the thread from outside */
    pthread_join(t, NULL);
    int rc = pthread_mutex_trylock(&m);
    printf("after cancelling the thread: balance = %d (invariant broken), trylock = %s\n",
           balance, rc == EBUSY ? "EBUSY: the mutex is locked forever" : "acquired");
    return 0;
}

after cancelling the thread: balance = 70 (invariant broken), trylock = EBUSY: the mutex is locked forever

using System;
using System.Threading;
using System.Threading.Tasks;

using var appShutdown = new CancellationTokenSource();
using var requestTimeout = new CancellationTokenSource(TimeSpan.FromMilliseconds(80));
using var linked = CancellationTokenSource.CreateLinkedTokenSource(appShutdown.Token, requestTimeout.Token);

linked.Token.Register(() => Console.WriteLine("registered callback: releasing the connection slot"));

try
{
    for (int step = 1; ; step++)
    {
        linked.Token.ThrowIfCancellationRequested();       // cooperative check in a CPU loop
        await Task.Delay(30, linked.Token);                // cancellable wait
        Console.WriteLine($"step {step} done");
    }
}
catch (OperationCanceledException)
{
    Console.WriteLine($"cancelled: timeout={requestTimeout.IsCancellationRequested}, shutdown={appShutdown.IsCancellationRequested}");
}

step 1 done
step 2 done
registered callback: releasing the connection slot
cancelled: timeout=True, shutdown=False

Line-by-Line Explanation

Retry: three transient failures, each followed by a jittered sleep below an exponentially growing cap; the fourth attempt succeeds. The idempotency key makes a later duplicate harmless (applied stays 1).

Budget: retries are limited to 10% of requests, so a 100-request outage produces 10 retries instead of 100–300.

Circuit breaker: three failures open the circuit at t = 2; calls during the 5 s cool-down (t = 3, 4, 6) are rejected without touching the dependency; at t = 8 the cool-down has elapsed, the breaker goes half-open, the trial call succeeds (the dependency recovered at t = 7) and the circuit closes. If the trial had failed, the breaker would have re-opened for another cool-down.

Go: each call computes what is left of the 300 ms budget; pricing would need 500 ms but only ~100 ms remain, so it is abandoned when the context expires.

C: pthread_cancel acts at the cancellation point sleep(); the thread dies holding the mutex with the balance half-updated. Cleanup handlers (pthread_cleanup_push) can mitigate this, which is why POSIX has them — but cooperative flags are simpler and safer.

C#: the linked token fires when either source cancels; Register runs cleanup on cancellation; Task.Delay(…, token) ends early with TaskCanceledException (a subclass of OperationCanceledException).

Output

Shown under each program; the simulation is deterministic (seeded); the Go and .NET outputs depend on timing but are stable with these margins.

Interleaving or State Trace

Circuit-breaker states:

closed, closed, open (opened at t = 2) open (cooling until t = 7) open → half-open

Correctness and Safety Analysis

Retries are safe only for idempotent operations or with idempotency keys; timeouts must be paired with cancellation and cleanup; killing threads is never safe; cancellation must flow through every layer to be useful.

Common Mistakes and Misconceptions

Retrying non-idempotent operations without a key (double charges).

Retrying at every layer (storms); no jitter (thundering herds).

Timeouts without cancellation (work continues, resources leak).

Swallowing cancellation exceptions.

Using huge default timeouts (or none) on outbound calls.

Believing a timeout means the remote side did nothing.

Failure Modes and Debugging Steps

Load spikes after a dependency recovers: synchronized retries — add jitter and budgets.

Hung requests: missing timeouts — audit every outbound call.

Connection-pool exhaustion after timeouts: connections not released — check cleanup on the timeout path.

Breaker never closes: cool-down too long or trial traffic too small — tune and monitor state transitions.

Performance and Scalability Implications

Well-tuned timeouts and breakers cut wasted work and protect capacity; hedging trades extra load (typically +5%) for large tail-latency reductions; retries add load precisely when the system is weakest — budget them.

Security Considerations

Without timeouts, slow clients or dependencies can tie up all workers (slowloris-style DoS). Rate limits and bulkheads protect shared resources; idempotency keys prevent duplicate side effects that attackers could otherwise trigger by replaying requests.

Platform and Runtime Caveats

asyncio.TimeoutError is an alias of TimeoutError only from Python 3.11.

Java removed Thread.stop's ability to stop threads (it throws UnsupportedOperationException since Java 20); use interruption.

pthread_cancel semantics (deferred vs asynchronous, cancellation points) are POSIX-specific; Windows has no safe equivalent.

CancellationTokenSource(TimeSpan) uses a timer; dispose sources to release it.

Real-World Applications

gRPC deadlines, AWS SDK retry modes with jitter and budgets, Resilience4j/Polly/Hystrix circuit breakers, Kubernetes graceful termination, payment APIs with idempotency keys (Stripe), search engines returning partial results.

Best Practices

Every outbound call: a timeout, cancellation, and a clear retry policy.

Propagate deadlines, not fixed timeouts.

Retry transient failures only, with capped exponential backoff, full jitter and a budget, at one layer.

Make writes idempotent or use idempotency keys.

Wrap fragile dependencies in circuit breakers and bulkheads; degrade gracefully.

Never kill threads; cancel cooperatively and test cancellation paths.

Exercises

E1 · Beginner. Why is "PUT /users/42 {name: 'Ada'}" safe to retry while "POST /payments {amount: 10}" is not?

Answer: The PUT sets a state; repeating it leaves the same state (idempotent). The POST creates a new payment each time, so a retry after an ambiguous timeout may charge twice — unless it carries an idempotency key.

E2 · Intermediate. A request has a 2 s deadline. It makes three sequential downstream calls with fixed 2 s timeouts. What is wrong and how do you fix it?

Answer: The calls can take up to 6 s in total, long after the caller gave up. Propagate the deadline: each call gets min(own timeout, remaining budget), and calls stop as soon as the request's context is cancelled.

E3 · Advanced challenge. Design a hedged read: send to replica A; if no reply within the p95 (50 ms), also send to replica B; return the first reply and cancel the other.

Answer: Start A; await first_completed(A, sleep(50ms)); if A finished, return it; otherwise start B and await first_completed(A, B); cancel the loser and await its cancellation. Only for idempotent reads, with a hedging budget (e.g. ≤ 5% extra requests).

Quiz

? What does a timeout guarantee by itself?
- That the remote operation was not performed :: The server may complete it anyway.
+ That the caller stops waiting after the time limit :: Stopping the work also needs cancellation.
- That resources are released :: Cleanup must be programmed.
- That retrying is safe :: Only for idempotent operations.

? Why add jitter to exponential backoff?
+ So clients that failed together do not retry at the same moments. :: Randomisation spreads the load.
- To make retries faster. :: It may make individual retries slower.
- To guarantee success. :: Nothing guarantees success.
- Because exponential growth is too slow. :: Jitter is about synchronisation, not growth.

? Which are dangerous? (select all that apply)
+ Forcibly killing a thread that holds a lock :: The lock and data stay inconsistent.
+ Retrying at every layer of a call chain :: Multiplies load (retry storm).
- Propagating the remaining deadline to downstream calls :: That is the recommended practice.
+ Catching and ignoring cancellation exceptions :: Scopes cannot cancel the task.

Summary

Cancellation is cooperative: tokens and contexts carry requests, code checks them at safe points and cleans up. Timeouts bound waiting; deadlines propagate budgets. Retries need transient errors, idempotency, backoff with jitter and budgets. Circuit breakers, bulkheads, rate limits, hedging and graceful degradation contain failures. Never kill threads; test every failure path.