Resilience, Incident Response, and Recovery

Examples include request floods, expensive queries, algorithmic complexity, resource exhaustion, retry storms, dependency outages, queue buildup, and maliciously large files.

18.1 Availability threats

Examples include request floods, expensive queries, algorithmic complexity, resource exhaustion, retry storms, dependency outages, queue buildup, and maliciously large files.

18.2 Resilience controls

Timeouts and cancellation Bounded queues and concurrency Rate limits and quotas Circuit breakers and bulkheads Backpressure and load shedding Caching with safe authorization context

Health checks that do not expose sensitive details Capacity planning and alerting Tested backups and restoration

Retries should be bounded and use backoff/jitter when appropriate. Retrying every failure immediately can worsen an outage.

Bounded retries with exponential backoff and jitter Output (CPython 3.13.1):

"Full jitter" picks a random delay below an exponentially growing, capped bound, so many clients that failed at the same moment do not retry in lockstep. After the last attempt the caller gives up and reports the failure instead of retrying forever.

import random

def backoff_delays(attempts: int, base: float = 0.2, cap: float = 5.0, seed: int = 7) -> list[float]:
    rng = random.Random(seed)            # seeded only so this example prints the same numbers
    return [round(rng.uniform(0, min(cap, base * 2 ** attempt)), 3) for attempt in range(attempts)]

delays = backoff_delays(6)
print("delays (s):", delays)
print("upper bounds:", [min(5.0, 0.2 * 2 ** a) for a in range(6)])
print("total wait before giving up:", round(sum(delays), 3), "s")

delays (s): [0.065, 0.06, 0.521, 0.116, 1.715, 1.828]
upper bounds: [0.2, 0.4, 0.8, 1.6, 3.2, 5.0]
total wait before giving up: 4.305 s

18.3 Secure failure modes

Decide which operations may continue during an identity, policy, database, or key-management outage. Do not silently allow privileged actions because a security service is unavailable. For other functions, a carefully designed degraded mode may preserve availability without exposing sensitive data.

18.4 Incident response lifecycle

Prepare roles, contacts, runbooks, and access.

Detect and triage.

Contain the incident.

Preserve evidence appropriately.

Eradicate root cause.

Recover services and rotate affected credentials.

Notify required stakeholders under applicable policies/law.

Conduct a blameless post-incident review.

Track corrective actions and verify them.

Text equivalent: preparation comes first; then detection and triage, containment, evidence preservation, eradication, recovery (including credential rotation), required notifications, a blameless review, and tracked corrective actions — whose lessons feed back into preparation.

flowchart TD
  P[Prepare] --> D[Detect and triage]
  D --> C[Contain]
  C --> E[Preserve evidence]
  E --> ER[Eradicate root cause]
  ER --> R[Recover and rotate credentials]
  R --> N[Notify as required]
  N --> PIR[Blameless review]
  PIR --> A[Track and verify actions]

18.5 Incident severity

Use a documented severity scheme based on impact, data sensitivity, exploitability, spread, and service disruption. Do not delay containment while waiting for perfect certainty when the risk of ongoing harm is high.

18.6 Vulnerability disclosure and patching

Provide a channel for reports, acknowledge submissions, reproduce safely, assess scope, coordinate fixes, communicate responsibly, and avoid exposing users during remediation. Define emergency patch authority and rollback criteria.

18.7 Forensics and evidence

Preserve relevant logs and timestamps, restrict access to evidence, document handling, and avoid unnecessary modification of affected systems. Follow organizational and legal requirements for evidence retention and disclosure.

Exercise

Write an incident runbook for a leaked API key: containment, revocation, log review, impact analysis, replacement, verification, communication, and prevention.

Quiz

? Every client retries a failed call immediately, forever. What happens during a partial outage?
+ A retry storm multiplies load and slows recovery. :: Bounded retries with backoff and jitter avoid this.
- The outage ends sooner. :: Extra load usually prolongs it.
- Nothing — retries are free. :: Each retry consumes capacity.
- Requests become idempotent. :: Retries make idempotency *necessary*, not automatic.

? A production API key is found in a public repository. What comes first?
- Write the post-incident review. :: That comes after containment and recovery.
+ Revoke or rotate the key to contain the incident. :: Stop ongoing misuse first.
- Wait for proof that someone used it. :: Do not delay containment for certainty when the risk is high.
- Delete the repository. :: Copies may exist; revocation is what stops use.

? Which are resilience controls? (select all that apply)
+ Timeouts and cancellation :: Stop waiting on stuck dependencies.
+ Bounded queues and load shedding :: Keep overload from spreading.
+ Circuit breakers :: Stop calling a failing dependency for a while.
- Health checks that print database passwords :: Health checks must not expose sensitive details.