Resilience

Networks drop packets, services restart, databases throttle, ECUs answer "busy".

Concept

Networks drop packets, services restart, databases throttle, ECUs answer "busy". Resilience strategies keep an application working — or failing gracefully — when dependencies misbehave.

Repeat an operation that failed *transiently* Wait longer between attempts (exponential: 50, 100, 200 ms...)

Randomize delays so many clients don't retry in lockstep (thundering herd)

Give up on slow operations; free resources Circuit breaker

After many failures, open the circuit and fail fast for a while; half-open lets a trial call through; success closes it

Return a degraded answer (cached value, default) Limit concurrent calls to a dependency so it can't exhaust all threads/connections Rate limiting

Limit request rate (protect yourself or respect a partner's quota; HTTP 429)

Make repeated requests safe (idempotency keys), a precondition for retrying writes

Why not retry everything? Retrying non-transient errors (400 Bad Request, validation errors, "invalid DID", authentication failures) wastes time and load; retrying non-idempotent operations can duplicate side effects (two flash jobs, two payments); retries multiply load on an already struggling service (retry storms). Retry only transient failures, with limits, backoff and jitter, and combine with timeouts and a circuit breaker.

In .NET, Polly (Polly.Core) implements these strategies as a composable ResiliencePipeline; Microsoft.Extensions.Http.Resilience adds AddStandardResilienceHandler() for HttpClient (rate limiter + total timeout + retry + circuit breaker + attempt timeout); System.Threading.RateLimiting and ASP.NET Core's AddRateLimiter provide limiters.

Why It Matters

A diagnostic back-end talks to thousands of vehicles over mobile networks; a test bench talks to flaky hardware. Without resilience, one slow dependency cascades into a system-wide outage.

Syntax

var pipeline = new ResiliencePipelineBuilder()
    .AddRetry(new RetryStrategyOptions { ShouldHandle = new PredicateBuilder().Handle<TimeoutException>(), MaxRetryAttempts = 3,
                                         BackoffType = DelayBackoffType.Exponential, UseJitter = true })
    .AddCircuitBreaker(new CircuitBreakerStrategyOptions { FailureRatio = 0.5, BreakDuration = TimeSpan.FromSeconds(30) })
    .AddTimeout(TimeSpan.FromSeconds(2))
    .Build();
await pipeline.ExecuteAsync(async ct => await CallEcuAsync(ct), token);
builder.Services.AddHttpClient<EcuApiClient>().AddStandardResilienceHandler();

Example

58.1 Retry, timeout, circuit breaker, fallback, bulkhead, rate limiting and idempotency

The program uses the Polly.Core and Polly.RateLimiting packages.

#:package [email protected]
#:package [email protected]
#:property PublishAot=false
using System.Threading.RateLimiting;
using Polly;
using Polly.CircuitBreaker;
using Polly.Fallback;
using Polly.Retry;
using Polly.Timeout;

// ---- Retry with exponential backoff + jitter, only for transient errors ----
int calls = 0;
var retry = new ResiliencePipelineBuilder<string>()
    .AddRetry(new RetryStrategyOptions<string>
    {
        ShouldHandle = new PredicateBuilder<string>().Handle<TimeoutException>(),   // NOT ArgumentException etc.
        MaxRetryAttempts = 4,
        BackoffType = DelayBackoffType.Exponential,
        UseJitter = true,
        Delay = TimeSpan.FromMilliseconds(50),
        OnRetry = a => { Console.WriteLine($"  retry {a.AttemptNumber + 1} after {a.Outcome.Exception!.GetType().Name}, waiting ~{a.RetryDelay.TotalMilliseconds:F0} ms (jittered)"); return default; },
    })
    .Build();
string result = await retry.ExecuteAsync(async _ => { await Task.Yield(); if (++calls < 3) throw new TimeoutException(); return "ECU answered"; });
Console.WriteLine($"retry: '{result}' after {calls} calls");

calls = 0;
try { await retry.ExecuteAsync<string>(async _ => { await
…

Expected Result

The jittered delays differ on every run. The circuit breaker opened after 4 calls with a 100% failure ratio (minimum throughput 4), rejected calls 5 and 6 *without calling the dependency*, and after the break duration let a trial call through, which succeeded and closed the circuit. Polly validates its options: an earlier draft of this program used a 300 ms break duration and failed at start-up with *"The field BreakDuration must be between 00:00:00.5000000 and 1.00:00:00"*.

retry 1 after TimeoutException, waiting ~67 ms (jittered)
  retry 2 after TimeoutException, waiting ~16 ms (jittered)
retry: 'ECU answered' after 3 calls
non-transient error is NOT retried: 1 call
timeout: operation cancelled after 100 ms
call 1: failed (503)
call 2: failed (503)
call 3: failed (503)
  circuit OPENED
call 4: failed (503)
call 5: rejected immediately (circuit open, dependency not called)
call 6: rejected immediately (circuit open, dependency not called)
  circuit HALF-OPEN (trial call allowed)
  circuit CLOSED
call 7: success
fallback: sensor offline -> using last known value 12 V
bulkhead: job 1 ran, job 2 ran, job 3 ran, job 4 rejected (bulkhead full)
rate limit (3/s): ok ok ok 429 429
idempotency: job-1 for VCU, retry -> job-1 for VCU, jobs stored 1

Common Mistakes

Retrying every exception, including bugs and validation errors.

Retrying without backoff/jitter (retry storms) or without a maximum.

Retrying non-idempotent operations.

Timeouts longer than the caller's own timeout; retries inside retries (multiplying attempts).

A fallback that hides failures completely (no logging/metrics).

One circuit breaker shared across unrelated dependencies.

Best Practices

Classify failures: transient (retry), permanent (fail fast), overload (back off, shed load).

Order strategies deliberately: total timeout → retry → circuit breaker → attempt timeout.

Use AddStandardResilienceHandler for HttpClient and tune it.

Log and emit metrics for retries, breaks and fallbacks (Polly integrates with telemetry).

Design write APIs with idempotency keys.

Interview Relevance

High: "explain the circuit breaker states", "why add jitter?", "which errors should be retried?", "what is idempotency and why does it matter for retries?", "bulkhead vs rate limiter?", "how does Polly work?".

Practice Questions

Add a HandleResult predicate so that HTTP 503 responses (not only exceptions) are retried.

Combine retry and circuit breaker so that a retry stops immediately when the circuit is open.

Configure AddStandardResilienceHandler for a typed client and observe it against the flaky endpoint from Module 36.

Add ASP.NET Core rate limiting (AddRateLimiter + RequireRateLimiting) to an endpoint and test the 429 response.

Quick Quiz

Q1. (Multiple choice) In which circuit-breaker state are calls rejected without reaching the dependency? A) Closed B) Open C) Half-open D) Isolated

Answer: B (verified: calls 5 and 6 were rejected immediately).

Q2. (True/False) An ArgumentException("invalid DID") should be retried.

Answer: False. It's not transient; the example's retry pipeline let it through after 1 call (verified).

Q3. (Predict the output) A fixed-window limiter allows 3 requests per second; 5 requests arrive at once.

Answer: ok ok ok 429 429 (verified).

Q4. (Find the bug) POST /flash-jobs is retried on timeout and two flash jobs start for the same ECU.

Answer: The operation isn't idempotent; send an idempotency key and have the server return the existing job for a repeated key.

Coding Challenge

Without any library, write RetryAsync<T>(Func<Task<T>> action, int maxAttempts, Func<Exception, bool> isTransient) with exponential backoff (25 ms × 2^attempt) and use it on an operation that fails twice with TimeoutException.

int calls = 0;
string result = await RetryAsync(async () =>
{
    await Task.Yield();
    if (++calls < 3) throw new TimeoutException("no response");
    return "62 F1 90 ...";
}, maxAttempts: 5, isTransient: ex => ex is TimeoutException or IOException);
Console.WriteLine($"{result} after {calls} calls");

static async Task<T> RetryAsync<T>(Func<Task<T>> action, int maxAttempts, Func<Exception, bool> isTransient)
{
    for (int attempt = 1; ; attempt++)
    {
        try { return await action(); }
        catch (Exception ex) when (attempt < maxAttempts && isTransient(ex))
        {
            var delay = TimeSpan.FromMilliseconds(25 * Math.Pow(2, attempt));
            Console.WriteLine($"attempt {attempt} failed: {ex.Message}; waiting {delay.TotalMilliseconds} ms");
            await Task.Delay(delay);
        }
    }
}