Reliability and Robustness

Learning Objectives

Preserve invariants under failure; choose fail-fast vs graceful degradation.

Apply timeouts, cancellation, cleanup, idempotency and safe retries.

Isolate faults and recover without noisy defensive code.

Prerequisites: Modules 14, 51, 61.

Intuition and Definition

Reliability is continuing to deliver correct service; robustness is behaving sensibly when inputs or the environment are wrong.

Fail fast: detect programming errors and corrupted state immediately and stop (assertions, exceptions) — before damage spreads.

Graceful degradation: for *environmental* failures (a sensor offline, a service unreachable), keep a reduced but safe service (show "--" instead of a stale speed).

Idempotency: performing an operation twice has the same effect as once — prerequisite for safe retries.

Fault isolation: a failure in one component doesn't take down others (bulkheads, separate processes, supervisors).

Why It Matters

Everything fails eventually: networks, disks, sensors, dependencies. Robust objects keep their invariants and make failure states explicit and safe.

Syntax and Language Notes

using var cts = new CancellationTokenSource(TimeSpan.FromSeconds(2));   // timeout as cancellation
await client.SendAsync(request, cts.Token);

Complete Example

A fuel-level display that degrades gracefully on sensor faults, and an idempotent command handler that makes retries safe:

#include <iostream>
#include <optional>
#include <set>
#include <string>
#include <vector>

// graceful degradation with plausibility checks
class FuelGauge {
    std::optional<double> lastGood_;
    int faultyReadings_ = 0;
public:
    std::string update(std::optional<double> raw) {
        if (!raw || *raw < 0 || *raw > 100) {                 // missing or implausible
            if (++faultyReadings_ >= 3) { lastGood_.reset(); return "-- (sensor fault, see service)"; }
            return lastGood_ ? std::to_string(static_cast<int>(*lastGood_)) + "% (holding last value)" : "--";
        }
        faultyReadings_ = 0;
        lastGood_ = raw;
        return std::to_string(static_cast<int>(*raw)) + "%";
    }
};

// idempotent handler: duplicate deliveries (retries) are harmless
class ChargeHandler {
    std::set<std::string> processed_;
    long long balance_ = 0;
public:
    std::string handle(const std::string& requestId, long long amount) {
        if (!processed_.insert(requestId).second) return "duplicate " + requestId + " ignored";
        balance_ += amount;
        return "charged " + std::to_string(amount) + " (balance " + std::to_string(balance_) + ")";
    }
};

int mai
…

fuel: 42%
fuel: 42% (holding last value)
fuel: 41%
fuel: 41% (holding last value)
fuel: 41% (holding last value)
fuel: -- (sensor fault, see service)
fuel: 40%
charged 500 (balance 500)
charged 200 (balance 700)
duplicate req-1 ignored

Step-by-Step Walkthrough

A single missing or implausible reading holds the last good value (labelled as such) — the driver isn't misled by noise.

Three faulty readings in a row switch to an explicit fault display instead of a stale number.

A good reading resets the fault counter.

The charge handler records request ids; the retried req-1 doesn't double-charge.

Common Mistakes

Swallowing errors and continuing with corrupted state ("defensive" catch (...) {}).

Showing stale data as if it were current. Retrying non-idempotent operations.

Noise: null checks and try/catch everywhere instead of validating at boundaries and trusting invariants inside.

No cleanup on failure paths (use RAII / finally / using).

Trade-offs and Alternatives

Fail fast for bugs, degrade gracefully for environmental failures. Supervisors (Erlang/Akka) restart failed components; watchdogs reset stuck ECUs. Safety-critical systems define safe states formally (ISO 26262) — the examples here illustrate the idea, not a certified design.

Real-World Use

Instrument clusters showing "--" for invalid signals, telematics units buffering data offline, payment APIs with idempotency keys (Stripe), Kubernetes restarting crashed containers, circuit breakers (Module 51).

Practice

Q1. (Easy) What does idempotent mean?

Answer: Doing it twice has the same effect as doing it once.

Q2. (Medium) Fail fast or degrade gracefully when an internal invariant is violated?

Answer: Fail fast — it indicates a bug; continuing risks corrupting more state.

Q3. (Advanced) How should the processed-id set be bounded in a long-running service?

Answer: Store ids with timestamps and expire them after the maximum retry window (or persist them with a TTL in a database/cache), so memory doesn't grow forever while duplicates within the window are still detected.

Interview Question

Q1. "How do you make an operation safe to retry?" Answer: Make it idempotent: give each logical operation a unique id (idempotency key) chosen by the client, record processed ids with their results, and return the stored result for duplicates; or design operations as absolute state changes ("set balance to X at version V" with optimistic concurrency) instead of relative ones ("add X"). Combine with timeouts, bounded exponential back-off with jitter, and a retry budget.

Quick Quiz

Q1. Graceful degradation is appropriate for…?

Answer: Environmental/external failures (sensor offline, dependency unavailable).

Q2. What makes duplicated messages harmless?

Answer: Idempotent handling (e.g. deduplication by message/request id).

Q3. What should replace scattered null checks?

Answer: Validation at boundaries and strong invariants inside objects.

Summary

Keep invariants; fail fast on bugs, degrade gracefully on environmental faults.

Timeouts, cancellation, cleanup, idempotency and isolation make systems robust.

Related: Modules 14, 51, 61, 76.