Backups, Disaster Recovery, Incident Response, and SLOs

Design a backup strategy covering databases, files, and configuration; understand why restore testing is mandatory; define RTO/RPO for disaster recovery; establish incident response practices; and define and calculate SLOs, SLIs, and error budgets.

This lesson covers ground the rest of the course — including the Observability module — does not touch: what happens when things go badly wrong despite good monitoring. Observability tells you something is broken; this lesson covers what you do about it structurally, ahead of time and during the incident itself — how you back up data so a disaster doesn't mean permanent loss, how you plan for how much data and downtime you can actually tolerate, how a team responds when an incident is happening right now, and how you formally define 'reliable enough' so that reliability work competes fairly against feature work instead of always losing.

What to Back Up, and How Often

A complete backup strategy for a typical PHP web application covers three categories of state, each with different tools and different risk profiles. The database is usually the most critical and most frequently changing: automated backups (a nightly full dump plus, for anything write-heavy, more frequent incremental backups or continuous point-in-time recovery via binary/WAL logs) stored somewhere physically separate from the primary database server. Uploaded files and object storage — user avatars, generated PDFs, attachments — are often overlooked because they don't live in the database, but losing them is just as real a loss to a user as losing their account row; object storage services typically offer built-in versioning and cross-region replication that should be turned on deliberately, not assumed to be default.

Configuration — environment variables, infrastructure-as-code definitions, secrets-manager contents, deployment scripts — is the third category, and the easiest to forget because 'it's just config, we know what's in it'. In practice, configuration drift and tribal knowledge mean that reconstructing a production environment from memory after a disaster is far harder than it sounds; configuration belongs in version control and/or a backed-up secrets manager, not solely in a senior engineer's head or an untracked server file.

Backup frequency and retention should be driven by two things: how much data you can afford to lose (covered formally below as RPO) and any compliance or business requirements for how long historical data must be retrievable. A common pattern is a tiered retention policy — daily backups kept for a couple of weeks, weekly backups kept for a few months, monthly backups kept for a year or more — which balances storage cost against the ability to recover from a problem that wasn't noticed immediately.

• Database: automated dumps plus (for critical systems) point-in-time recovery via binary/WAL logs. • Files/object storage: don't forget uploads — enable versioning and cross-region replication deliberately. • Configuration: infra-as-code, env vars, secrets — belongs in version control/secrets manager, not memory. • Tiered retention (daily/weekly/monthly, kept for different lengths) balances cost against recovery flexibility.

Restore Testing: A Backup You've Never Restored Is Not a Backup

The single most common backup failure mode has nothing to do with taking backups — it's discovering, during an actual emergency, that the backup is corrupted, incomplete, encrypted with a key nobody can find, or simply that nobody remembers the exact steps required to restore it under time pressure. A backup that has never been restored is, for practical purposes, an unverified claim that data protection exists, not an actual guarantee of it.

The fix is to treat restoring as a routine, ideally automated, operational task rather than a break-glass procedure improvised for the first time during a crisis. A well-run team periodically (monthly is a common cadence for less critical systems, more frequently for critical ones) spins up a fresh environment from the latest backup, verifies the restored data is complete and consistent, and confirms the entire restore procedure — including credentials, network access, and any manual steps — actually works end to end. Automating this as a scheduled job that restores into a staging environment and runs a handful of sanity checks (row counts, checksums, a smoke test against the app) turns restore testing from a chore someone might skip under time pressure into a fact that's continuously verified.

This discipline also has a valuable side effect: it measures how long a real restore actually takes, which is essential input to setting a realistic recovery time objective — a number nobody can honestly commit to without having actually timed the restore process at least once.

• An untested backup is an assumption, not a guarantee — corruption, missing keys, and stale procedures are discovered too late otherwise. • Schedule regular (ideally automated) restore drills into a fresh/staging environment, not just backup creation. • Verify restored data with sanity checks: row counts, checksums, an application smoke test. • Restore drills also produce the real timing data needed to set an honest recovery time objective.

Disaster Recovery: RTO and RPO

Recovery Time Objective (RTO) is the maximum acceptable duration between a disaster occurring and service being restored — 'how long can we be down before it's unacceptable'. Recovery Point Objective (RPO) is the maximum acceptable amount of data loss, measured as a duration — 'how much recent data can we afford to lose', expressed as 'up to the last hour' or 'up to the last 24 hours' rather than a byte count. These two numbers are independent: a system can have a strict RTO (must be back up in 15 minutes) but a looser RPO (losing the last hour of writes is tolerable), or vice versa, and the appropriate backup and failover architecture differs sharply depending on which constraint actually matters more for the business.

Backup frequency should be designed backward from the RPO target, not chosen arbitrarily. If the RPO is 'no more than 15 minutes of data loss', nightly database dumps are nowhere near sufficient — that architecture implies up to 24 hours of loss in the worst case — and the design instead needs continuous replication or frequent incremental/point-in-time-recovery backups so that at most 15 minutes of writes are ever at risk. Conversely, an internal admin tool used a few times a week might genuinely be fine with an RPO of 'up to 24 hours', for which a simple nightly dump is entirely adequate and buying continuous replication for it would be wasted engineering effort.

RTO drives a parallel set of architectural decisions: a tight RTO usually implies pre-provisioned standby infrastructure (a warm or hot standby database, infrastructure-as-code that can stand up a full environment in minutes) because manually provisioning a replacement environment from scratch during an incident routinely takes hours, which blows past any RTO measured in minutes.

• RTO (Recovery Time Objective): maximum acceptable downtime duration after a disaster. • RPO (Recovery Point Objective): maximum acceptable data loss, measured as a duration ('up to the last N minutes/hours'). • Backup frequency must be designed backward from RPO — nightly dumps cannot satisfy a 15-minute RPO. • A tight RTO usually requires pre-provisioned standby infrastructure, not manual rebuild-from-scratch during the incident.

Incident Response: Runbooks, On-Call, and Blameless Postmortems

When an incident is actively happening, the worst time to figure out 'who does what' is during the incident itself. A runbook is a written, specific procedure for a known type of failure — 'database replica lag exceeds threshold: do X, then Y, escalate to Z if unresolved after N minutes' — that turns a stressful, ambiguous situation into a checklist someone can follow even at 3am while only half awake. Runbooks are built from experience: every incident that lacked one is an opportunity to write one before the same failure happens again.

An on-call rotation designates, at any given moment, exactly one person (or a small, clearly-ordered escalation chain) responsible for responding to a paging alert, so that an incident never depends on 'whoever happens to notice'. A functioning rotation needs clear handoff points, a way to escalate when the primary on-call doesn't respond within a set window, and — critically — enough people in the rotation that no single person is perpetually on call, which is both an availability risk and a fast route to burnout.

After an incident is resolved, a blameless postmortem (also called a retro) documents what happened, its impact, the timeline, the root cause, and concrete follow-up action items — deliberately without assigning individual blame. The premise is that almost every real incident traces back to a system or process gap (missing validation, an untested backup, an alert that didn't fire, a runbook that didn't exist) rather than a single person's carelessness, and a culture that punishes the person who happened to be on call when a systemic gap surfaced teaches engineers to hide problems and avoid on-call duty, rather than surfacing and fixing the actual gap. The follow-up action items from postmortems are exactly the mechanism by which an organization's incident response, monitoring, and runbooks actually improve over time instead of repeating the same failure.

• Runbook: a written, specific procedure for a known failure type, reducing 3am improvisation. • On-call rotation: exactly one clear owner (plus escalation chain) at any moment; needs enough people to avoid burnout. • Blameless postmortem: documents timeline/root cause/impact/action items without assigning individual blame. • Most incidents trace to systemic gaps, not individual carelessness — blame culture suppresses the reporting that would fix those gaps.

SLIs, SLOs, and Error Budgets

A Service Level Indicator (SLI) is a specific, measured metric describing some aspect of the service's behavior from a user's perspective — for example, 'the percentage of HTTP requests that complete in under 200ms' or 'the percentage of requests that return a non-5xx status'. It's a raw measurement, not a target — just a number you can actually observe, typically pulled straight from the metrics and traces the Observability module's golden signals already track (latency and error rate are the two most common SLIs).

A Service Level Objective (SLO) attaches a target and a time window to an SLI: '99.9% of requests complete in under 200ms, measured over a rolling 30-day window' is an SLO built on the latency SLI above. The SLO is a commitment — internal or, in the stricter case of a formal SLA with customers, contractual — about how reliable the service should be, and it's deliberately set below 100%, because chasing perfect reliability past a certain point has rapidly diminishing returns and directly trades off against how fast a team can ship new features.

An error budget is the direct consequence of accepting an SLO below 100%: if the SLO is 99.9% over 30 days, the error budget is the remaining 0.1% — the amount of unreliability the team is explicitly allowed to spend before it's considered a problem. The error budget reframes reliability from an abstract virtue into a concrete, spendable resource: as long as the budget isn't exhausted, the team can reasonably prioritize shipping new features, even features that carry some risk; once the budget is exhausted (the service has already been unreliable beyond what the SLO allows for that window), the team should shift focus to reliability work — fixing root causes, paying down risky shortcuts — until the budget resets or recovers, rather than continuing to add feature risk on top of a service that's already over its reliability budget for the period.

• SLI: a measured indicator, e.g. '% of requests under 200ms' or '% of requests returning non-5xx'. • SLO: a target + time window on an SLI, e.g. '99.9% of requests under 200ms over 30 days'. • Error budget: the allowed amount of unreliability (100% - SLO) before the team must prioritize reliability over features. • Exhausted error budget = pause risky feature work and focus on reliability until it recovers.

Code Examples

The backup step alone only proves a file was written; the separate weekly restore drill actually restores that file into a real database and runs a sanity check (a minimum expected row count here, though checksums or an application smoke test work too), which is what turns 'we take backups' into 'we know our backups work'.

gunzip -c /tmp/restore_test.sql.gz | mysql restore_test_db With a 99.9% SLO over 50,000,000 requests, the error budget is 0.1% of 50,000,000 = 50,000 requests allowed to miss the 200ms target. Because the actual count (61,000) exceeds that budget, the concrete, numeric conclusion is that the team should pause new feature risk and focus on reliability until the rolling window's budget recovers. Expected result: Error budget: 50000 slow requests allowed Error budget EXHAUSTED — prioritize reliability work.

Taking nightly backups and assuming they're valid because the cron job reports success

Schedule regular, ideally automated, restore drills that actually restore into a fresh environment and sanity-check the result.

A backup job 'succeeding' only proves a file was written, not that the file is complete, uncorrupted, or restorable under the exact conditions of a real incident — only an actual restore proves that.

Setting an RPO/RTO target without ever timing an actual restore or failover

Run a real restore/failover drill first, measure how long it actually takes, then set RTO/RPO targets grounded in that measurement.

An RTO or RPO chosen without evidence is just a guess the team can't actually deliver on; restore drills provide the real timing data needed to commit to a number honestly.

Running a incident postmortem that focuses on which engineer made the mistake

Run a blameless postmortem focused on the systemic gap (missing test, no runbook, no alert) and concrete follow-up actions.

Blame-focused postmortems teach engineers to hide problems and avoid owning incidents, which suppresses the very information needed to find and fix the underlying systemic gap.

Treating an SLO as a target to hit exactly 100% of the time with zero tolerance

Set an SLO deliberately below 100% (e.g. 99.9%) and use the resulting error budget to balance reliability work against feature velocity.

Chasing 100% reliability has steeply diminishing returns and eliminates any error budget, which removes the team's ability to ship features that carry any risk at all — an SLO is meant to be a realistic, sustainable target, not a perfection requirement.

What does 'a backup you've never restored is not a backup' mean in practice? A. Backups should never be encrypted B. A backup's existence proves nothing about whether it can actually be restored successfully; only a tested restore verifies that C. Backups should be stored only on local disks D. Restoring backups is unnecessary if the backup job reports success

B. A backup's existence proves nothing about whether it can actually be restored successfully; only a tested restore verifies that — A backup file existing (and even a backup job reporting success) doesn't guarantee it's complete, uncorrupted, or restorable under real conditions — only an actual restore drill verifies that.

What is the difference between RTO and RPO? A. They are two names for the same concept B. RTO is the maximum acceptable downtime; RPO is the maximum acceptable data loss, measured as a duration C. RTO applies only to databases; RPO applies only to files D. RPO is measured in dollars; RTO is measured in minutes

B. RTO is the maximum acceptable downtime; RPO is the maximum acceptable data loss, measured as a duration — RTO (Recovery Time Objective) bounds how long service can be down; RPO (Recovery Point Objective) bounds how much recent data can be lost, expressed as a time duration like 'up to 15 minutes'.

Given an SLO of 99.9% over a 30-day window with 10,000,000 total requests, how many 'failing' requests does the error budget allow? A. 100 B. 1,000 C. 10,000 D. 100,000

C. 10,000 — The allowed failure rate is 100% - 99.9% = 0.1%. 0.1% of 10,000,000 is 10,000 requests — that is the error budget for the window.

Why should postmortems after an incident be blameless? A. Because no one is ever actually responsible for incidents B. Because assigning individual blame discourages people from surfacing the systemic gaps that actually caused the incident C. Because blameless postmortems are required by law D. Because it makes the postmortem document shorter

B. Because assigning individual blame discourages people from surfacing the systemic gaps that actually caused the incident — Most incidents stem from systemic gaps rather than individual carelessness; a blame culture suppresses honest reporting of those gaps, which is exactly the information needed to prevent recurrence.

# Nightly backup (cron, runs at 02:00)
mysqldump --single-transaction --quick app_db | gzip > /backups/app_db_$(date +%F).sql.gz
aws s3 cp /backups/app_db_$(date +%F).sql.gz s3://acme-backups/db/ --storage-class STANDARD_IA

# Weekly restore drill (cron, runs Sunday 03:00) — restores into a scratch DB
# and runs a sanity check, alerting if anything fails.
LATEST=$(aws s3 ls s3://acme-backups/db/ | sort | tail -n 1 | awk '{print $4}')
aws s3 cp "s3://acme-backups/db/${LATEST}" /tmp/restore_test.sql.gz
gunzip -c /tmp/restore_test.sql.gz | mysql restore_test_db

ROW_COUNT=$(mysql -N -e "SELECT COUNT(*) FROM restore_test_db.orders;")
if [ "$ROW_COUNT" -lt 1000 ]; then
    echo "RESTORE DRILL FAILED: order row count suspiciously low ($ROW_COUNT)" | mail -s "Restore drill alert" [email protected]
fi

<?php
declare(strict_types=1);

// SLI: percentage of requests completing under 200ms.
// SLO: 99.9% of requests under 200ms, measured over a rolling 30-day window.

function calculateErrorBudget(int $totalRequests, float $sloPercent): array {
    $allowedFailurePercent = 100.0 - $sloPercent;
    $allowedFailingRequests = (int) floor($totalRequests * ($allowedFailurePercent / 100.0));

    return [
        'slo_percent' => $sloPercent,
        'total_requests' => $totalRequests,
        'error_budget_requests' => $allowedFailingRequests,
    ];
}

// Suppose the service handles 50,000,000 requests over the 30-day window.
$budget = calculateErrorBudget(totalRequests: 50_000_000, sloPercent: 99.9);
echo "Error budget: {$budget['error_budget_requests']} slow requests allowed" . PHP_EOL;

// If 61,000 requests were actually over 200ms this window, budget is exhausted:
$actualSlowRequests = 61_000;
$budgetExhausted = $actualSlowRequests > $budget['error_budget_requests'];
echo $budgetExhausted
    ? "Error budget EXHAUSTED — prioritize reliability work."
    : "Within error budget — feature work can continue.";