Disaster recovery drills

RTO / RPO definitions and alignment

• Declare RTO/RPO by layer (ingress, app, data, messaging, batch)—avoid one site-wide number hiding weak spots.
• When customers or regulators impose commitments, drill frequency and measured outcomes must trace to compliance matrices or contracts.
• Recovery paths must be validated (including periodic restore drills), not runbook-only.

#!/bin/bash
# DR drill: RTO/RPO timeline recorder
# Usage: source this script, call record_event at each key step

DRILL_START=$(date +%s)
DRILL_LOG="dr-drill-$(date +%Y%m%d-%H%M%S).log"

record_event() {
  local label="$1"
  local now=$(date +%s)
  local elapsed=$(( now - DRILL_START ))
  echo "[T+${elapsed}s] $label" | tee -a "$DRILL_LOG"
}

# Example drill sequence:
# record_event "INCIDENT DECLARED - alert fired"
# record_event "ON-CALL RESPONDED - runbook opened"
# record_event "DIAGNOSIS COMPLETE - root cause identified"
# record_event "RECOVERY ACTION STARTED"
# record_event "SERVICE RESTORED - health checks green"
# record_event "INCIDENT CLOSED"

echo "Drill log: $DRILL_LOG"
echo "Target RTO: 30 min | Target RPO: 5 min"

Runbook structure

The runbook is shared by drills and real cutovers: each step names the executor, preconditions, commands or console paths, expected output, and rollback or escalation on failure. When the agent drafts content, require links to dashboards, ticket templates, and on-call owners.

1. Triage and severity: which symptoms escalate to DR, who may declare, handoff to incident management (incident ID).
2. Dependency topology: ordered DNS, LB, compute, data, secrets, third-party APIs; mark read-only deps and deferrable work.
3. Execution: numbered steps, estimated duration, parallelism; explicit human approval where automation is forbidden.
4. Validation: probes, canaries, sampled business transactions, data checks—aligned to RTO timeboxes.
5. Comms: status page, enterprise customer messaging, internal war room channels and cadence.
6. Abort and rollback: when to stop cutover, how to return to the original region or last known good.
7. Retrospective: actual RTO/RPO, gap analysis, improvements and due dates.

#!/bin/bash
# Disaster scenario playbooks

# --- Scenario A: Database primary failover ---
db_failover() {
  echo "[DR] Starting DB failover procedure..."
  # 1. Promote read replica to primary
  aws rds failover-db-cluster --db-cluster-identifier myapp-cluster
  # 2. Update application connection string
  kubectl set env deployment/myapp DB_HOST="$NEW_PRIMARY_ENDPOINT"
  # 3. Verify connectivity
  kubectl exec -it deploy/myapp -- pg_isready -h "$NEW_PRIMARY_ENDPOINT"
}

# --- Scenario B: Region failover ---
region_failover() {
  echo "[DR] Initiating region failover: us-east-1 -> us-west-2..."
  # 1. Update Route 53 health check and failover routing
  aws route53 change-resource-record-sets \
    --hosted-zone-id Z123... \
    --change-batch file://failover-record.json
  # 2. Scale up standby region
  aws autoscaling update-auto-scaling-group \
    --auto-scaling-group-name myapp-us-west-2 \
    --min-size 3 --desired-capacity 6
}

# --- Scenario C: Data restore from backup ---
restore_from_backup() {
  echo "[DR] Restoring from latest snapshot..."
  SNAPSHOT_ID=$(aws rds describe-db-snapshots \
    --db-instance-identifier myapp-db \
    --query 'sort_by(DBSnapshots,&SnapshotCreateTime)[-1].DBSnapshotIdentifier' \
    --output text)
  aws rds restore-db-instance-from-db-snapshot \
    --db-instance-identifier myapp-db-restored \
    --db-snapshot-identifier "$SNAPSHOT_ID"
  echo "RPO check: $(aws rds describe-db-snapshots \
    --db-snapshot-identifier $SNAPSHOT_ID \
    --query 'DBSnapshots[0].SnapshotCreateTime' --output text)"
}

DR decision and execution flow

  [ Monitoring / human detects anomaly ]
        │
        ▼
  ┌─────────────┐     Align: blast radius, false positive, declare or not
  │ Triage      │──── Out: incident owner, initial comms cadence
  └─────────────┘
        │
        ▼
  ┌─────────────┐     Against runbook: RTO/RPO, abort gates, approvals
  │ War room    │──── Log: executor per step, timestamps, command/ticket links
  └─────────────┘
        │
        ▼
  ┌─────────────┐     DNS / traffic / compute / data in order or parallel (predefined)
  │ Execute     │──── Secrets & config in target region; queue & batch plan ready
  └─────────────┘
        │
        ▼
  ┌─────────────┐     Probes + business samples + data reconciliation (per RPO)
  │ Tech verify │──── Abort or rollback if thresholds missed—no silent “close enough”
  └─────────────┘
        │
        ▼
  ┌─────────────┐     Status page / customer notice / internal sync
  │ External    │──── Align with legal and enterprise processes if applicable
  └─────────────┘
        │
        ▼
  ┌─────────────┐     Gaps, SPOFs, doc & automation holes → trackable items
  │ Drill retro │
  └─────────────┘

Drill scenarios and success criteria

• Define success criteria and resource caps separately for single AZ, whole region, and major vendor API outages.
• Comms plan: status page, enterprise notifications, war room channels; during drills avoid sending “real disaster” copy externally (use agreed test markers).
• After drills, log improvements: automation gaps, ambiguous docs, on-call blind spots—with owners and due dates.

# Post-drill retrospective template (5 Whys format)

## DR Drill Retrospective - [Date] [Scenario Name]

### Timeline Summary
| Time (T+) | Event |
|-----------|-------|
| 0s        | Incident declared / drill started |
| Xs        | On-call responded |
| Xs        | Root cause identified |
| Xs        | Recovery action started |
| Xs        | Service restored |

### Metrics vs Targets
| Metric | Target | Actual | Pass/Fail |
|--------|--------|--------|-----------|
| RTO    | 30 min |        |           |
| RPO    | 5 min  |        |           |

### 5 Whys Root Cause Analysis
- Why 1: Why did the incident occur?
- Why 2: Why did that condition exist?
- Why 3: Why was that allowed?
- Why 4: Why wasn't it caught earlier?
- Why 5: Why does the underlying gap exist?

### Action Items
- [ ] Runbook update: [what needs to change]
- [ ] Automation: [what to automate to reduce RTO]
- [ ] Monitoring: [what detection gap to close]
- [ ] Training: [who needs to practice what]

### Sign-off
- Drill owner: ___  |  Next drill date: ___

RTO minute timebox checklist

Enter a target RTO (minutes) to generate cumulative deadlines and checkable steps for tabletop timing or printouts. Ratios can be rewritten per org in your runbook; this is a generic template.