Topic lessonTopic: Audit coverage and recovery design
AWS Regional outage recovery: keeping a single writer
Choose which Region can accept writes before moving traffic. Follow fictional recovery evidence to decide when to proceed, hold or abort.
Order incident confirmation, write isolation, standby checks and gradual resumption using observed evidence and assigned roles.
This is a tabletop exercise for a fictional two-site application. It contains no live fault injection or service-specific promotion commands.
Check the business impact and the alternative route first
AWS recovery guidance calls for procedures independent of the failed site and a way to halt unsafe recovery. This scenario allows reads and writes to resume independently.
- Confirm impact → Isolate EastIncident lead checks impact
- Isolate East → Check WestRecovery lead verifies
- Check West → ResumeBusiness owner authorizes
- Confirm impact
- East order writes are unstable
- Isolate East
- Stop old-side updates
- Check West
- Verify consistent data and dependencies
- Resume
- Only West accepts writes
Hold updates while isolation is uncertain
| Time | Observation | Decision |
|---|---|---|
| 10:00 | East order writes fail | Declare the affected scope |
| 10:04 | West reads work; East write status is unknown | Hold West writes |
| 10:09 | East writes stopped; West recovery point confirmed | Run a synthetic West order |
| 10:13 | Synthetic order and read-back succeed | Resume on a limited route |
Recovery automation needs a clear trigger and a way to stop. An unknown state can hide an active write path, so starting writes on the new side could permit both sides to update.
Define what to do when evidence is missing
Separate the sources of evidence. Plan how to confirm state from the standby site when control in the failed site is unavailable. Verify that approach in a drill.
Trace uncertain writes. Distinguish accepted, committed and unresolved operations. Decide whether to replay only after deduplication and reconciliation checks.
Treat failback as a new switch. After the old site recovers, recheck data differences and write authority before directing traffic back.
Check your decision
Choose an option to see the result and explanation.
Check, then explain in your own words
- Record impact and current write status.
- Check isolation and standby dependencies.
- Verify a synthetic transaction before gradually reopening.
Check your understanding: Is a successful read in West enough to enable writes there?
Wait for evidence that East writes have stopped and West has a usable recovery point.
Read access and safe write authority are separate facts.
Sources and conditions checked
The official references used in this lesson are collected here with their scope, conditions and corresponding sections. Independent expert review and live AWS comparison have not been performed.
- Automate recoveryChecked:
Scope and conditionsPlan recovery independently of the failed site and provide an abort control.
Used in this lessonCheck the business impact and the alternative route first Hold updates while isolation is uncertain Define what to do when evidence is missing