Topic lessonTopic: CI/CD design: from checks to release and recovery
Incident postmortems: evidence, customer impact and verified corrective actions
An incident postmortem uses evidence to investigate contributing factors and produce verifiable corrective actions. Go beyond “be more careful”: distinguish observations from unknowns and identify missing checks in a fictional deployment incident.
Separate timeline, impact, evidence and hypotheses, then define corrective actions with owners, deadlines and acceptance checks.
In preceding case R2, application B wrote archived into a shared database and old version A could not read it. Returning to A left those values in place. This lesson reviews that fictional incident; it is not a report of an actual outage.
I1: read the incident within the limits of the record
I1 concerns reading order details. All timestamps are on one fictional day, and affected-order counts are cumulative distinct orders, not repeated attempts. These are supplied teaching records, not live AWS or customer measurements.
| Time | Observed fact | What it does not establish |
|---|---|---|
| 10:00 | B deployment enables archived writes | This does not prove all orders failed from 10:00 |
| 10:03 | Read failures detected: A rejects values in 12 distinct orders | Not necessarily 12 people; the total impact ceiling remains unknown |
| 10:04 | Traffic returns to A; the same 12 orders still fail to read | Completed rollback is not completed recovery |
| 10:08 | Compatible fix C deployed; targeted read checks pass | Does not establish that every feature and customer is unaffected |
| 10:12 | Retries for the 12 orders, required updates and preserved valid orders are checked | Does not prove zero unrecorded loss or zero out-of-scope impact |
Keep evidence, hypotheses and unknowns separate
- Timeline and impact → Matched evidenceCorrelate relevant records
- Matched evidence → Missing checkReview test coverage
- Missing check → Owned corrective actionDefine a concrete change
- Owned corrective action → Verify the actionRun acceptance checks
- Timeline and impact
- Twelve orders; bounded observations.
- Matched evidence
- A errors matched to archived values.
- Missing check
- Read B-written values using A.
- Owned corrective action
- Add old/new read-write tests.
- Verify the action
- Confirm the failure is detected.
| Category | Example statement | Next treatment |
|---|---|---|
| Fact | A’s errors are matched to 12 orders containing archived | Retain version, timestamp and evidence references |
| Candidate contributing factor | Pre-release tests omitted B-write/A-read combinations | Inspect test definitions and results before confirming |
| Unsupported assertion | The operator was careless; all AWS was down | Investigate mechanisms and evidence instead of asserting |
| Unknown | Affected people and exact pre-detection onset | Label unknown; assign an investigation method and owner |
Review user impact, evidence supporting causal explanations and why existing tests missed the issue. Focus on improvements that help other changes rather than blaming individuals; keep facts and responsibilities explicit.
Verify the gap before choosing the corrective action
Extend I1 with an explicit investigation result: the saved test definitions and runs cover only A-write/A-read and B-write/B-read. There are no cross-version checks. This additional evidence, rather than timeline proximity alone, supports identifying the coverage gap as a contributing factor.
| Writer | Reader | Value | Supplied expectation in I1 |
|---|---|---|---|
| A | A | new / done | Readable |
| A | B | new / done | Readable |
| B | B | archived | Readable |
| B | A | archived | Not readable: detect the defect |
| D | C | archived | Readable: prepared recovery target |
Give actions owners, deadlines and acceptance conditions
| Owner and deadline | Concrete change | Acceptance condition |
|---|---|---|
| Application owner: before the next state-value change | Add writer/reader fixtures to CI | Detect B→A archived failure; specified compatible C/D cases pass |
| Release-process owner: before the next release | Bind recovery version, configuration and data conditions to the gate | Reject incompatible A; allow only validated compatible C |
| Operations owner: within three working days of review | Record detection, rollback and recovery-check times separately | Avoid counting the same orders twice and retain unknowns |
| Review owner: at the next release decision | Compare action results with the original failure case | Require evidence of defect detection, not merely an edited document |
These are proposed actions for I1, not claims of changes implemented in a live CI or system. Overdue, unverified or harmful actions remain open or need revision; link completion to actual verification results.
Choose an evidence-based incident review
Choose an option to see the result and explanation.
Check, then explain in your own words
- Explain the twelve confirmed affected orders and unknown scope in I1.
- Separate facts, hypotheses and additional investigation evidence.
- Identify the test that detects the B→A failure.
- Define action owners, deadlines, acceptance checks and follow-up.
Check your understanding: “Rollback to A succeeded, but the same twelve orders still failed.” What belongs in the review?
Record completed rollback separately from continuing business impact. Do not recount the twelve orders; retain evidence that A cannot read archived, targeted checks after C, the 10:12 reconciliation and unknowns.
A timeline should relate actions to user outcomes and evidence, not merely list operations.
Sources and conditions checked
The official references used in this lesson are collected here with their scope, conditions and corresponding sections. Independent expert review and live AWS comparison have not been performed.
Scope and conditionsCustomer impact, supporting evidence, contributing factors, blameless analysis and corrective actions. Documentation review, not live AWS/database validation.
Used in this lessonKeep evidence, hypotheses and unknowns separate