INFRASTRUCTURE LEARNINGINFRASTRUCTURE LEARNING
インフラ大全インフラ大全IT INFRASTRUCTURE / LEARNING
Infrastructure learning guide
Find a lesson or lab/

Esc to close · Search terms stay on your device

LEARNING日本語

Topic lessonTopic: CI/CD design: from checks to release and recovery

Incident postmortems: evidence, customer impact and verified corrective actions

An incident postmortem uses evidence to investigate contributing factors and produce verifiable corrective actions. Go beyond “be more careful”: distinguish observations from unknowns and identify missing checks in a fictional deployment incident.

What you will learn

Separate timeline, impact, evidence and hypotheses, then define corrective actions with owners, deadlines and acceptance checks.

Prerequisite

In preceding case R2, application B wrote archived into a shared database and old version A could not read it. Returning to A left those values in place. This lesson reviews that fictional incident; it is not a report of an actual outage.

I1: read the incident within the limits of the record

I1 concerns reading order details. All timestamps are on one fictional day, and affected-order counts are cumulative distinct orders, not repeated attempts. These are supplied teaching records, not live AWS or customer measurements.

I1 supplied timeline
TimeObserved factWhat it does not establish
10:00B deployment enables archived writesThis does not prove all orders failed from 10:00
10:03Read failures detected: A rejects values in 12 distinct ordersNot necessarily 12 people; the total impact ceiling remains unknown
10:04Traffic returns to A; the same 12 orders still fail to readCompleted rollback is not completed recovery
10:08Compatible fix C deployed; targeted read checks passDoes not establish that every feature and customer is unaffected
10:12Retries for the 12 orders, required updates and preserved valid orders are checkedDoes not prove zero unrecorded loss or zero out-of-scope impact

Keep evidence, hypotheses and unknowns separate

From I1 to improvement: connect evidence to a testable action
From I1 to improvement: connect evidence to a testable action1: Timeline and impact — Correlate relevant records — Matched evidence. 2: Matched evidence — Review test coverage — Missing check. 3: Missing check — Define a concrete change — Owned corrective action. 4: Owned corrective action — Run acceptance checks — Verify the action1234Timeline and impactMatched evidenceMissing checkOwned correctiveactionVerify the actionFrom I1 to improvement: connect evidence to a testable action1: Timeline and impact — Correlate relevant records — Matched evidence. 2: Matched evidence — Review test coverage — Missing check. 3: Missing check — Define a concrete change — Owned corrective action. 4: Owned corrective action — Run acceptance checks — Verify the action1234Timeline andimpactMatchedevidenceMissing checkOwnedcorrectiveactionVerify theaction
  1. Timeline and impact → Matched evidenceCorrelate relevant records
  2. Matched evidence → Missing checkReview test coverage
  3. Missing check → Owned corrective actionDefine a concrete change
  4. Owned corrective action → Verify the actionRun acceptance checks
Timeline and impact
Twelve orders; bounded observations.
Matched evidence
A errors matched to archived values.
Missing check
Read B-written values using A.
Owned corrective action
Add old/new read-write tests.
Verify the action
Confirm the failure is detected.
Statements supported by I1
CategoryExample statementNext treatment
FactA’s errors are matched to 12 orders containing archivedRetain version, timestamp and evidence references
Candidate contributing factorPre-release tests omitted B-write/A-read combinationsInspect test definitions and results before confirming
Unsupported assertionThe operator was careless; all AWS was downInvestigate mechanisms and evidence instead of asserting
UnknownAffected people and exact pre-detection onsetLabel unknown; assign an investigation method and owner

Review user impact, evidence supporting causal explanations and why existing tests missed the issue. Focus on improvements that help other changes rather than blaming individuals; keep facts and responsibilities explicit.

Verify the gap before choosing the corrective action

Extend I1 with an explicit investigation result: the saved test definitions and runs cover only A-write/A-read and B-write/B-read. There are no cross-version checks. This additional evidence, rather than timeline proximity alone, supports identifying the coverage gap as a contributing factor.

Corrective-action fixture: writer × reader
WriterReaderValueSupplied expectation in I1
AAnew / doneReadable
ABnew / doneReadable
BBarchivedReadable
BAarchivedNot readable: detect the defect
DCarchivedReadable: prepared recovery target

Give actions owners, deadlines and acceptance conditions

I1 proposed actions: fictional roles and deadlines
Owner and deadlineConcrete changeAcceptance condition
Application owner: before the next state-value changeAdd writer/reader fixtures to CIDetect B→A archived failure; specified compatible C/D cases pass
Release-process owner: before the next releaseBind recovery version, configuration and data conditions to the gateReject incompatible A; allow only validated compatible C
Operations owner: within three working days of reviewRecord detection, rollback and recovery-check times separatelyAvoid counting the same orders twice and retain unknowns
Review owner: at the next release decisionCompare action results with the original failure caseRequire evidence of defect detection, not merely an edited document

These are proposed actions for I1, not claims of changes implemented in a live CI or system. Overdue, unverified or harmful actions remain open or need revision; link completion to actual verification results.

Make the record useful for the next change

  1. Describe the scope, versions, failed user operations, confirmed impact and unknowns.
  2. Associate detection, containment, rollback and recovery-check times with evidence.
  3. Separate facts, investigated contributing factors and remaining hypotheses.
  4. Match missing checks to actions that can detect the original failure.
  5. Assign owners, deadlines and acceptance checks; do not close unperformed work.
  6. Review results at the next release gate and apply lessons to related procedures.

Choose an evidence-based incident review

Choose an option to see the result and explanation.

Twelve orders fail at 10:03; the same twelve fail at 10:04. Confirmed affected orders?
Read the answer and reason

Answer: Twelve; distinguish orders from attempts and people. · The supplied distinct-order count is twelve; investigate the total impact ceiling separately.

Detected at 10:03; targeted reads pass at 10:08. Was the whole incident exactly five minutes?
Read the answer and reason

Answer: No; five minutes is detection to targeted-read confirmation. · Onset is unknown, and the 10:12 recovery check is a separate milestone.

Which action is most directly verifiable against this failure?
Read the answer and reason

Answer: Add a B-write/A-read archived test and confirm it detects failure. · That directly connects the coverage gap to the original defect.

A document describing the new test exists. Close the corrective action?
Read the answer and reason

Answer: Not yet; verify execution results and detection of the original failure. · Distinguish proposed actions from implemented and verified improvements.

Check, then explain in your own words

  1. Explain the twelve confirmed affected orders and unknown scope in I1.
  2. Separate facts, hypotheses and additional investigation evidence.
  3. Identify the test that detects the B→A failure.
  4. Define action owners, deadlines, acceptance checks and follow-up.
Check your understanding: “Rollback to A succeeded, but the same twelve orders still failed.” What belongs in the review?

Record completed rollback separately from continuing business impact. Do not recount the twelve orders; retain evidence that A cannot read archived, targeted checks after C, the 10:12 reconciliation and unknowns.

A timeline should relate actions to user outcomes and evidence, not merely list operations.

Sources and conditions checked

The official references used in this lesson are collected here with their scope, conditions and corresponding sections. Independent expert review and live AWS comparison have not been performed.