INFRASTRUCTURE LEARNINGINFRASTRUCTURE LEARNING
インフラ大全インフラ大全IT INFRASTRUCTURE / LEARNING
Infrastructure learning guide
Find a lesson or lab/

Esc to close · Search terms stay on your device

LEARNING日本語

Topic lessonTopic: Monitoring and recovering event processing

Build an incident timeline from evidence, mitigation, and recovery

An incident timeline ties observations, working hypotheses, and mitigations to specific times. Practice with an image-processing backlog and decide what evidence supports recovery and what a preventive test should measure.

What you will learn

Describe the user impact, record an evidence-based timeline, and give one preventive action a measurable acceptance criterion.

Prerequisite

It helps to understand the difference between queue depth and oldest-message age.

Trace user impact through mitigation and recovery

Post-incident analysis records user impact, key times, observed evidence, and contributing conditions. Keep an unresolved explanation as a hypothesis and record the evidence that could change the decision.

A fictional image-processing backlog
A fictional image-processing backlog1: User impact — Record the observed impact — Compare signals. 2: Compare signals — Use evidence to choose an action — Mitigation. 3: Mitigation — Apply the recovery criteria — Verify recovery123User impactCompare signalsMitigationVerify recoveryA fictional image-processing backlog1: User impact — Record the observed impact — Compare signals. 2: Compare signals — Use evidence to choose an action — Mitigation. 3: Mitigation — Apply the recovery criteria — Verify recovery123User impactCompare signalsMitigationVerify recovery
  1. User impact → Compare signalsRecord the observed impact
  2. Compare signals → MitigationUse evidence to choose an action
  3. Mitigation → Verify recoveryApply the recovery criteria
User impact
Images take longer to finish
Compare signals
Intake, completions, and dependencies
Mitigation
Temporarily limit accepted work
Verify recovery
Completions recover and waits fall

Combine four observations into one timeline

Fictional timestamps use the same time zone
TimeObservationDecision to record
10:02Images take longer than usual to become availableImpact includes delayed outputs.
10:04Normal intake, fewer completions, higher DB latencyInvestigate dependency latency as a hypothesis.
10:06Intake was reduced and the backlog stopped worseningContinue observing the mitigation.
10:18Oldest age, completions, and DB latency recoveredCheck outcomes for waiting work before recording recovery.

Higher dependency latency is a clue. Improvement after reducing intake alone cannot explain why the dependency slowed down. Record the additional logs and time ranges needed for investigation.

Give the preventive action a measurable acceptance criterion

State a hypothesis you can check. For example, if dependency delays lower the completion rate, compare completions, dependency waits, and failures over the same observation window.

Assign ownership and an acceptance test. In this scenario, replay a fixed input with a slow dependency. Check that limiting intake lets priority jobs complete and avoids leaving old work behind.

Decide what the timeline supports

Choose an option to see the result and explanation.

DB latency rose as completions fell. How do you record that?
Read the answer and reason

Answer: Record the observations and test dependency latency as a hypothesis · Keep the observation and the explanation distinct.

After limiting intake, what evidence shows recovery?
Read the answer and reason

Answer: Output completions, backlog, and dependency latency · Check resumed completions and remaining work.

Which acceptance criterion makes the preventive action testable?
Read the answer and reason

Answer: Reproduce a slow dependency and check priority completions and backlog · Choose a reproducible condition and a measurable outcome.

Check, then explain in your own words

  1. Place impact, detection, mitigation, and recovery on one timeline.
  2. Distinguish observations, hypotheses, and evidence still needed.
  3. Give one action an owner and a reproducible acceptance criterion.
Check your understanding: What can you infer when the service improves after limiting intake?

Describe the observed change in completions and backlog after the mitigation.

To explain why the dependency slowed down, correlate individual processing records with evidence from the database.

Sources and conditions checked

The official references used in this lesson are collected here with their scope, conditions and corresponding sections. Independent expert review and live AWS comparison have not been performed.