Topic lessonTopic: Monitoring and recovering event processing
Build an incident timeline from evidence, mitigation, and recovery
An incident timeline ties observations, working hypotheses, and mitigations to specific times. Practice with an image-processing backlog and decide what evidence supports recovery and what a preventive test should measure.
Describe the user impact, record an evidence-based timeline, and give one preventive action a measurable acceptance criterion.
It helps to understand the difference between queue depth and oldest-message age.
Trace user impact through mitigation and recovery
Post-incident analysis records user impact, key times, observed evidence, and contributing conditions. Keep an unresolved explanation as a hypothesis and record the evidence that could change the decision.
- User impact → Compare signalsRecord the observed impact
- Compare signals → MitigationUse evidence to choose an action
- Mitigation → Verify recoveryApply the recovery criteria
- User impact
- Images take longer to finish
- Compare signals
- Intake, completions, and dependencies
- Mitigation
- Temporarily limit accepted work
- Verify recovery
- Completions recover and waits fall
Combine four observations into one timeline
| Time | Observation | Decision to record |
|---|---|---|
| 10:02 | Images take longer than usual to become available | Impact includes delayed outputs. |
| 10:04 | Normal intake, fewer completions, higher DB latency | Investigate dependency latency as a hypothesis. |
| 10:06 | Intake was reduced and the backlog stopped worsening | Continue observing the mitigation. |
| 10:18 | Oldest age, completions, and DB latency recovered | Check outcomes for waiting work before recording recovery. |
Higher dependency latency is a clue. Improvement after reducing intake alone cannot explain why the dependency slowed down. Record the additional logs and time ranges needed for investigation.
Give the preventive action a measurable acceptance criterion
State a hypothesis you can check. For example, if dependency delays lower the completion rate, compare completions, dependency waits, and failures over the same observation window.
Assign ownership and an acceptance test. In this scenario, replay a fixed input with a slow dependency. Check that limiting intake lets priority jobs complete and avoids leaving old work behind.
Decide what the timeline supports
Choose an option to see the result and explanation.
Check, then explain in your own words
- Place impact, detection, mitigation, and recovery on one timeline.
- Distinguish observations, hypotheses, and evidence still needed.
- Give one action an owner and a reproducible acceptance criterion.
Check your understanding: What can you infer when the service improves after limiting intake?
Describe the observed change in completions and backlog after the mitigation.
To explain why the dependency slowed down, correlate individual processing records with evidence from the database.
Sources and conditions checked
The official references used in this lesson are collected here with their scope, conditions and corresponding sections. Independent expert review and live AWS comparison have not been performed.
- Perform post-incident analysisChecked:
Scope and conditionsTimeline, impact, contributing factors, and prevention.
Used in this lessonTrace user impact through mitigation and recovery Combine four observations into one timeline Give the preventive action a measurable acceptance criterion