Topic lessonTopic: CloudWatch, CloudTrail, Config and incident investigation
AWS incident log investigation: correlate time, request IDs and evidence
Errors near the same time are not necessarily the same request or proof of a cause. Separate ingress, application, dependency and configuration-change records, then correlate justified evidence. Use a fictional report API to distinguish what a 504 and a timeout record establish from what remains unknown.
Align request identity and observation points, then separate recorded facts, hypotheses and the evidence still needed.
An HTTP status describes a response at an observation point. A dependency is another service called by an application. This is a fixed reading exercise: no AWS connection, real logs or credentials are needed.
Start with the observer behind each record
Metrics are measurements for defined subjects and intervals; logs record events. CloudWatch helps examine them, but does not mean every needed log is automatically collected. CloudTrail account activity is also distinct from an access log of every application HTTP request.
- Viewer → Ingress proxyObservation of the HTTP request and response
- Ingress proxy → Report APIForwarding to the application in this example
- Report API → Storage APIThe application invokes a dependency
- Viewer
- Observes the ingress response
- Ingress proxy
- Request forwarding and response
- Report API
- Its processing and dependency call
- Storage API
- Received requests and processing
| Observer | Example evidence | What it does not establish alone |
|---|---|---|
| Ingress HTTP log | Status returned and timing recorded at ingress | How time was spent inside the application |
| Application log | Recorded operations and dependency errors | Internal state of the dependency server |
| Account activity event | A recorded configuration request and its outcome | Whether it caused this particular request to fail |
R1: follow one request from ingress into the application
2026-10-24T12:00:00.000Z ingress request=demo-r1 event=received path=/reports
2026-10-24T12:00:00.010Z app request=demo-r1 event=begin
2026-10-24T12:00:00.020Z app request=demo-r1 event=dependency-call deadline_ms=1500
2026-10-24T12:00:01.520Z app request=demo-r1 event=dependency-timeout
2026-10-24T12:00:01.525Z app request=demo-r1 event=respond status=504
2026-10-24T12:00:01.530Z ingress request=demo-r1 event=respond status=504R1 states that the application recorded a dependency timeout and returned 504, which ingress also returned. The interval from dependency-call to timeout is 1,500 milliseconds. It is an application-side observation, not proof that the dependency server spent 1,500 milliseconds processing.
| Step | Conclusion from R1 |
|---|---|
| Observed | Application timeout record followed by 504 returned at ingress. |
| Next focus | Dependency connection, deadlines, path and authorized dependency-side evidence. |
| Unknown | Arrival at the dependency, internal slowness or connection/path waiting. |
| Avoid | Disabling controls, increasing deadlines or restarting everything without evidence. |
Correlate request identity, time and subject
Request identifiers have issuer-specific meanings. Do not blindly equate an ALB trace_id or CloudTrail requestID with an unrelated application ID. Verify propagation or a documented mapping, and align account, Region, environment, resource, path and time.
| Case | Premature conclusion | Appropriate check |
|---|---|---|
| R2: demo-r2 error near the same time | It is R1’s dependency failure. | Keep it separate unless a justified mapping exists. |
| R3: no matching Storage API record | The request definitely never arrived. | Check collection, gaps, retention, clocks and search criteria. |
| R4: traffic-rule change two minutes earlier | The change must be the cause. | Test the hypothesis against affected paths and authorized before/after evidence. |
| R5: clock synchronization unknown | A cross-host difference is exact network time. | Verify time bases and distinguish trustworthy within-process measurements. |
Keep an evidence trail for the investigation
- Fix the symptom, reproduction time, environment and expected behavior.
- Check collection scope, retention, clocks and identifier meaning at each observer.
- Order only facts that can be correlated to the same request.
- Separate facts, hypotheses and disconfirming or additional evidence.
- If a change is needed, review authority, impact and recovery, then minimize its scope.
- Retest expected responses and requests that should remain denied; retain unresolved questions.
An R1 note can say: “Observed: dependency-timeout for app request demo-r1, followed by 504. Unknown: Storage API arrival and internal processing. Next: verify collection scope and correlate connection/dependency evidence.” Do not rewrite it as “Cause: the database is slow.”
Choose conclusions the evidence supports
Choose an option to see the result and explanation.
Check, then explain in your own words
- Identify R1’s request ID and application-side 1,500 ms interval.
- Explain what remains unproved in R2–R5.
- Write an investigation note separating facts, hypotheses and next observations.
Check your understanding: What should you separate before declaring an incident cause?
Separate observed facts, hypotheses and evidence still needed.
Similar timestamps, statuses or earlier changes do not establish identity or causation. Verify correlation, observation points and collection scope.
Sources and conditions checked
The official references used in this lesson are collected here with their scope, conditions and corresponding sections. Independent expert review and live AWS comparison have not been performed.
- Amazon CloudWatch overviewChecked:
Scope and conditionsOperational visibility through metrics, logs and related features. Documentation review, not live AWS verification.
Used in this lessonStart with the observer behind each record
- CloudWatch Logs overviewChecked:
Scope and conditionsLog collection, search and retention; distinct from metrics or causal proof. Documentation review, not live AWS verification.
Used in this lessonStart with the observer behind each record
- CloudTrail eventsChecked:
Scope and conditionsEvent categories and collection conditions; records are not a complete ordered call trace. Documentation review, not live AWS verification.
Used in this lessonStart with the observer behind each record
Scope and conditionsSeparate ingress/target status, timing and trace identifiers. Best-effort logs are not complete request accounting. Documentation review, not live AWS verification.
Used in this lessonStart with the observer behind each record Correlate request identity, time and subject Choose conclusions the evidence supports
- CloudTrail event record fieldsChecked:
Scope and conditionsMeaning of eventTime, eventName and requestID; do not assume one automatic cross-layer identifier. Documentation review, not live AWS verification.
Used in this lessonStart with the observer behind each record Correlate request identity, time and subject Choose conclusions the evidence supports
- CloudTrail event historyChecked:
Scope and conditionsPast 90 days of management events in an account/Region; excludes data and other event categories. Documentation review, not live AWS verification.
Used in this lessonCorrelate request identity, time and subject