INFRASTRUCTURE LEARNINGINFRASTRUCTURE LEARNING
インフラ大全インフラ大全IT INFRASTRUCTURE / LEARNING
Infrastructure learning guide
Find a lesson or lab/

Esc to close · Search terms stay on your device

LEARNING日本語

Topic lessonTopic: CloudWatch, CloudTrail, Config and incident investigation

AWS incident log investigation: correlate time, request IDs and evidence

Errors near the same time are not necessarily the same request or proof of a cause. Separate ingress, application, dependency and configuration-change records, then correlate justified evidence. Use a fictional report API to distinguish what a 504 and a timeout record establish from what remains unknown.

What you will learn

Align request identity and observation points, then separate recorded facts, hypotheses and the evidence still needed.

Prerequisite

An HTTP status describes a response at an observation point. A dependency is another service called by an application. This is a fixed reading exercise: no AWS connection, real logs or credentials are needed.

Start with the observer behind each record

Metrics are measurements for defined subjects and intervals; logs record events. CloudWatch helps examine them, but does not mean every needed log is automatically collected. CloudTrail account activity is also distinct from an access log of every application HTTP request.

Fictional request path; arrows do not imply automatic logging or prove a cause
Fictional request path; arrows do not imply automatic logging or prove a cause1: Viewer — Observation of the HTTP request and response — Ingress proxy. 2: Ingress proxy — Forwarding to the application in this example — Report API. 3: Report API — The application invokes a dependency — Storage API123ViewerIngress proxyReport APIStorage APIFictional request path; arrows do not imply automatic logging or prove a cause1: Viewer — Observation of the HTTP request and response — Ingress proxy. 2: Ingress proxy — Forwarding to the application in this example — Report API. 3: Report API — The application invokes a dependency — Storage API123ViewerIngress proxyReport APIStorage API
  1. Viewer → Ingress proxyObservation of the HTTP request and response
  2. Ingress proxy → Report APIForwarding to the application in this example
  3. Report API → Storage APIThe application invokes a dependency
Viewer
Observes the ingress response
Ingress proxy
Request forwarding and response
Report API
Its processing and dependency call
Storage API
Received requests and processing
Do not make one record speak for every layer
ObserverExample evidenceWhat it does not establish alone
Ingress HTTP logStatus returned and timing recorded at ingressHow time was spent inside the application
Application logRecorded operations and dependency errorsInternal state of the dependency server
Account activity eventA recorded configuration request and its outcomeWhether it caused this particular request to fail

R1: follow one request from ingress into the application

R1: fixed records for demo-r1
2026-10-24T12:00:00.000Z ingress request=demo-r1 event=received path=/reports
2026-10-24T12:00:00.010Z app     request=demo-r1 event=begin
2026-10-24T12:00:00.020Z app     request=demo-r1 event=dependency-call deadline_ms=1500
2026-10-24T12:00:01.520Z app     request=demo-r1 event=dependency-timeout
2026-10-24T12:00:01.525Z app     request=demo-r1 event=respond status=504
2026-10-24T12:00:01.530Z ingress request=demo-r1 event=respond status=504

R1 states that the application recorded a dependency timeout and returned 504, which ingress also returned. The interval from dependency-call to timeout is 1,500 milliseconds. It is an application-side observation, not proof that the dependency server spent 1,500 milliseconds processing.

Narrow the investigation without jumping to a cause
StepConclusion from R1
ObservedApplication timeout record followed by 504 returned at ingress.
Next focusDependency connection, deadlines, path and authorized dependency-side evidence.
UnknownArrival at the dependency, internal slowness or connection/path waiting.
AvoidDisabling controls, increasing deadlines or restarting everything without evidence.

Correlate request identity, time and subject

Request identifiers have issuer-specific meanings. Do not blindly equate an ALB trace_id or CloudTrail requestID with an unrelated application ID. Verify propagation or a documented mapping, and align account, Region, environment, resource, path and time.

Other records found near R1
CasePremature conclusionAppropriate check
R2: demo-r2 error near the same timeIt is R1’s dependency failure.Keep it separate unless a justified mapping exists.
R3: no matching Storage API recordThe request definitely never arrived.Check collection, gaps, retention, clocks and search criteria.
R4: traffic-rule change two minutes earlierThe change must be the cause.Test the hypothesis against affected paths and authorized before/after evidence.
R5: clock synchronization unknownA cross-host difference is exact network time.Verify time bases and distinguish trustworthy within-process measurements.

Keep an evidence trail for the investigation

  1. Fix the symptom, reproduction time, environment and expected behavior.
  2. Check collection scope, retention, clocks and identifier meaning at each observer.
  3. Order only facts that can be correlated to the same request.
  4. Separate facts, hypotheses and disconfirming or additional evidence.
  5. If a change is needed, review authority, impact and recovery, then minimize its scope.
  6. Retest expected responses and requests that should remain denied; retain unresolved questions.

An R1 note can say: “Observed: dependency-timeout for app request demo-r1, followed by 504. Unknown: Storage API arrival and internal processing. Next: verify collection scope and correlate connection/dependency evidence.” Do not rewrite it as “Cause: the database is slow.”

Choose conclusions the evidence supports

Choose an option to see the result and explanation.

What does R1 establish?
Read the answer and reason

Answer: The application recorded a dependency timeout and returned 504 · The matching demo-r1 records show this sequence; the dependency’s internal cause remains unknown.

What is the interval from dependency-call to timeout in R1?
Read the answer and reason

Answer: 1,500 ms; an application-side observation · 1.520−0.020 seconds is 1.500 seconds. It is not a measurement of dependency-server processing time.

How should you handle demo-r2 near the same time in R2?
Read the answer and reason

Answer: Keep it separate and verify a justified mapping · Verify identifier scope and propagation before correlating; do not mix it into R1 without evidence.

What follows first from no matching Storage API record in R3?
Read the answer and reason

Answer: Arrival is unverified; check collection, clocks and queries · Check what should be recorded before interpreting gaps or scope mismatches.

What does a traffic-rule change two minutes earlier establish in R4?
Read the answer and reason

Answer: A candidate cause; examine the path and before/after evidence · Timing is a lead; attributing this request’s failure requires evidence of scope and effect.

Check, then explain in your own words

  1. Identify R1’s request ID and application-side 1,500 ms interval.
  2. Explain what remains unproved in R2–R5.
  3. Write an investigation note separating facts, hypotheses and next observations.
Check your understanding: What should you separate before declaring an incident cause?

Separate observed facts, hypotheses and evidence still needed.

Similar timestamps, statuses or earlier changes do not establish identity or causation. Verify correlation, observation points and collection scope.

Sources and conditions checked

The official references used in this lesson are collected here with their scope, conditions and corresponding sections. Independent expert review and live AWS comparison have not been performed.