Topic lessonTopic: Monitoring and recovering event processing
Read distributed traces to separate queue wait from execution time
Distributed traces follow related work across service boundaries. Use a fictional image conversion to distinguish time before a message reaches a worker from the time spent processing it.
Distinguish a trace, a measured processing segment, and the business request ID. Decide what to inspect when an observation is missing.
The example uses a queue between image intake and a Lambda worker. All example timestamps have been aligned to the same clock reference.
Follow the work and identify what each timing measures
A trace connects related observations. Supported SQS integrations can carry or link trace context to a consumer. A business request ID lets you reconcile saved outcomes independently of identifiers used by telemetry.
- Intake → QueueIdentify the same work
- Queue → ConversionMove from wait to execution
- Conversion → OutputFollow the work to its result
- Intake
- Record the business request ID
- Queue
- Record enqueue and receive times
- Conversion
- Measure the processing segment
- Output
- Reconcile the saved result
Break down a total of 500 ms
| Observation | Time after intake | Elapsed since previous observation |
|---|---|---|
| Enqueued | 20 ms | 20 ms since intake |
| Worker receives message | 380 ms | 360 ms since enqueue |
| Result stored | 500 ms | 120 ms since receive |
Waiting to receive dominates this example, so investigate it separately from function execution. If clock references and observation points differ, the timestamp difference may not be an accurate wait time.
Investigate missing segments and find other evidence
Check propagation and instrumentation. A record from a sender does not establish that the consumer emitted a segment. Check support for the integration, sampling, and collection at each end.
Measure the workload separately. A trace shows one sampled path. Use metrics with a defined population when you need the workload’s distribution.
Interpret timings and missing evidence
Choose an option to see the result and explanation.
Check, then explain in your own words
- Match the request ID to the observations.
- Distinguish enqueue, receive, and result timestamps.
- Check reporting gaps and compare trends with metrics.
Check your understanding: The segment after a queue is missing. What can you conclude?
The trace contains no observation of that segment.
Use matching logs and business results to distinguish work that did not run from work that was not collected.
Sources and conditions checked
The official references used in this lesson are collected here with their scope, conditions and corresponding sections. Independent expert review and live AWS comparison have not been performed.
- Amazon SQS and AWS X-RayChecked:
Scope and conditionsTrace context propagation and linkage across SQS.
Used in this lessonFollow the work and identify what each timing measures Break down a total of 500 ms Investigate missing segments and find other evidence
- AWS X-Ray conceptsChecked:
Scope and conditionsTraces, segments, and sampling.
Used in this lessonFollow the work and identify what each timing measures Break down a total of 500 ms Investigate missing segments and find other evidence