INFRASTRUCTURE LEARNINGINFRASTRUCTURE LEARNING
インフラ大全インフラ大全IT INFRASTRUCTURE / LEARNING
Infrastructure learning guide
Find a lesson or lab/

Esc to close · Search terms stay on your device

LEARNING日本語

Topic lessonTopic: Monitoring and recovering event processing

Read distributed traces to separate queue wait from execution time

Distributed traces follow related work across service boundaries. Use a fictional image conversion to distinguish time before a message reaches a worker from the time spent processing it.

What you will learn

Distinguish a trace, a measured processing segment, and the business request ID. Decide what to inspect when an observation is missing.

Prerequisite

The example uses a queue between image intake and a Lambda worker. All example timestamps have been aligned to the same clock reference.

Follow the work and identify what each timing measures

A trace connects related observations. Supported SQS integrations can carry or link trace context to a consumer. A business request ID lets you reconcile saved outcomes independently of identifiers used by telemetry.

Four observation points for an image conversion
Four observation points for an image conversion1: Intake — Identify the same work — Queue. 2: Queue — Move from wait to execution — Conversion. 3: Conversion — Follow the work to its result — Output123IntakeQueueConversionOutputFour observation points for an image conversion1: Intake — Identify the same work — Queue. 2: Queue — Move from wait to execution — Conversion. 3: Conversion — Follow the work to its result — Output123IntakeQueueConversionOutput
  1. Intake → QueueIdentify the same work
  2. Queue → ConversionMove from wait to execution
  3. Conversion → OutputFollow the work to its result
Intake
Record the business request ID
Queue
Record enqueue and receive times
Conversion
Measure the processing segment
Output
Reconcile the saved result

Break down a total of 500 ms

One fictional request with aligned clocks and matched observations
ObservationTime after intakeElapsed since previous observation
Enqueued20 ms20 ms since intake
Worker receives message380 ms360 ms since enqueue
Result stored500 ms120 ms since receive

Waiting to receive dominates this example, so investigate it separately from function execution. If clock references and observation points differ, the timestamp difference may not be an accurate wait time.

Investigate missing segments and find other evidence

Check propagation and instrumentation. A record from a sender does not establish that the consumer emitted a segment. Check support for the integration, sampling, and collection at each end.

Measure the workload separately. A trace shows one sampled path. Use metrics with a defined population when you need the workload’s distribution.

Interpret timings and missing evidence

Choose an option to see the result and explanation.

A message is enqueued at 20 ms and received at 380 ms. What is the interval?
Read the answer and reason

Answer: 360 ms from enqueue to receive · The aligned timestamps differ by 360 ms.

The consumer segment is missing. What should you investigate first?
Read the answer and reason

Answer: Consumer instrumentation, sampling, and the same request’s outcome · Check context linkage, instrumentation, and reporting conditions.

One trace shows a long wait. What would help describe the whole workload?
Read the answer and reason

Answer: Wait and volume metrics over the same window · Compare it with an aggregate based on a defined population.

Check, then explain in your own words

  1. Match the request ID to the observations.
  2. Distinguish enqueue, receive, and result timestamps.
  3. Check reporting gaps and compare trends with metrics.
Check your understanding: The segment after a queue is missing. What can you conclude?

The trace contains no observation of that segment.

Use matching logs and business results to distinguish work that did not run from work that was not collected.

Sources and conditions checked

The official references used in this lesson are collected here with their scope, conditions and corresponding sections. Independent expert review and live AWS comparison have not been performed.