OpenTelemetry Visual Handbook · lab
A timestamp is not a link
The alert fires. The error line says 12:00:02 payment
declined. You have traces, and you have logs — and the only
thing they share is the clock. In that second, thirty-two requests
were running. One of them is the one you want.
Joining by time feels good enough until you count. The number of
candidates is the number of requests in flight, so it grows with
traffic and not with how bad the error is. A trace_id on
the log line makes it one, at any load.
Break it
- Search by time at 20 req/s. Count the candidates the second returns, and read the clock.
- Go to 80 req/s. Same error, same second — several times the candidates and the minutes.
- Turn on trace_id on the log line. One candidate, at any traffic. The cost stops depending on load.
Predict before you look
Answer first, then try it in the simulation.
A service handles 80 requests a second, each lasting up to a second. An error log has a timestamp to the second and no trace_id. Roughly how many traces could it belong to?
Commit to an answer first. You cannot change it afterwards.
What people get wrong
Each row is a belief the simulation is built to take apart. The right column is the specification, with the link.
| What people believe | What the spec says |
|---|---|
| Correlation is a luxury; you can always join by timestamp. | A timestamp joins a log line to every request active in that instant. At production traffic that is dozens or hundreds of candidates, and the count grows with load, not with the problem. source |
| Better log timestamps would solve it. | Finer time narrows the window but never to one request when requests overlap. Only an identifier carried by both signals — the trace context — names the request. source |
Where this connects
Three pillars is the same blind spot on one incident; this lab is what it costs as traffic grows. Hard context and baggage is how the id gets there, and A log that knows its trace is how it gets onto the log line.
Check your understanding
The assessment for this lab has not been authored yet.