OpenTelemetry Visual Handbook · lab
Three recorders, one event, three different memories
Every introduction presents traces, metrics and logs as three kinds of data. They are not. They are three decisions about what to throw away — taken on the same event, at the moment it is written — and what one of them discards, no backend can hand back to you later.
Below, one service handles requests. Three recorders watch the exact same ones — and clicking any item on a wire follows it: the panel shows what each recorder is about to keep of that one request, and the fields that vanish on the way in are the ones you will not be able to ask for later. Let it run, then ask the question at the bottom: which requests took longer than 250 ms? One of the three answers with a list. One answers with a number, at best. One answers only if the person who wrote the code happened to mention it.
Break it
Three moves, and each one is a thing that happens to real systems.
- Turn off “the code logs every request”. The log keeps only failures, and the counter of requests it said nothing about starts climbing. That number is the part of your traffic no log will ever describe — and the decision was taken in the editor, not in configuration.
- Add
user.idto the metric attributes. Watch the series count climb: one attribute is not one more column, it is one more series per distinct value. Then drop the cardinality limit to 6 and watch the overflow row appear — nothing is lost and nothing is double-counted, and the breakdown you wanted is gone. - Ask for 300 ms instead of 250. The metric stops being able to answer even how many: 300 falls inside a bucket, and a bucket is a count, not a list. The buckets were chosen before the first measurement — that is what makes the loss irreversible.
Predict before you look
Answer first, then run it. Predicting and being wrong is what makes the answer stick.
A histogram says 12 requests landed above 500 ms. You need to know which ones. Where do you get them?
Commit to an answer first. You cannot change it afterwards.
Instrument it
Two decisions from the file that actually runs in
labs/three-pillars/. The correct block is lifted out of
that file, so it cannot drift from the code that compiles. Both wrong
answers err on the side of looking more careful — more attributes, a
clearer name — which is how they happen.
One request has just been handled. The span for it is open and already carries the route and the duration. Now the same request has to reach the histogram, and a histogram records a number against a set of attributes.
Which attributes belong on the measurement?
Pick one. You cannot change it afterwards — and that is the point.
labs/three-pillars/app/src/main/java/pillars/Pillars.java · OpenTelemetry Java 1.65.0
Start-up, where the instrument is built once. The backend that will read this has never seen this service, and nobody will tell it what the numbers mean.
What makes this histogram readable by something that does not know your code?
DoubleHistogram duracao =
otel.getMeter("checkout.http")
.build();Pick one. You cannot change it afterwards — and that is the point.
labs/three-pillars/app/src/main/java/pillars/Pillars.java · OpenTelemetry Java 1.65.0
Read the theory
What a signal is argues the same thing in prose, from first principles — and The seam between the signals picks up where this lab leaves off: once you know each signal has its own blind spot, the work is the crossing between them.
Check your understanding
The assessment for this lab has not been authored yet.