OpenTelemetry Visual Handbook · lab
A counter is not a number that goes up
Your dashboard shows −9 requests per second for ten seconds after a deploy. Nothing is broken: the service restarted, its cumulative counter started again from zero with a new start time, and the backend subtracted the old point from the new one.
The SDK never sends measurements. It folds them into an aggregator in process memory, and a reader pulls one point per interval. Each point says how many since when — and the “since when” is the part people forget. Watch it in the showcase, then break it in the simulation.
Break it
- Record a few requests, collect, record more, collect. In cumulative the second point is bigger than the first and keeps the same start time; switch to delta and every point carries its own start time and only its own requests.
- Now restart the process between two collects. In cumulative the next point is smaller, with a new start time, and the backend reads a negative number. Turn on backend honours start time and the reading is right again — that check is what a correct backend does.
- Restart with requests not yet collected. They are gone in both temporalities. The verdict line counts them: the aggregator lived in memory, and so did they.
- Record one fast and one slow request, and read the buckets. The histogram kept two counts and a sum. Which request was slow is not in the point — that is a trace’s job.
Predict before you look
Answer first, then try it in the simulation.
A service exports a cumulative request counter every 10 s. The points read 6, then 11. The process restarts, and the next point reads 2. How many requests does a backend that just subtracts consecutive points report for that interval?
Commit to an answer first. You cannot change it afterwards.
What people get wrong
Each row is a belief the simulation above is built to take apart. The right column is the specification, with the link.
| What people believe | What the spec says |
|---|---|
| A counter is a number that goes up, and the backend stores that number. | What leaves the SDK is a data point with a temporality and a start time. A cumulative point means 'this many since start time'; a delta point means 'this many since the last collect'. The same requests produce different numbers on the wire. source |
| A cumulative series only ever goes up. | It resets whenever the process restarts, and the reset is announced by a new start time. A backend that subtracts across a changed start time computes a negative rate. source |
| Delta is safer, so nothing is lost when a process dies. | Either way, the aggregator lives in process memory. Whatever was measured after the last collect and before the crash was never exported, in cumulative or in delta. source |
| A histogram stores every latency it saw. | It stores a count per bucket, a sum and a count — the latencies themselves are gone. Bucket upper bounds are inclusive, so 50 ms lands in the bucket that ends at 50. source |
Where this connects
Three pillars is where a
metric first throws the individual request away; this lab is what is
left. Cardinality and the
bill multiplies these points by every attribute set, and
Reading an OTLP
payload is the envelope they travel in — the same resource and
scope wrapping as spans, with startTimeUnixNano on every
point.
Check your understanding
The assessment for this lab has not been authored yet.