OpenTelemetry Visual Handbook · lab
Where spans wait when the backend is down, and where they die
The backend was down for twelve seconds. Someone raised
sending_queue.queue_size after the last incident so
“nothing gets lost”. This time the Collector restarted
with an out-of-memory kill, and the traces from those twelve seconds
are gone — along with the ones that were queued before.
A span can die in four places on the way out: refused by the
memory_limiter, dropped because the queue is full,
dropped because the retries gave up, or lost with the process.
Only the first one is something the application ever hears about.
Break it
- Run a few ticks, then take the backend down. The queue climbs one batch a tick, and the retries spread out: 1, 2, 4 ticks.
- Keep it down with queue_size 4. New batches are dropped as “queue full”, and the oldest give up after their retries run out.
- Switch to queue_size 15 and do it again. With the limiter, batches are refused at the edge before memory runs out. Turn the limiter off: the queue grows past the process's memory and the crash takes everything queued with it.
- Bring the backend back. The queue drains a couple of batches a tick, not all at once — recovery takes time too.
Predict before you look
Answer first, then try it in the simulation.
A Collector has no memory_limiter and a sending_queue larger than the memory it can use. The backend goes down for longer than usual. What happens to the queued spans?
Commit to an answer first. You cannot change it afterwards.
What people get wrong
Each row is a belief the simulation is built to take apart. The right column is the documentation, with the link.
| What people believe | What the docs say |
|---|---|
| A bigger sending_queue means nothing is lost. | The queue lives in process memory. Without a memory_limiter, a queue bigger than the memory you have turns an outage into an out-of-memory crash, and the in-memory queue goes with the process. source |
| Retry means the Collector keeps trying until the backend is back. | retry_on_failure backs off exponentially and gives up after max_elapsed_time. A batch older than that is dropped even if the backend returns a second later. source |
| If the Collector drops data, the application will see an error. | Only a refusal at the receiver (the memory_limiter saying no) goes back to the sender. Queue-full and retry drops happen inside the exporter, and only the Collector's own metrics count them. source |
Where this connects
The Collector pipeline is
where memory_limiter and the exporter sit, and why the
order of processors matters. Agent
or gateway is where this queue lives — on every node, or in a few
gateways — and so how much one outage costs.
Check your understanding
The assessment for this lab has not been authored yet.