OpenTelemetry Visual Handbook · lab

You added a filter, and the data kept arriving

A Collector config reads like a set of boxes you plug together, and that reading survives right up to the first incident. Two things in it are not what they look like: processors: is an ordered list, not a set — the order is the behaviour — and a receiver named in two pipelines is one receiver that copies, not two.

Below, one app exports spans into one otlp receiver, and that receiver feeds two pipelines. Watch the drawing first: two lines leave the same port and never touch again. Then turn on the filter and read the two counters under the stage. One of them does not move — and clicking any item on a wire follows it, so you can watch a single span take both branches at once.

Break it

Three moves, and each one is a ticket someone has already written.

  1. Turn the filter on. The stdout counter halves and the backend counter does not move at all. The filter is real, it works, and it is in the other pipeline — so the exporter you were not watching keeps shipping every health check you thought you had stopped paying for.
  2. Drop spans per second to 1 and raise send_batch_size to 10. The batch never fills, and telemetry still leaves — look at closed by timeout climbing while closed by size stays at zero. This is the half of batch that the phrase “it sends when it is full” leaves out, and without it a low-traffic service would hold its spans until a batch filled up, which can be never.
  3. Raise traffic to 5 and drop memory_limiter to 2. Loss appears in dropped, attributed to memory_limiter — and that is the point of it being a port and not a silence. With the filter also on, the same box shows two numbers from two causes: one is capacity failing, the other is configuration working. Summing them would erase the only difference that matters when you are paged.

Predict before you look

Answer first, then run it. Predicting and being wrong is what makes the answer stick.

Your config declares one otlp receiver in two pipelines: traces (batch → otlp exporter) and traces/audit (filter → debug exporter). The filter drops /health. What reaches the otlp exporter?

Commit to an answer first. You cannot change it afterwards.

Configure it

Two lines from the collector.yaml that actually runs in labs/collector-pipeline/. The correct block is lifted out of that file, so it cannot drift from the config that boots. The first one is the thesis the stage above cannot show you: the order of processors: is the behaviour, and changing the order changes the drawing — which a skeleton cannot do with a slider.

Both wrong answers are valid YAML, boot cleanly, and export telemetry. That is the whole difficulty of a Collector: nothing type-checks a config, and nothing fails.

The pipeline that feeds your backend. `memory_limiter` and `batch` are both declared above, with the settings you want — this line decides only which runs first.

What goes in `processors:`?

traces:
  receivers: [otlp]
 
exporters: [otlp/backend]

Pick one. You cannot change it afterwards — and that is the point.

labs/collector-pipeline/collector.yaml · OpenTelemetry Collector 0.160.0

The batch processor for a service whose traffic varies: busy at midday, nearly idle at night. Spans still have to reach the backend at night.

What closes a batch?

batch:
  # Os mesmos dois números do lab da tela. O `timeout` é a metade que a frase
  # "ele manda quando enche" esquece: sem ele, tráfego baixo guardaria a
  # telemetria até o lote encher, o que pode ser nunca.
 

Pick one. You cannot change it afterwards — and that is the point.

labs/collector-pipeline/collector.yaml · OpenTelemetry Collector 0.160.0

What people get wrong

Every row below is a belief worth naming, because the drawing above is arranged to take it apart. The right column is the documentation, with the link.

What people believe What the docs say
I added a filter processor, so that data stops leaving the Collector. A processor belongs to ONE pipeline. A receiver that feeds two pipelines hands each of them its own copy, so a filter in one changes nothing in the other — and the exporter you were not looking at keeps exporting everything. source
Two pipelines fed by the same receiver run independently of each other. They share the receiver, and the hand-off is a synchronous call: the receiver passes the data to a fan-out consumer, which calls the first processor of each pipeline. If one of them blocks, the others do not get the data and the receiver stops taking in new data — so a slow exporter in the pipeline you do not care about is back-pressure on the one you do. source
`processors:` is the set of processors the pipeline uses. It is an ordered list, and the order is the behaviour: data goes through it top to bottom. `memory_limiter` first and `batch` after it is not style — it decides whether the limiter sees single items or whole batches. source
The batch processor sends when it is full. It sends when it is full **or** when the timeout expires, whichever comes first. Without the timeout, a service with low traffic would hold its telemetry until the batch filled up — which can be never. source
A processor that drops data hides it — the number only shows up in the backend that stopped receiving. The Collector counts what it refuses. `otelcol_processor_refused_spans` and the dropped counters exist precisely so that loss inside the Collector is readable where it happened, instead of being inferred from an absence downstream. source

Run it for real

labs/collector-pipeline/ is this drawing as a config that runs: one app, one otlp receiver, and the same two pipelines. The collector.yaml there is the counterpart of the stage above — same receiver in both pipelines, same filter in only one — so the claim this lab makes is checkable against the binary and not only against the simulation.

Check your understanding

The assessment for this lab has not been authored yet.

Continue to the handbook assessment →