OpenTelemetry Visual Handbook · lab
The trace you needed was the one you threw away
Sampling is usually discussed as a percentage, which hides the only thing that matters about it: when the decision is taken. A head sampler decides at the root span, before the request has done anything — so it cannot keep “the traces that failed”, because nothing has failed yet. A tail sampler decides on the finished trace and can, at the price of holding every span of every trace in memory until it decides.
Below, each trace has a few spans across two services, and the error is always in the last one. Turn the head sampler up and watch the counter of error traces cut climb: those are gone, and no tail policy can bring them back, because they never arrived. Then turn the load balancer's trace routing off.
Break it
Three moves. The third one is a production incident that reports nothing.
- Head sampler to 1 in 10, tail keeping everything. The cut before the error existed counter climbs, and error traces · whole stops growing. This is the whole argument against head sampling as a reliability tool: the decision could not depend on the error, so what you keep is a slice of traffic and not a slice of each category. A failure in 1 in 500 requests, sampled at 10%, is a failure you meet once in 5 000.
- Head sampler back to 1 in 1, tail keeping 1 in 10 of the rest. Now every error trace survives — kept · error equals the number that failed — and the boring traffic is thinned out. Then raise the decision window and watch holding in both gateways: that is memory, it belongs to traces that are about to be discarded, and it is the price of deciding late.
- Turn off
load balancer routes by traceID. Nothing looks broken. The gateways keep deciding, the backend keeps filling, and arrived broken starts climbing: traces that are in the backend without the span that failed. Each gateway decided correctly on the fragment it received, and the fragment was not a trace. In production nothing reports this, because no backend knows how many spans a trace had.
Predict before you look
Answer first, then run it. Predicting and being wrong is what makes the answer stick.
Your SDKs sample at 10% (head). Your Collector runs tail sampling with a policy that keeps every trace containing an error. What fraction of error traces reaches the backend?
Commit to an answer first. You cannot change it afterwards.
Configure it
Three lines from labs/head-vs-tail-sampling/, which is
this model as something that boots. The correct block of each one is
lifted out of the file that runs, so it cannot drift from the config.
They are in three different files, and that is the point rather
than variety: the sampling decision is taken in three places owned by
three different people. The SDK sampler belongs to whoever writes the
application, routing_key to whoever operates the cluster,
and the policies to whoever looks after observability. None of the
three sees the other two files, and each of them can, alone, delete the
trace of the error.
Your Collector runs tail sampling with a policy that keeps every trace containing an error. This is the SDK in the application, and the line that decides what the Collector gets to see.
What sampler belongs here?
// O amostrador do SDK quando o Collector amostra na cauda. Dois
// amostradores em série se MULTIPLICAM, e o primeiro é o que ainda
// não sabe nada — então o que se quer dele aqui é que ele não corte.
// `parentBased` é o que faz a decisão da raiz valer para o trace
// inteiro, em vez de cada span decidir por si e produzir trace
// furado.
Pick one. You cannot change it afterwards — and that is the point.
labs/head-vs-tail-sampling/app/src/main/java/sampling/App.java · OpenTelemetry Java 1.65.0
The Collector in front of two tail-sampling gateways. It samples nothing — it only decides which instance each span goes to.
What goes in the exporter's routing key?
Pick one. You cannot change it afterwards — and that is the point.
labs/head-vs-tail-sampling/gateway.yaml · OpenTelemetry Collector 0.160.0
The tail sampling processor on both gateways. You are paying for the decision window already — this block is what you get for it.
What goes in `policies:`?
# As políticas são OR: o trace é mantido se QUALQUER uma casar. Então a
# primeira guarda o que falhou, e a segunda é a linha de base do tráfego
# normal — sem ela, o backend não teria com o que comparar o erro.
Pick one. You cannot change it afterwards — and that is the point.
labs/head-vs-tail-sampling/tail.yaml · OpenTelemetry Collector 0.160.0
What people get wrong
Every row below is a belief worth naming, because the model above is arranged to take it apart. The right column is the documentation, with the link.
| What people believe | What the docs say |
|---|---|
| Sampling at 10% keeps 10% of my errors — enough to see them. | Head sampling decides at the root span, before the request has done anything. The decision cannot depend on an error that has not happened yet, so what you keep is a slice of traffic and not a slice of each category: a failure in 1 in 500 requests, sampled at 10%, is a failure you see 1 time in 5 000. source |
| The sampling decision is taken by each service, so I can set it per service. | The default sampler is parent-based: a service that receives a sampled parent keeps its spans, and one that receives an unsampled parent drops them. The decision is made once, at the root, and travels in the `sampled` flag of the W3C `traceparent` header — which is why turning sampling up in a downstream service changes nothing. source |
| Tail sampling is head sampling with better rules. | It is a different place and a different cost. Deciding on the finished trace means holding every span of every trace in memory until the decision window closes — so the Collector pays memory and latency for the traces it is going to throw away, and the window is a bet on how long a trace takes. source |
| To scale tail sampling, add gateway replicas behind the load balancer. | The tail sampling processor can only decide correctly if **all** spans of a trace reach the same Collector instance. A standard Kubernetes Service balances connections and not requests, so spans of one trace land on different replicas — each replica then decides on a fraction of the trace, and the output still looks like sampling. source |
| A trace that arrives incomplete shows up as incomplete. | Nothing downstream knows how many spans the trace had. The backend renders what it received, and a trace missing the span that failed looks exactly like a trace that succeeded — which is the one defect in this lab that no counter in production would report to you. source |
What this model does not claim
Two simplifications, declared rather than hidden. First, the sampling here is deterministic: the decision is a hash of the trace id, reproducible across runs, so two regimes can be compared instead of two draws. Real head sampling hashes the trace id too — that is why its rate is approximate — but it is a rate and not a schedule.
Second, all spans of a trace leave the application in the same tick. In a real system they are produced over the life of the request, which is the actual reason a tail sampler needs a decision window at all. The window here is real and it is a parameter, but what it waits for is the model's wiring and not the duration of a request.
Check your understanding
The assessment for this lab has not been authored yet.