Who owns the pipeline

There are three places a piece of telemetry can be shaped, and they are owned by three different people, on three different release cycles, with three different blast radii. Almost every argument about OpenTelemetry configuration is really an argument about which of the three you should be touching.

The three are the API, the SDK and the Collector. Getting the division right is worth more than knowing any individual setting, because the settings are searchable and the division is not.

The API owns a promise, and a no-op

The API is the surface that instrumentation writes against: start a span, record a measurement, emit a record. What it deliberately does not contain is any machinery for doing anything with the result.

That sounds like an incomplete library until you look at who calls it. A database driver, an HTTP client, a web framework — these want to emit telemetry, and they absolutely must not choose an exporter, a sampling rate or a destination on behalf of the application that depends on them. So the API is designed to be a dependency a library can take without imposing anything (source), and the application separately installs an SDK that gives those calls an effect.

Which produces the most consequential default in the whole system. If no SDK is installed, the API still works: it returns a no-op implementation, the calls succeed, and nothing happens (source). No exception, no warning, no log line. Instrumented code runs at essentially no cost and produces nothing at all.

That is the correct design — a library that crashed because the host application had not configured telemetry would be a library nobody could safely depend on — and it is also, by a wide margin, the most common reason for the sentence “my spans are not showing up”. The system is not failing. It is doing precisely what it promises when nobody has installed the part that does the work.

The SDK owns every decision

The SDK is where the decisions live, and the specification is close to being a list of them: which sampler, which processors and in what order, what limits, how identifiers are generated, what the resource is, when a flush happens (source). The metrics side is explicit that configuration must be owned by the provider (source), and the logs side says the same about its processors (source).

What is striking is the other half of that list — the part that is empty. A tracer has no configuration. You obtain one by name, it remembers that name, and it holds nothing else you can set. There is no per-tracer sampler, no per-tracer exporter, no per-tracer resource. If you want to change what happens to telemetry, there is exactly one object to change it on, and it is the provider.

That is not an omission. It is what makes the previous article’s claim about the envelope true: the resource sits above the spans in the payload because there is only one place it could have been configured. A per-tracer resource would make the nesting a lie.

It also fixes where the process boundary falls. Shutdown and flush are provider operations, and they cascade to the processors the provider owns (source). So the answer to “what happens to telemetry when the process dies” is decided entirely inside the SDK, by whether anything asked the provider to flush. Nothing downstream can help: a Collector cannot receive a batch that was still in a queue in a process that no longer exists.

The Collector owns what has already left

The Collector runs out of process. It receives, it transforms, and it exports (source), and its power is real: it can add attributes, redact them, drop whole streams, route to several destinations, convert between formats, batch, retry, and buffer to disk. All of that without touching the application, which is why platform teams like it so much and why it is usually the right place to put policy.

Its limits are equally real, and they all come from one fact: it only sees what was sent to it.

A span dropped by the sampler at creation never existed on the wire. A batch lost because a process exited without flushing was never transmitted. An attribute the instrumentation did not record cannot be added, only invented. The Collector is extremely good at reshaping telemetry and completely unable to recover telemetry, and no amount of configuration changes that.

Which gives the rule that decides most of these arguments, and it is the same one this handbook’s first article arrives at from the other direction:

Detail can be removed downstream. It can never be restored downstream.

So the decision about what to keep has to be taken as early as you can afford to take it, and the Collector is where you decide what to do with what you kept. Head sampling in the SDK and tail sampling in the Collector are not two implementations of one feature — they are decisions at two positions on a one-way street, and the second one can only choose among what the first one let through.

The asymmetry nobody expects

A last observation, because it undoes a reasonable assumption early.

The three providers are not three instances of one pattern. Traces and logs are pushed: the instrumentation hands a finished item to a processor, the processor queues it, and a timer or a full queue sends the batch on. Metrics are pulled: the SDK keeps aggregated state in memory, and a reader asks for it on its own schedule (source).

That difference is not cosmetic, and it shows up in three places you will meet in practice. The default intervals differ by more than an order of magnitude, so a metric and a trace describing the same second arrive at very different times. The flush semantics differ: forcing a flush is meaningful for a push exporter and close to meaningless for a pull one, since a pull exporter can only respond when something scrapes it. And the failure mode under pressure differs — a trace queue that fills up drops spans, while metric state that exceeds its cardinality limit collapses the excess into a single overflow series. Both are data loss, they look nothing alike, and the second one keeps the totals correct while making the breakdown useless.

The logs side has its own asymmetry: it has no sampler at all (source). What it has instead is a switch on the logger config that drops records belonging to traces that were not sampled — which means turning your trace sampling down can silently delete logs, through a decision made in a different provider. That is the kind of coupling that is obvious once drawn and nearly invisible in configuration files, and it is the reason this phase has a model instead of a table.

Where to go next: What a signal is if you want the first-principles version of the one-way street, or The envelope is the object graph for what ownership looks like in the bytes. The full path is on the handbook page.