OpenTelemetry Visual Handbook · phase 03 · lab
Where does the
Collector live?
The same six Pods, two requests, and four ways to deploy the Collector. Each one moves spans along a different path. Each one also changes three things: what the backend sees, what you configure, and whether any single Collector ever holds a whole trace.
The paths, counts and colours are computed from a deployment model. Change the deployment and the film is rebuilt.
Packet flow
- T1 spans
- T2 spans
| No. | Step | Source | Destination | Protocol | Size | Info |
|---|---|---|---|---|---|---|
| 1 | 02 | frontend-1 | gateway-a | OTLP/gRPC | — | T1 frontend span · connection pinned by the Service |
| 2 | 02 | cart-2 | gateway-b | OTLP/gRPC | — | T1 cart span · connection pinned by the Service |
| 3 | 02 | payment-1 | gateway-b | OTLP/gRPC | — | T1 payment span · connection pinned by the Service |
| 4 | 02 | frontend-2 | gateway-b | OTLP/gRPC | — | T2 frontend span · connection pinned by the Service |
| 5 | 02 | cart-1 | gateway-a | OTLP/gRPC | — | T2 cart span · connection pinned by the Service |
| 6 | 02 | payment-2 | gateway-a | OTLP/gRPC | — | T2 payment span · connection pinned by the Service |
| 7 | 04 | gateway-a | backend | OTLP → backend | — | T1 frontend span |
| 8 | 04 | gateway-b | backend | OTLP → backend | — | T1 cart span |
| 9 | 04 | gateway-b | backend | OTLP → backend | — | T1 payment span |
| 10 | 04 | gateway-b | backend | OTLP → backend | — | T2 frontend span |
| 11 | 04 | gateway-a | backend | OTLP → backend | — | T2 cart span |
| 12 | 04 | gateway-a | backend | OTLP → backend | — | T2 payment span |
Break it, one choice at a time
- Start with “Gateway”. Two backend connections instead of six, and credentials in one place. Then reach “Who holds a whole trace?”. Every replica shows fragments.
- Switch to “Agent + gateway”. Traffic stays on the node first. The traces are still split, because each agent is pinned to one replica.
- Turn on “Route by trace ID”. The agent's
loadbalancingexporter hashes the trace ID. Each replica now holds one whole trace, and a tail sampling policy can see errors and latency before it decides. - Try “SDK → backend”. It has the fewest moving parts and the most connections. No Collector means nowhere to batch, retry, redact or tail-sample.
Why the IP matters
k8sattributes identifies the sending Pod by the connection's source IP. A gateway behind agents only sees the agents' IPs. With agents, enrich in the agent, or pass the Pod IP along. Collector deployment patterns → · loadbalancing exporter →
The boundary of this experiment
The cluster is fixed: two nodes, three services and two requests, with each request crossing both nodes. Connection pinning is one possible arrangement, fixed so that no trace lands whole by chance. Trace-ID routing assigns T1 and T2 to different replicas. It does not model retries, queue sizes, autoscaling, OTLP/HTTP connection reuse or the sampling policy itself.
A gateway Deployment has two replicas behind a Kubernetes Service. Can a tail sampler on each replica see whole traces?
Commit to an answer first. You cannot change it afterwards.