OpenTelemetry Visual Handbook · lab
Where k8s.pod.name comes from, and why it went missing
The Pod dashboard has CPU per Pod. The traces have no Pod at all —
ever since k8sattributes moved to the gateway “to
keep the agents light”. Nothing errored; the lookup just stopped
finding anyone.
The agent on each node collects two kinds of signal that already know
their names: hostmetrics reads the node, and
kubeletstats asks the kubelet about its Pods. Spans are
different: they arrive from an IP, and
k8sattributes asks the API server which Pod owns it. The
answer depends on which IP it saw.
Break it
- Run k8sattributes in the gateway. Every span asks about 10.0.1.4 or 10.0.1.5 — the agents' addresses — and no Pod has them. Zero spans can be grouped by Pod.
- Move it to the agent. The connection now comes from the Pod, and every span gets its name, namespace, deployment and node.
- Back to the gateway, and associate by
k8s.pod.ip. The SDK put the Pod's IP on the resource, so the gateway can find the Pod without seeing its connection. - Look at the right-hand list in any setting. Host and kubelet metrics always carry their names: they never needed a lookup.
Predict before you look
Answer first, then try it in the simulation.
Apps send spans to an agent Collector on their node, which forwards to a gateway. k8sattributes runs only in the gateway, with the default association by connection IP. What do the spans carry?
Commit to an answer first. You cannot change it afterwards.
What people get wrong
Each row is a belief the simulation is built to take apart. The right column is the documentation, with the link.
| What people believe | What the docs say |
|---|---|
| k8sattributes knows which Pod sent the data. | It looks the sender up, by default by the IP of the incoming connection. Behind another Collector, that IP is the forwarding Collector's, not the Pod's, and the lookup finds nothing. source |
| Pod metrics and Pod traces always line up. | kubeletstats metrics come with pod names from the kubelet. Spans only get them if k8sattributes (or the SDK resource detector) adds them; without that, the two cannot be joined by Pod. source |
| Enrich in the gateway, it is the one place that sees everything. | Either run k8sattributes in the agent, where the connection is the Pod's, or associate by a resource attribute such as k8s.pod.ip that the SDK sets — then any Collector can find the Pod. source |
Where this connects
Agent or gateway is the deployment decision this lab depends on — the source IP is one more thing that changes between them. Three pillars is why joining Pod metrics to Pod traces matters during an incident.
Check your understanding
The assessment for this lab has not been authored yet.