I’ve worked in the observability industry for a while and set up a lot of collectors for customers. Wanted to put together an end to end writeup covering the things most blogs skip when monitoring a Kubernetes cluster with OTel.

Covers the full agent + gateway pattern. Agent runs as a DaemonSet for node local stuff (container logs, kubelet stats, hostmetrics, OTLP from local pods). Gateway is a Deployment for cluster-wide telemetry(k8s_events, k8s_cluster, and the only connection to your backend).

Goes through every receiver and processor on both sides, the right processor order on the agent, the k8sattributes pod_association gotcha that breaks pod lookups, and the service.name fallback chain from the OTel K8s spec.

I didn’t cover auto-instrumentation in this blog. I focused mainly on the OTel collector setup

Curious to see how others are monitoring their Kubernetes clusters