Kubernetes Cluster Observability

Summary

This blueprint outlines a reference for Platform Engineering and SRE teams responsible for providing observability infrastructure in Kubernetes clusters. It covers two verticals: resource usage and operational state of workloads, and health of Kubernetes critical components (DNS, networking, ingress).

By implementing the patterns in this blueprint, organizations can expect to achieve:

• Out-of-the-box visibility into workload resource usage, OOM events, probe failures, and pod lifecycle state without application-side changes.
• Path to reliable monitoring of Kubernetes critical components (CoreDNS, CNI plugins, Ingress controllers, KEDA, etc)
• Uniform metadata enrichment using OTel semantic conventions, enabling correlated investigation across metrics, logs, and traces.
• Self-monitoring of the telemetry collection infrastructure so silent data gaps are detected before they affect incident response.

Background

Kubernetes clusters host two classes of observable entities:

1. Workloads are application containers and the Kubernetes primitives (Statefulset, Deployment, Daemonset, ReplicaSet, etc) that manage them. They emit signals using OTel SDKs, but their resource utilization and operational state (CPU throttling, OOM kills, pod phase, probe results) are only visible through Kubernetes-specific APIs. Additionally, some of those components are managed in the node level, like pods and containers, while others are managed in the cluster/control-plane level like ReplicaSet states, HPAs and so on.
2. Critical infrastructure components include CoreDNS, CNI plugins, Ingress controllers, volume subsystems, KEDA, and similar platform services. They are (generally) platform-owned, expose Prometheus metrics natively, and require dedicated scrape configurations.

This blueprint focuses on what to collect and how to label it. Collector pipeline topology is referenced only when needed to observe Kubernetes-specific components; recommendations for Collector topologies for your own telemetry are documented separately in the Managed Telemetry Platforms Blueprint.

Common challenges

1. Workload telemetry is incomplete

CPU throttling, OOM kills, pod phase transitions, and probe failures are not emitted by application code. They are only accessible through the Kubernetes API server, and platform teams must explicitly collect them.

This leads to:

• Invisible resource pressure: CPU throttling at the cgroup level surfaces only as increased latency, with no infrastructure attribution.
• OOM kills appear as application crashes: Without a correlated OOM signal, operators cannot distinguish a memory misconfiguration from an application bug.
• Throttled versus OOM is indistinguishable: Both cause pod restarts. Without container restart and last-terminated-reason signals (k8s.container.restarts from the API server) correlated with kubelet CPU/memory metrics, there is no basis for remediation.

2. Metadata is inconsistent across signals and layers

Each scraper attaches different label schemas (pod_name, pod, kubernetes_namespace, namespace). OTel semantic convention attributes (k8s.pod.name, k8s.namespace.name) are not applied automatically.

This leads to:

• Disjointed infra telemetry from app telemetry: Cognitive load on operators increases when troubleshooting if a certain issue is caused by the app/container or infra/resources
• Alert rules break silently on scraper changes: An alert written against pod_name stops matching when a new scraper uses pod.
• Organizational context is absent: Team ownership, environment, and tier labels from pod annotations are rarely in telemetry, making alert routing and cost attribution manual.

3. Critical cluster components are not observable by default

CoreDNS, CNI plugins, Ingress controllers, KEDA, cert-manager and many others each expose metrics with no standard discovery mechanism.

Some examples of what this leads to:

• DNS latency spikes look like application problems: CoreDNS slowdowns appear as upstream timeouts, indistinguishable from a slow downstream service without a DNS-layer metric.
• CNI packet drops are unattributed: Packets dropped at the network policy layer surface as intermittent pod connectivity failures with no network-layer attribution.
• KEDA failures leave workloads silently under-provisioned: A scaler that cannot reach its source metric stops autoscaling without producing any user-visible error.

4. Telemetry collection infrastructure has no self-monitoring

Silent gaps in Collector and SDK pipelines — crashed scrapers, export failures, and data drops under backpressure — are covered in Challenge 5 of the Managed Telemetry Platforms for Kubernetes Workloads blueprint.

General guidelines

1. Use OTel native receivers to collect workload and infrastructure signals

Challenges addressed: 1, 2

This blueprint recommends the usage of OTel native receivers:

Outcomes:

• Complete workload-level resource coverage (throttling, OOM, pod phase, probe failures) without any application code changes.

2. Apply uniform metadata enrichment using OpenTelemetry semantic conventions

Challenges addressed: 2

Telemetry that reaches the backend should carry consistent OTel Kubernetes resource attributes so signals can join without remapping. How that metadata is obtained depends on the source:

• OTel-native Kubernetes receivers (k8s_cluster, kubeletstats, k8s_objects, k8s_events) already emit the core identity attributes (k8s.pod.name, k8s.namespace.name, k8s.node.name, workload UIDs, and similar). Prefer those receivers’ built-in attributes rather than re-deriving them.
• Application OTLP and Prometheus-scraped telemetry typically lack that context — enrich them with the k8sattributesprocessor, which associates data to pods and adds standard k8s.* attributes.
• Organizational labels/annotations (team, environment, tier) are not emitted by the infra receivers; extract them with k8sattributesprocessor extract.labels / extract.annotations when you need them on app or pod-scoped signals.

Outcomes:

• Metrics, logs, and traces join on consistent attributes with no per-query remapping.
• OpenTelemetry semantic conventions are adhered to, making adoption of observability tools easier.

3. Use the OpenTelemetry operator as the collector control plane, and scrape Prometheus targets node-locally

Challenges addressed: 3

The OpenTelemetry Operator must manage all OTel Collectors in the cluster. The OpenTelemetryCollector CRD declares Collector configuration as a Kubernetes object, enabling GitOps workflows, versioned rollouts, and the deployment modes this blueprint uses — DaemonSet for per-node collection and optional Deployment for cluster-scoped work when leader election is not available.

Scraping Prometheus-native cluster components (CoreDNS, CNI plugins, Ingress controllers, KEDA, cert-manager, and similar /metrics endpoints) does not require the Target Allocator. Each DaemonSet Collector pod should discover and scrape only targets on its own node, using Kubernetes pod service discovery filtered by spec.nodeName (kube-stack injects ${OTEL_K8S_NODE_NAME} on DaemonSet collectors). Pods opt in with the classic Prometheus annotations (prometheus.io/scrape=true, plus optional prometheus.io/port, prometheus.io/path, and prometheus.io/scheme).

Because a given pod lives on exactly one node, each target is scraped by exactly one Collector replica, meaning there’s no cluster-wide target list, no Target Allocator, and no double-scraping from overlapping ServiceMonitor/PodMonitor jobs. Do not combine this daemonset pattern with Target Allocator (or any other cluster-wide scrape) against the same endpoints.

The Operator additionally introduces the Instrumentation CRD for zero-code auto-instrumentation injection. While that is out of scope here, it makes the Operator the correct foundational dependency for the full cluster observability stack.

Outcomes:

• Each Prometheus target is scraped exactly once by the Collector on the node where the target runs.
• Collector lifecycle is managed automatically with autohealing.

Implementation

The following diagram shows the resulting architecture with the chart’s default values: one DaemonSet Collector per node, node-local receivers on every node, and cluster-scoped receivers running only on the elected leader.

An optional Collector Gateway, not in the scope of this blueprint, is illustrated in the diagram. See the Managed Telemetry Platforms for Kubernetes Workloads blueprint for more information about the Collector Gateway pattern for application-level telemetry.

---
title: "Figure 1: OpenTelemetry Kube Stack architecture (DaemonSet defaults with an optional Gateway)."
config:
  flowchart:
    curve: basis
---
flowchart LR
    subgraph Cluster["Kubernetes Cluster"]
        subgraph Node2["Node 2"]
            SDK2["📦 App Pods<br>(OTel SDK)"]:::node
            Prom2["📊 App Pods<br>(Prometheus)"]:::node
            Kubelet2["☸️ Kubelet"]:::node
            Host2["🖥️ Node Host"]:::node
            Col2["🔀 Collector<br>(DaemonSet, leader)"]:::node
        end

        subgraph Node1["Node 1"]
            SDK1["📦 App Pods<br>(OTel SDK)"]:::node
            Prom1["📊 App Pods<br>(Prometheus)"]:::node
            Kubelet1["☸️ Kubelet"]:::node
            Host1["🖥️ Node Host"]:::node
            Col1["🔀 Collector<br>(DaemonSet)"]:::node
        end

        Gateway["🔀 Collector Gateway<br>(Optional)"]:::optional

        subgraph ControlPlane["Control Plane"]
            APIServer["☸️ kube-apiserver"]:::node
        end
    end

    Backend[("🗄️ Observability Backend")]:::node

    SDK1 L_SDK1@-- OTLP --> Gateway
    Prom1 L_Prom1@<-- prometheus --> Col1
    Kubelet1 L_Kubelet1@<-- "kubelet_stats<br>prometheus (cAdvisor)" --> Col1
    Host1 L_Host1@<-- "file_log<br>host_metrics" --> Col1

    SDK2 L_SDK2@-- OTLP --> Gateway
    Prom2 L_Prom2@<-- prometheus --> Col2
    Kubelet2 L_Kubelet2@<-- "kubelet_stats<br>prometheus (cAdvisor)" --> Col2
    Host2 L_Host2@<-- "file_log<br>host_metrics" --> Col2

    Col2 L_Col2_API@-- "k8s_cluster<br>k8s_objects" --> APIServer

    Col1 L_Col1_Gateway@-- OTLP --> Gateway
    Col2 L_Col2_Gateway@-- OTLP --> Gateway
    Gateway L_Gateway_Backend@-- OTLP --> Backend
    %% Invisible link: keep the Backend in the last column
    ControlPlane ~~~ Backend

    classDef node fill:#ffffff, stroke:#818cf8, stroke-width:2px, color:#6b7280
    classDef optional fill:#ffffff, stroke:#818cf8, stroke-width:2px, stroke-dasharray:5 5, color:#6b7280
    style Cluster fill:#eef2ff, stroke:#818cf8, stroke-width:1px, color:#818cf8
    style ControlPlane fill:#dde4ff, stroke:#818cf8, stroke-width:1px, color:#818cf8
    style Node1 fill:#dde4ff, stroke:#818cf8, stroke-width:1px, color:#818cf8
    style Node2 fill:#dde4ff, stroke:#818cf8, stroke-width:1px, color:#818cf8
    linkStyle 0,4,8,9,10,11 stroke:#a3e635, fill:none, stroke-width:3px
    %% Pull-based receivers: keep only the source-side arrowhead and reverse the dash animation,
    %% so both point from the Collector to what it scrapes
    linkStyle 1,2,3,5,6,7 stroke:#a3e635, fill:none, stroke-width:3px, marker-end:none, animation-direction:reverse

    L_SDK1@{ animation: fast }
    L_Prom1@{ animation: fast }
    L_Kubelet1@{ animation: fast }
    L_Host1@{ animation: fast }
    L_SDK2@{ animation: fast }
    L_Prom2@{ animation: fast }
    L_Kubelet2@{ animation: fast }
    L_Host2@{ animation: fast }
    L_Col2_API@{ animation: fast }
    L_Col1_Gateway@{ animation: fast }
    L_Col2_Gateway@{ animation: fast }
    L_Gateway_Backend@{ animation: fast }

1. Deploy the opentelemetry-kube-stack

Guidelines supported: 1, 2, 3

Deploy the opentelemetry-kube-stack Helm chart as the foundation for this blueprint. The chart installs the OpenTelemetry Operator together with a suite of Collectors managed as OpenTelemetryCollector CRs, so you don’t need separate opentelemetry-operator and opentelemetry-collector chart releases.

Out of the box, the chart deploys a DaemonSet collector with the presets this blueprint relies on (hostMetrics, kubeletMetrics, kubernetesAttributes, kubernetesEvents, and clusterMetrics, logsCollection, among others). Cluster-wide metrics use leader election on that DaemonSet so only one replica emits them; if leader election is not an option, use the chart’s no-leader-election alternative, which separates cluster-scoped collection. The same DaemonSet scrapes Prometheus-native components on the local node (Implementation step 6).

Rather than hand-writing receiver, processor, and RBAC configuration, configure collection through the chart’s presets under collectors.*. Each preset wires the matching receiver or processor into the pipeline and generates the required RBAC, volumes, and mounts. The remaining steps are values.yaml fragments for this chart.

2. Collect cluster state metrics with the clusterMetrics preset

Guidelines supported: 1, 2

The k8s_cluster receiver provides object state (pod phase, restart counts, replica state, node conditions) by watching the Kubernetes API — the OTel-native replacement for kube-state-metrics. It does not provide resource consumption; that comes from the kubelet (Implementation step 3).

Because the receiver gathers cluster-wide telemetry, only one collector replica should emit it. With opentelemetry-kube-stack, enable the clusterMetrics preset on the DaemonSet collector (collectors.daemon.presets.clusterMetrics); leader election ensures a single replica produces the data. If leader election is not available, use the chart’s separated cluster collector instead. The preset adds the k8s_cluster receiver and the required RBAC automatically.

This emits OTel-native equivalents such as k8s.pod.phase, k8s.container.restarts, k8s.deployment.available/k8s.deployment.desired, k8s.node.condition_ready, k8s.pod.status_reason and k8s.container.status.reason. Although they are optional, status reason metrics are especially important for understanding why a pod might be failing. See the k8s_cluster receiver optional metrics documentation for how to enable them.

3. Prefer kubeletstats for resource usage; use hostmetrics only for gaps

Guidelines supported: 1, 2

Both the kubeletstats receiver and hostmetrics receiver can report node CPU and memory. Prefer kubeletstats for node, pod, and container resource usage because it is the kubelet/cAdvisor-aligned source and avoids duplicating the same CPU/memory series from hostmetrics. Enable it on the kube-stack DaemonSet collector (collectors.daemon.presets.kubeletMetrics) with metric_groups: [node, pod, container].

Use hostmetrics preset only for signals kubeletstats preset cannot provide — typically process metrics, plus host-level filesystem, disk I/O, network, load, and paging. When both presets are enabled, disable the overlapping cpu and memory scrapers in hostmetrics so those come solely from kubeletstats.

Important: To guarantee coverage on control-plane and tainted nodes (GPU, spot), add tolerations to the DaemonSet so every node is scraped. Any node without a Collector pod is a coverage gap.

4. Collect Kubernetes events and object state with the k8s_objects receiver

Guidelines supported: 1, 2

The k8s_objects receiver collects Kubernetes API data as logs. Use it for two complementary signals:

• Events from the kubernetesEvents preset: Watch cluster events as they happen, including scheduling failures, probe failures, OOM kills, volume attach errors, and similar operational signals that metrics alone often miss.
• Object resource state from the kubernetesObjects preset: Periodically pull (and optionally watch) Kubernetes objects such as pods, deployments, nodes, and related resources so you retain object state history as logs.

Important: Both signals must run as a Deployment (single-replica). Use leader-election for multiple replica resiliency, if needed.

5. Use k8sattributesprocessor to enrich telemetry with required and optional organizational labels

Guidelines supported: 2

The kubernetesAttributes preset enriches telemetry with relevant Kubernetes metadata. Some of them are required for proper observability, such as k8s.pod.name, k8s.namespace.name, and k8s.node.name.

Configure extract.labels / extract.annotations explicitly for the organizational keys you care about, such as team, squad, owner, etc. If k8s.cluster.name is not resolvable from cloud metadata, inject it as a static resource attribute using the resourceprocessor or set clusterName in the kube-stack values.

Important: Never use a custom label where the K8s semantic conventions has an option for it already.

6. Scrape Prometheus-native components on each node via DaemonSet mode

Guidelines supported: 3

Manual, cluster-wide scrape targets on the Prometheus receiver are prone to double scraping, which inflates metric values and makes them unreliable. The Target Allocator solves that problem by sharding targets across Collector replicas, but it’s unnecessary when every Collector already runs as a DaemonSet and discovery is restricted to the same node the Collector runs on.

The recommended implementation is the opentelemetry-kube-stack DaemonSet pattern: the Prometheus receiver uses Kubernetes pod service discovery restricted to the local node (spec.nodeName=${OTEL_K8S_NODE_NAME}). Each Collector pod scrapes only annotated pods on that node. A pod lives on one node, so it is scraped once.

Out of the box, kube-stack loads collectors.daemon.scrape_configs_file: daemon_scrape_configs.yaml, which already includes a job for pods carrying prometheus.io/scrape=true. To get the same behavior without the scrape file, set scrape_configs_file: "" and enable collectors.daemon.presets.prometheus.podAnnotations. Those presets are mutually exclusive with scrape_configs_file and require mode: daemonset.

Ensure critical cluster components (CoreDNS, CNI plugins, Ingress controllers, KEDA, cert-manager, and similar) expose a /metrics endpoint and carry prometheus.io/scrape=true. Do not also scrape those same endpoints with Target Allocator, ServiceMonitor, or PodMonitor — overlapping discovery is what reintroduces double scraping.

Reference implementations

If you have a production-grade OpenTelemetry architecture that implements this blueprint and would like to be featured on this page, open an issue in the End-User SIG repository!