Semantic conventions for container metrics

Status: Development

Container metrics

This document describes instruments and attributes for common container level metrics in OpenTelemetry. These metrics are collected from technology-specific, well-defined APIs (e.g. Kubelet’s API or container runtimes).

• Metric: container.uptime
• Metric: container.cpu.time
• Metric: container.cpu.usage
• Metric: container.memory.usage
• Metric: container.memory.available
• Metric: container.memory.rss
• Metric: container.memory.working_set
• Metric: container.paging.faults
• Metric: container.disk.io
• Metric: container.network.io
• Metric: container.filesystem.available
• Metric: container.filesystem.capacity
• Metric: container.filesystem.usage

Metric: container.uptime

This metric is recommended.

Name	Instrument Type	Unit (UCUM)	Description	Stability	Entity Associations
container.uptime	Gauge	s	The time the container has been running. [1]		container

[1]: Instrumentations SHOULD use a gauge with type double and measure uptime in seconds as a floating point number with the highest precision available. The actual accuracy would depend on the instrumentation and operating system. In K8s environments, this metric can be derived from Kubelet’s stats API and specifically from the ContainerStats.StartTime field, where StartTime represents the container creation time as noted in the Kubelet source. This value resets on every container restart and aligns with ContainerStateRunning.startedAt from the K8s API.

Metric: container.cpu.time

This metric is recommended.

Name	Instrument Type	Unit (UCUM)	Description	Stability	Entity Associations
container.cpu.time	Counter	s	Total CPU time consumed. [1]		container

[1]: CPU time consumed by the specific container on all available CPU cores

Attributes:

Key	Stability	Requirement Level	Value Type	Description	Example Values
cpu.mode		Conditionally Required [1]	string	The CPU mode for this data point. A container’s CPU metric SHOULD be characterized either by data points with no mode labels, or only data points with mode labels. [2]	user; system (see more)

[1] cpu.mode: Required if mode is available, i.e. metrics coming from the Docker Stats API, containerd stats or cAdvisor.

[2] cpu.mode: Following modes SHOULD be used for containers if available: user, system. In implementations like opencontainers/cgroup, Moby/Docker or cAdvisor, the respective states are retrieved from the cgroup stats directly. In cgroup v1 the user/system modes come from the cpuacct.stat file and the user/system stats respectively. In cgroup v2 the user/system modes come from the cpu.stat file and the user_usec/system_usec stats respectively. Kubernetes only exposes the total CPU time (cpu.stat.usage_usec) through Kubelet’s stats API, hence the metric should be reported without the mode attribute implying it is the total of all modes. In pure container environments the CPU’s mode is usually available and cpu.mode should be reported. In that case summarising the CPU time over the different modes gives the total CPU time.

cpu.mode has the following list of well-known values. If one of them applies, then the respective value MUST be used; otherwise, a custom value MAY be used.

Value	Description	Stability
idle	Idle
interrupt	Interrupt
iowait	IO Wait
nice	Nice
steal	Steal
system	System
user	User

Metric: container.cpu.usage

This metric is opt-in.

Name	Instrument Type	Unit (UCUM)	Description	Stability	Entity Associations
container.cpu.usage	Gauge	{cpu}	Container’s CPU usage, measured in CPUs. Range from 0 to the number of allocatable CPUs. [1]		container

[1]: CPU usage of the specific container on all available CPU cores. It is calculated as the change in cumulative CPU time (container.cpu.time) over a measurement interval, divided by the elapsed time: usageCores = (cpuTimeEnd - cpuTimeStart) / elapsedSeconds

Attributes:

Key	Stability	Requirement Level	Value Type	Description	Example Values
cpu.mode		Conditionally Required [1]	string	The CPU mode for this data point. A container’s CPU metric SHOULD be characterized either by data points with no mode labels, or only data points with mode labels. [2]	user; system (see more)

[1] cpu.mode: Required if mode is available, i.e. metrics coming from the Docker Stats API, containerd stats or cAdvisor.

[2] cpu.mode: Following modes SHOULD be used for containers if available: user, system.

cpu.mode has the following list of well-known values. If one of them applies, then the respective value MUST be used; otherwise, a custom value MAY be used.

Value	Description	Stability
idle	Idle
interrupt	Interrupt
iowait	IO Wait
nice	Nice
steal	Steal
system	System
user	User

Metric: container.memory.usage

This metric is opt-in.

Name	Instrument Type	Unit (UCUM)	Description	Stability	Entity Associations
container.memory.usage	UpDownCounter	By	Memory usage of the container. [1]		container

[1]: Current memory usage, including all memory regardless of when it was accessed. In general, this metric can be derived from cadvisor and specifically the container_memory_usage_bytes metric. In K8s, this metric is derived from the MemoryStats.UsageBytes field of the ContainerStats.Memory of the Kubelet’s stats API.

Metric: container.memory.available

This metric is opt-in.

Name	Instrument Type	Unit (UCUM)	Description	Stability	Entity Associations
container.memory.available	UpDownCounter	By	Container memory available. [1]		container

[1]: Available memory for use. This is defined as the memory limit - workingSetBytes. If memory limit is undefined, the available bytes is omitted. In general, this metric can be derived from cadvisor and by subtracting the container_memory_working_set_bytes metric from the container_spec_memory_limit_bytes metric. In K8s, this metric is derived from the MemoryStats.AvailableBytes field of the ContainerStats.Memory of the Kubelet’s stats API.

Metric: container.memory.rss

This metric is opt-in.

Name	Instrument Type	Unit (UCUM)	Description	Stability	Entity Associations
container.memory.rss	UpDownCounter	By	Container memory RSS. [1]		container

[1]: In general, this metric can be derived from cadvisor and specifically the container_memory_rss metric. In K8s, this metric is derived from the MemoryStats.RSSBytes field of the ContainerStats.Memory of the Kubelet’s stats API.

Metric: container.memory.working_set

This metric is opt-in.

Name	Instrument Type	Unit (UCUM)	Description	Stability	Entity Associations
container.memory.working_set	UpDownCounter	By	Container memory working set. [1]		container

[1]: In general, this metric can be derived from cadvisor and specifically the container_memory_working_set_bytes metric. When derived from cAdvisor on Linux, the working set is calculated by excluding inactive file-backed memory from memory usage (inactive_file on cgroup v2, total_inactive_file on cgroup v1), so active file-backed memory remains included. In K8s, this metric is derived from the MemoryStats.WorkingSetBytes field of the ContainerStats.Memory of the Kubelet’s stats API.

Metric: container.paging.faults

This metric is opt-in.

Name	Instrument Type	Unit (UCUM)	Description	Stability	Entity Associations
container.paging.faults	Counter	{fault}	Container memory paging faults. [1]		container

[1]: In general, this metric can be derived from cadvisor. The container_memory_failures_total{failure_type=pgfault, scope=container} counter reports the total page faults (minor + major), while container_memory_failures_total{failure_type=pgmajfault, scope=container} reports only major faults. The minor page faults MUST be derived as the difference between the two. In K8s, this metric is derived from the MemoryStats.PageFaults and MemoryStats.MajorPageFaults fields of the ContainerStats.Memory of the Kubelet’s stats API. MemoryStats.PageFaults reports the total number of page faults (equivalent to pgfault in the cgroup memory.stat), with minor and major combined. The minor page faults MUST therefore be derived as MemoryStats.PageFaults - MemoryStats.MajorPageFaults, so that summing the metric over system.paging.fault.type yields the total number of page faults.

Attributes:

Key	Stability	Requirement Level	Value Type	Description	Example Values
system.paging.fault.type		Recommended	string	The paging fault type	minor (see more)

system.paging.fault.type has the following list of well-known values. If one of them applies, then the respective value MUST be used; otherwise, a custom value MAY be used.

Value	Description	Stability
major	major
minor	minor

Metric: container.disk.io

This metric is opt-in.

Name	Instrument Type	Unit (UCUM)	Description	Stability	Entity Associations
container.disk.io	Counter	By	Disk bytes for the container. [1]		container

[1]: The total number of bytes read/written successfully (aggregated from all disks).

Attributes:

Key	Stability	Requirement Level	Value Type	Description	Example Values
disk.io.direction		Recommended	string	The disk IO operation direction.	read (see more)
system.device.name		Recommended	string	The OS level identifier for a device, such as disk, partition or network interface.	/dev/sda; eth0; \network-drive

disk.io.direction has the following list of well-known values. If one of them applies, then the respective value MUST be used; otherwise, a custom value MAY be used.

Value	Description	Stability
read	read
write	write

Metric: container.network.io

This metric is opt-in.

Name	Instrument Type	Unit (UCUM)	Description	Stability	Entity Associations
container.network.io	Counter	By	Network bytes for the container. [1]		container

[1]: The number of bytes sent/received on all network interfaces by the container.

Attributes:

Key	Stability	Requirement Level	Value Type	Description	Example Values
network.interface.name		Recommended	string	The network interface name.	lo; eth0
network.io.direction		Recommended	string	The direction of traffic from the perspective of the observing host’s physical or virtual network interface. It should not be used to represent the logical direction of a stateful connection or network flow.	transmit (see more)

network.io.direction has the following list of well-known values. If one of them applies, then the respective value MUST be used; otherwise, a custom value MAY be used.

Value	Description	Stability
receive	receive
transmit	transmit

Metric: container.filesystem.available

This metric is recommended.

Name	Instrument Type	Unit (UCUM)	Description	Stability	Entity Associations
container.filesystem.available	UpDownCounter	By	Container filesystem available bytes. [1]		container

[1]: Available filesystem space for the container’s rootfs. In K8s, this metric is derived from the FsStats.AvailableBytes field of the ContainerStats.Rootfs of the Kubelet’s stats API.

Metric: container.filesystem.capacity

This metric is recommended.

Name	Instrument Type	Unit (UCUM)	Description	Stability	Entity Associations
container.filesystem.capacity	UpDownCounter	By	Container filesystem capacity. [1]		container

[1]: Total filesystem capacity for the container’s rootfs. In general, this metric can be derived from cadvisor and specifically the container_fs_limit_bytes metric for the container’s rootfs device. In K8s, this metric is derived from the FsStats.CapacityBytes field of the ContainerStats.Rootfs of the Kubelet’s stats API.

Metric: container.filesystem.usage

This metric is recommended.

Name	Instrument Type	Unit (UCUM)	Description	Stability	Entity Associations
container.filesystem.usage	UpDownCounter	By	Container filesystem usage. [1]		container

[1]: Filesystem space used by the container’s rootfs. In general, this metric can be derived from cadvisor and specifically the container_fs_usage_bytes metric for the container’s rootfs device. In K8s, this metric is derived from the FsStats.UsedBytes field of the ContainerStats.Rootfs of the Kubelet’s stats API. Capacity and available are reported for the backing filesystem, which may be shared with other workloads; therefore, usage may not equal capacity - available.