Injecting Auto-instrumentation

An implementation of auto-instrumentation using the OpenTelemetry Operator.

আপনি এই পৃষ্ঠার ইংরেজি সংস্করণ দেখছেন কারণ এটি এখনও সম্পূর্ণভাবে অনুবাদ করা হয়নি। সাহায্য করতে আগ্রহী? দেখুন Contributing।

The OpenTelemetry Operator supports injecting and configuring auto-instrumentation libraries for .NET, Java, Node.js, Python, and Go services.

Installation

First, install the OpenTelemetry Operator into your cluster.

You can do this with the Operator release manifest, the Operator helm chart, or with Operator Hub.

In most cases, you will need to install cert-manager. If you use the helm chart, there is an option to generate a self-signed certificate instead.

If you want to use Go auto-instrumentation, you need to enable the feature gate. See Controlling Instrumentation Capabilities for details.

Create an OpenTelemetry Collector (Optional)

It is a best practice to send telemetry from containers to an OpenTelemetry Collector instead of directly to a backend. The Collector helps simplify secret management, decouples data export problems (such as a need to do retries) from your apps, and lets you add additional data to your telemetry, such as with the k8sattributesprocessor component. If you chose not to use a Collector, you can skip to the next section.

The Operator provides a Custom Resource Definition (CRD) for the OpenTelemetry Collector which is used to create an instance of the Collector that the Operator manages. The following example deploys the Collector as a deployment (the default), but there are other deployment modes that can be used.

When using the Deployment mode the operator will also create a Service that can be used to interact with the Collector. The name of the service is the name of the OpenTelemetryCollector resource prepended to -collector. For our example that will be demo-collector.

kubectl apply -f - <<EOF
apiVersion: opentelemetry.io/v1beta1
kind: OpenTelemetryCollector
metadata:
  name: demo
spec:
  config:
    receivers:
      otlp:
        protocols:
          grpc:
            endpoint: 0.0.0.0:4317
          http:
            endpoint: 0.0.0.0:4318
    processors:
      memory_limiter:
        check_interval: 1s
        limit_percentage: 75
        spike_limit_percentage: 15
    exporters:
      debug:
        verbosity: basic

    service:
      pipelines:
        traces:
          receivers: [otlp]
          processors: [memory_limiter]
          exporters: [debug]
        metrics:
          receivers: [otlp]
          processors: [memory_limiter]
          exporters: [debug]
        logs:
          receivers: [otlp]
          processors: [memory_limiter]
          exporters: [debug]
EOF

The above command results in a deployment of the Collector that you can use as an endpoint for auto-instrumentation in your pods.

Configure Automatic Instrumentation

To be able to manage automatic instrumentation, the Operator needs to be configured to know what pods to instrument and which automatic instrumentation to use for those pods. This is done via the Instrumentation CRD.

Creating the Instrumentation resource correctly is paramount for getting auto-instrumentation working. Making sure all endpoints and environment variables are correct is required for auto-instrumentation to work properly.

.NET

The following command will create a basic Instrumentation resource that is configured specifically for instrumenting .NET services.

kubectl apply -f - <<EOF
apiVersion: opentelemetry.io/v1alpha1
kind: Instrumentation
metadata:
  name: demo-instrumentation
spec:
  exporter:
    endpoint: http://demo-collector:4318
  propagators:
    - tracecontext
    - baggage
  sampler:
    type: parentbased_traceidratio
    argument: "1"
EOF

By default, the Instrumentation resource that auto-instruments .NET services uses otlp with the http/protobuf protocol. This means that the configured endpoint must be able to receive OTLP over http/protobuf. Therefore, the example uses http://demo-collector:4318, which will connect to the http port of the otlpreceiver of the Collector created in the previous step.

Excluding auto-instrumentation

By default, the .NET auto-instrumentation ships with many instrumentation libraries. This makes instrumentation easy, but could result in too much or unwanted data. If there are any libraries you do not want to use you can set OTEL_DOTNET_AUTO_[SIGNAL]_[NAME]_INSTRUMENTATION_ENABLED=false where [SIGNAL] is the type of the signal and [NAME] is the case-sensitive name of the library.

apiVersion: opentelemetry.io/v1alpha1
kind: Instrumentation
metadata:
  name: demo-instrumentation
spec:
  exporter:
    endpoint: http://demo-collector:4318
  propagators:
    - tracecontext
    - baggage
  sampler:
    type: parentbased_traceidratio
    argument: '1'
  dotnet:
    env:
      - name: OTEL_DOTNET_AUTO_TRACES_GRPCNETCLIENT_INSTRUMENTATION_ENABLED
        value: false
      - name: OTEL_DOTNET_AUTO_METRICS_PROCESS_INSTRUMENTATION_ENABLED
        value: false

.NET auto-instrumentation also supports a runtime annotation to set the .NET Runtime Identifier (RID). Currently linux-x64 (default) and linux-musl-x64 are supported:

instrumentation.opentelemetry.io/inject-dotnet: "true"
instrumentation.opentelemetry.io/otel-dotnet-auto-runtime: "linux-x64"   # default, can be omitted
instrumentation.opentelemetry.io/otel-dotnet-auto-runtime: "linux-musl-x64"  # for musl-based images

Note: By default, the operator sets OTEL_DOTNET_AUTO_TRACES_ENABLED_INSTRUMENTATIONS to all available instrumentations supported by the consumed opentelemetry-dotnet-instrumentation release (e.g. AspNet,HttpClient,SqlClient). This value can be overridden by configuring the environment variable explicitly.

Learn more

For more details, see .NET Auto Instrumentation docs.

Deno

The following command creates a basic Instrumentation resource that is configured for instrumenting Deno services.

kubectl apply -f - <<EOF
apiVersion: opentelemetry.io/v1alpha1
kind: Instrumentation
metadata:
  name: demo-instrumentation
spec:
  env:
    - name: OTEL_DENO
      value: 'true'
  exporter:
    endpoint: http://demo-collector:4318
  propagators:
    - tracecontext
    - baggage
  sampler:
    type: parentbased_traceidratio
    argument: '1'
EOF

Deno processes automatically export telemetry data to the configured endpoint when they are started with the OTEL_DENO=true environment variable. Therefore, the example specifies this environment variable in the env field of the Instrumentation resource, so it is set for all services that have env vars injected with this Instrumentation resource.

By default, the Instrumentation resource that auto-instruments Deno services uses otlp with the http/proto protocol. This means that the configured endpoint must be able to receive OTLP over http/proto. Therefore, the example uses http://demo-collector:4318, which connects to the http/proto port of the otlpreceiver of the Collector created in the previous step.

নোট

Deno’s OpenTelemetry integration is not yet stable. As a result all workloads that want to be instrumented with Deno must have the --unstable-otel flag set when starting the Deno process.

Configuration options

By default, the Deno OpenTelemetry integration exports console.log() output as logs, while still printing the logs to stdout / stderr. You can configure these alternative behaviors:

OTEL_DENO_CONSOLE=replace: only export console.log() output as logs; do not print to stdout / stderr.
OTEL_DENO_CONSOLE=ignore: do not export console.log() output as logs; do print to stdout / stderr.

Learn more

For more details, see Deno’s OpenTelemetry integration documentation.

Go

The following command creates a basic Instrumentation resource that is configured specifically for instrumenting Go services.

kubectl apply -f - <<EOF
apiVersion: opentelemetry.io/v1alpha1
kind: Instrumentation
metadata:
  name: demo-instrumentation
spec:
  exporter:
    endpoint: http://demo-collector:4318
  propagators:
    - tracecontext
    - baggage
  sampler:
    type: parentbased_traceidratio
    argument: "1"
EOF

By default, the Instrumentation resource that auto-instruments Go services uses otlp with the http/protobuf protocol. This means that the configured endpoint must be able to receive OTLP over http/protobuf. Therefore, the example uses http://demo-collector:4318, which connects to the http/protobuf port of the otlpreceiver of the Collector created in the previous step.

The Go auto-instrumentation does not support disabling any instrumentation. See the Go Auto-Instrumentation repository for more details.

Java

The following command creates a basic Instrumentation resource that is configured for instrumenting Java services.

kubectl apply -f - <<EOF
apiVersion: opentelemetry.io/v1alpha1
kind: Instrumentation
metadata:
  name: demo-instrumentation
spec:
  exporter:
    endpoint: http://demo-collector:4318
  propagators:
    - tracecontext
    - baggage
  sampler:
    type: parentbased_traceidratio
    argument: "1"
EOF

By default, the Instrumentation resource that auto-instruments Java services uses otlp with the http/protobuf protocol. This means that the configured endpoint must be able to receive OTLP over http via protobuf payloads. Therefore, the example uses http://demo-collector:4318, which connects to the http port of the otlpreceiver of the Collector created in the previous step.

Excluding auto-instrumentation

By default, the Java auto-instrumentation ships with many instrumentation libraries. This makes instrumentation easy, but could result in too much or unwanted data. If there are any libraries you do not want to use you can set OTEL_INSTRUMENTATION_[NAME]_ENABLED=false where [NAME] is the name of the library. If you know exactly which libraries you want to use, you can disable the default libraries by setting OTEL_INSTRUMENTATION_COMMON_DEFAULT_ENABLED=false and then use OTEL_INSTRUMENTATION_[NAME]_ENABLED=true where [NAME] is the name of the library. For more details, see Suppressing specific instrumentation.

apiVersion: opentelemetry.io/v1alpha1
kind: Instrumentation
metadata:
  name: demo-instrumentation
spec:
  exporter:
    endpoint: http://demo-collector:4318
  propagators:
    - tracecontext
    - baggage
  sampler:
    type: parentbased_traceidratio
    argument: '1'
  java:
    env:
      - name: OTEL_INSTRUMENTATION_KAFKA_ENABLED
        value: false
      - name: OTEL_INSTRUMENTATION_REDISCALA_ENABLED
        value: false

Learn more

For more details, see Java agent Configuration.

Node.js

The following command creates a basic Instrumentation resource that is configured for instrumenting Node.js services.

kubectl apply -f - <<EOF
apiVersion: opentelemetry.io/v1alpha1
kind: Instrumentation
metadata:
  name: demo-instrumentation
spec:
  exporter:
    endpoint: http://demo-collector:4317
  propagators:
    - tracecontext
    - baggage
  sampler:
    type: parentbased_traceidratio
    argument: "1"
EOF

By default, the Instrumentation resource that auto-instruments Node.js services uses otlp with the grpc protocol. This means that the configured endpoint must be able to receive OTLP over grpc. Therefore, the example uses http://demo-collector:4317, which connects to the grpc port of the otlpreceiver of the Collector created in the previous step.

Excluding instrumentation libraries

By default, the Node.js zero-code instrumentation has all the instrumentation libraries enabled.

To enable only specific instrumentation libraries you can use the OTEL_NODE_ENABLED_INSTRUMENTATIONS environment variable as documented in the Node.js zero-code instrumentation documentation.

apiVersion: opentelemetry.io/v1alpha1
kind: Instrumentation
# ... other fields skipped from this example
spec:
  # ... other fields skipped from this example
  nodejs:
    env:
      - name: OTEL_NODE_ENABLED_INSTRUMENTATIONS
        value: http,nestjs-core # comma-separated list of the instrumentation package names without the `@opentelemetry/instrumentation-` prefix.

To keep all default libraries and disable only specific instrumentation libraries you can use the OTEL_NODE_DISABLED_INSTRUMENTATIONS environment variable. For details, see Excluding instrumentation libraries.

apiVersion: opentelemetry.io/v1alpha1
kind: Instrumentation
# ... other fields skipped from this example
spec:
  # ... other fields skipped from this example
  nodejs:
    env:
      - name: OTEL_NODE_DISABLED_INSTRUMENTATIONS
        value: fs,grpc # comma-separated list of the instrumentation package names without the `@opentelemetry/instrumentation-` prefix.

নোট

If both environment variables are set, OTEL_NODE_ENABLED_INSTRUMENTATIONS is applied first, and then OTEL_NODE_DISABLED_INSTRUMENTATIONS is applied to that list. Therefore, if the same instrumentation is included in both lists, that instrumentation will be disabled.

Learn more

For more details, see Node.js auto-instrumentation.

Python

The following command will create a basic Instrumentation resource that is configured specifically for instrumenting Python services.

kubectl apply -f - <<EOF
apiVersion: opentelemetry.io/v1alpha1
kind: Instrumentation
metadata:
  name: demo-instrumentation
spec:
  exporter:
    endpoint: http://demo-collector:4318
  propagators:
    - tracecontext
    - baggage
  sampler:
    type: parentbased_traceidratio
    argument: "1"
EOF

By default, the Instrumentation resource that auto-instruments Python services uses otlp with the http/protobuf protocol (gRPC is not supported at this time). This means that the configured endpoint must be able to receive OTLP over http/protobuf. Therefore, the example uses http://demo-collector:4318, which will connect to the http port of the otlpreceiver of the Collector created in the previous step.

As of operator v0.108.0, the Instrumentation resource automatically sets OTEL_EXPORTER_OTLP_PROTOCOL to http/protobuf for Python services. If you use an older version of the Operator you MUST set this environment variable to http/protobuf, or Python auto-instrumentation will not work.

Auto-instrumenting Python logs

By default, Python logs auto-instrumentation is disabled. If you would like to enable this feature, you must to set OTEL_PYTHON_LOGGING_AUTO_INSTRUMENTATION_ENABLED environment variable as follows:

apiVersion: opentelemetry.io/v1alpha1
kind: Instrumentation
metadata:
  name: python-instrumentation
  namespace: application
spec:
  exporter:
    endpoint: http://demo-collector:4318
  env:
  propagators:
    - tracecontext
    - baggage
  python:
    env:
      - name: OTEL_PYTHON_LOGGING_AUTO_INSTRUMENTATION_ENABLED
        value: 'true'

As of operator v0.111.0 setting OTEL_LOGS_EXPORTER to otlp is not required anymore.

Excluding auto-instrumentation

By default, the Python auto-instrumentation ships with many instrumentation libraries. This makes instrumentation easy, but can result in too much or unwanted data. If there are any packages you do not want to instrument, you can set the OTEL_PYTHON_DISABLED_INSTRUMENTATIONS environment variable.

apiVersion: opentelemetry.io/v1alpha1
kind: Instrumentation
metadata:
  name: demo-instrumentation
spec:
  exporter:
    endpoint: http://demo-collector:4318
  propagators:
    - tracecontext
    - baggage
  sampler:
    type: parentbased_traceidratio
    argument: '1'
  python:
    env:
      - name: OTEL_PYTHON_DISABLED_INSTRUMENTATIONS
        value:
          <comma-separated list of package names to exclude from
          instrumentation>

See the Python agent configuration docs for more details.

Learn more

For Python-specific quirks, see Python OpenTelemetry Operator docs and the Python agent configuration docs.

Now that your Instrumentation object is created, your cluster has the ability to auto-instrument services and send data to an endpoint. However, auto-instrumentation with the OpenTelemetry Operator follows an opt-in model. In order to activate automatic instrumentation, you’ll need to add an annotation to your deployment.

Add annotations to existing deployments

The final step is to opt in your services to automatic instrumentation. This is done by updating your service’s spec.template.metadata.annotations to include a language-specific annotation:

.NET: instrumentation.opentelemetry.io/inject-dotnet: "true"
Deno: instrumentation.opentelemetry.io/inject-sdk: "true"
Go: instrumentation.opentelemetry.io/inject-go: "true"
Java: instrumentation.opentelemetry.io/inject-java: "true"
Node.js: instrumentation.opentelemetry.io/inject-nodejs: "true"
Python: instrumentation.opentelemetry.io/inject-python: "true"

The possible values for the annotation can be

"true" - to inject Instrumentation resource with default name from the current namespace.
"my-instrumentation" - to inject Instrumentation CR instance with name "my-instrumentation" in the current namespace.
"my-other-namespace/my-instrumentation" - to inject Instrumentation CR instance with name "my-instrumentation" from another namespace "my-other-namespace".
"false" - do not inject

Alternatively, the annotation can be added to a namespace, which will result in all services in that namespace to opt-in to automatic instrumentation. See the Operators auto-instrumentation documentation for more details.

Opt-in a Go Service

Unlike other languages’ auto-instrumentation, Go works via an eBPF agent running via a sidecar. When opted in, the Operator will inject this sidecar into your pod. In addition to the instrumentation.opentelemetry.io/inject-go annotation mentioned above, you must also supply a value for the OTEL_GO_AUTO_TARGET_EXE environment variable.

You can set this environment variable via the instrumentation.opentelemetry.io/otel-go-auto-target-exe annotation.

instrumentation.opentelemetry.io/inject-go: 'true'
instrumentation.opentelemetry.io/otel-go-auto-target-exe: '/path/to/container/executable'

This environment variable can also be set via the Instrumentation resource, with the annotation taking precedence. Since Go auto-instrumentation requires OTEL_GO_AUTO_TARGET_EXE to be set, you must supply a valid executable path via the annotation or the Instrumentation resource. Failure to set this value causes instrumentation injection to abort, leaving the original pod unchanged.

Since Go auto-instrumentation uses eBPF, it also requires elevated permissions. When you opt in, the sidecar the Operator injects will require the following permissions:

securityContext:
  privileged: true
  runAsUser: 0

Auto-instrumenting a Python musl based container

Since operator v0.113.0 Python auto-instrumentation also honors an annotation that will permit it to run it on images with a different C library than glibc.

# for Linux glibc based images, this is the default value and can be omitted
instrumentation.opentelemetry.io/otel-python-platform: "glibc"
# for Linux musl based images
instrumentation.opentelemetry.io/otel-python-platform: "musl"

Multi-container pods

Single instrumentation

If nothing else is specified, instrumentation is performed on the first container available in the pod spec (from .spec.containers, not init containers). In some cases — for example when an Istio sidecar is injected — it becomes necessary to specify on which container(s) the injection must be performed.

Use the instrumentation.opentelemetry.io/container-names annotation to indicate one or more container names (from .spec.containers.name or .spec.initContainers.name) on which the injection must be made:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: my-deployment-with-multiple-containers
spec:
  selector:
    matchLabels:
      app: my-pod-with-multiple-containers
  replicas: 1
  template:
    metadata:
      labels:
        app: my-pod-with-multiple-containers
      annotations:
        instrumentation.opentelemetry.io/inject-java: 'true'
        instrumentation.opentelemetry.io/container-names: 'myapp,myapp2'
    spec:
      containers:
        - name: myapp
          image: myImage1
        - name: myapp2
          image: myImage2
        - name: myapp3
          image: myImage3

In the above case, myapp and myapp2 containers will be instrumented; myapp3 will not.

NOTE: Go auto-instrumentation does not support multi-container pods. When injecting Go auto-instrumentation the first container should be the only container you want instrumented.

Instrumenting init containers

Init containers can be instrumented by including their names in the container-names annotation. When an init container is targeted for instrumentation, the operator automatically inserts the instrumentation init container before the target init container in the pod’s init container sequence. This ensures the instrumentation agent files are available when the target init container runs.

Supported instrumentations for init containers: Java, Python, Node.js, .NET, and SDK-only injection.

Not supported for init containers: Go (does not support multi-container pods), Apache HTTPD, and NGINX.

Note: Kubernetes guarantees that container names are unique across both the initContainers and containers lists within a pod spec, allowing the operator to unambiguously identify whether a container name refers to an init container or a regular container.

Example with both init container and regular container instrumentation:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: my-deployment-with-init-container
spec:
  selector:
    matchLabels:
      app: my-app
  replicas: 1
  template:
    metadata:
      labels:
        app: my-app
      annotations:
        instrumentation.opentelemetry.io/inject-python: 'true'
        instrumentation.opentelemetry.io/container-names: 'my-init-job,myapp'
    spec:
      initContainers:
        - name: my-init-job
          image: my-python-init-image
      containers:
        - name: myapp
          image: my-python-app-image

In this example, both my-init-job (an init container) and myapp (a regular container) will be instrumented with Python auto-instrumentation.

Multiple instrumentations

Multi-instrumentation works only when the enable-multi-instrumentation feature flag is set to true. When enabled, use language-specific container name annotations to specify which containers should receive which instrumentation.

If language instrumentation specific container names are not specified, instrumentation is performed on the first regular container available in the pod spec (only if single instrumentation injection is configured).

In some cases, containers in the same pod use different technologies. Use the language-specific container name annotations to indicate one or more container names (from .spec.containers.name or .spec.initContainers.name) on which the injection must be made:

Language	Annotation
Java	`instrumentation.opentelemetry.io/java-container-names`
Node.js	`instrumentation.opentelemetry.io/nodejs-container-names`
Python	`instrumentation.opentelemetry.io/python-container-names`
.NET	`instrumentation.opentelemetry.io/dotnet-container-names`
Go	`instrumentation.opentelemetry.io/go-container-names`
Apache HTTPD	`instrumentation.opentelemetry.io/apache-httpd-container-names`
NGINX	`instrumentation.opentelemetry.io/nginx-container-names`
SDK only	`instrumentation.opentelemetry.io/sdk-container-names`

Example with Java and Python running in different containers:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: my-deployment-with-multi-containers-multi-instrumentations
spec:
  selector:
    matchLabels:
      app: my-pod-with-multi-containers-multi-instrumentations
  replicas: 1
  template:
    metadata:
      labels:
        app: my-pod-with-multi-containers-multi-instrumentations
      annotations:
        instrumentation.opentelemetry.io/inject-java: 'true'
        instrumentation.opentelemetry.io/java-container-names: 'myapp,myapp2'
        instrumentation.opentelemetry.io/inject-python: 'true'
        instrumentation.opentelemetry.io/python-container-names: 'myapp3'
    spec:
      containers:
        - name: myapp
          image: myImage1
        - name: myapp2
          image: myImage2
        - name: myapp3
          image: myImage3

In the above case, myapp and myapp2 will be instrumented with Java and myapp3 with Python instrumentation.

NOTE: Go auto-instrumentation does not support multi-container pods. NOTE: A single container cannot be instrumented with multiple language instrumentations. NOTE: The instrumentation.opentelemetry.io/container-names annotation is not used for this feature.

Using customized or vendor instrumentation

By default, the operator uses upstream auto-instrumentation libraries. Custom auto-instrumentation images can be configured by overriding the image fields in the Instrumentation CR:

apiVersion: opentelemetry.io/v1alpha1
kind: Instrumentation
metadata:
  name: my-instrumentation
spec:
  java:
    image: your-customized-auto-instrumentation-image:java
  nodejs:
    image: your-customized-auto-instrumentation-image:nodejs
  python:
    image: your-customized-auto-instrumentation-image:python
  dotnet:
    image: your-customized-auto-instrumentation-image:dotnet
  go:
    image: your-customized-auto-instrumentation-image:go
  apacheHttpd:
    image: your-customized-auto-instrumentation-image:apache-httpd
  nginx:
    image: your-customized-auto-instrumentation-image:nginx

The Dockerfiles for auto-instrumentation can be found in the autoinstrumentation directory. Follow the instructions in the Dockerfiles on how to build a custom container image.

Using Apache HTTPD auto-instrumentation

For Apache HTTPD auto-instrumentation, the operator assumes HTTPD version 2.4 and configuration directory /usr/local/apache2/conf by default (as used in the official httpd image). If you need version 2.2, a different config directory, or custom agent attributes, use the following example:

apiVersion: opentelemetry.io/v1alpha1
kind: Instrumentation
metadata:
  name: my-instrumentation
spec:
  apacheHttpd:
    image: your-customized-auto-instrumentation-image:apache-httpd
    version: '2.2'
    configPath: /your-custom-config-path
    attrs:
      - name: ApacheModuleOtelMaxQueueSize
        value: '4096'

A full list of available attributes can be found at otel-webserver-module.

Using NGINX auto-instrumentation

For NGINX auto-instrumentation, NGINX versions 1.22.0, 1.23.0, and 1.23.1 are supported. The NGINX configuration file is expected to be /etc/nginx/nginx.conf by default. Instrumentation also expects a conf.d directory in the same directory as the configuration file, with an include <config-file-dir-path>/conf.d/*.conf; directive in the http { ... } section. You can also adjust OpenTelemetry SDK attributes:

apiVersion: opentelemetry.io/v1alpha1
kind: Instrumentation
metadata:
  name: my-instrumentation
spec:
  nginx:
    image: your-customized-auto-instrumentation-image:nginx
    configFile: /my/custom-dir/custom-nginx.conf
    attrs:
      - name: NginxModuleOtelMaxQueueSize
        value: '4096'

A full list of available attributes can be found at otel-webserver-module.

Inject OpenTelemetry SDK environment variables only

You can configure the OpenTelemetry SDK for applications that cannot currently be auto-instrumented by using inject-sdk in place of inject-python or inject-java. This will inject environment variables like OTEL_RESOURCE_ATTRIBUTES, OTEL_TRACES_SAMPLER, and OTEL_EXPORTER_OTLP_ENDPOINT that you configure in the Instrumentation resource, but will not inject the SDK itself.

instrumentation.opentelemetry.io/inject-sdk: "true"

Controlling Instrumentation Capabilities

The operator allows specifying, via feature flags, which languages the Instrumentation resource may instrument. Languages enabled by default only need their gate supplied when disabling. Language support can be disabled by passing the flag with a value of false.

Language	Gate	Default Value
Java	`enable-java-instrumentation`	`true`
Node.js	`enable-nodejs-instrumentation`	`true`
Python	`enable-python-instrumentation`	`true`
.NET	`enable-dotnet-instrumentation`	`true`
Apache HTTPD	`enable-apache-httpd-instrumentation`	`true`
Go	`enable-go-instrumentation`	`false`
NGINX	`enable-nginx-instrumentation`	`false`

Multi-instrumentation (multiple languages in the same pod) can be enabled with the enable-multi-instrumentation flag, which defaults to false. For more information about multi-instrumentation feature capabilities, see Multi-container pods with multiple instrumentations.

Configure resource attributes

The OpenTelemetry Operator can automatically set resource attributes as defined in the OpenTelemetry Semantic Conventions.

Configure resource attributes with annotations

Use the resource.opentelemetry.io/ annotation prefix to add resource attributes to data produced by OpenTelemetry instrumentation:

apiVersion: v1
kind: Pod
metadata:
  name: example-pod
  annotations:
    resource.opentelemetry.io/service.name: 'my-service'
    resource.opentelemetry.io/service.version: '1.0.0'
    resource.opentelemetry.io/deployment.environment.name: 'production'
spec:
  containers:
    - name: main-container
      image: your-image:tag

Configure resource attributes with labels

You can also use common Kubernetes labels to set resource attributes (first entry wins). The following labels are supported:

app.kubernetes.io/instance → service.name
app.kubernetes.io/name → service.name
app.kubernetes.io/version → service.version

apiVersion: v1
kind: Pod
metadata:
  name: example-pod
  labels:
    app.kubernetes.io/name: 'my-service'
    app.kubernetes.io/version: '1.0.0'
    app.kubernetes.io/part-of: 'shop'
spec:
  containers:
    - name: main-container
      image: your-image:tag

This requires explicit opt-in via the Instrumentation CR:

apiVersion: opentelemetry.io/v1alpha1
kind: Instrumentation
metadata:
  name: my-instrumentation
spec:
  defaults:
    useLabelsForResourceAttributes: true

Priority for setting resource attributes

The priority for setting resource attributes is as follows (first found wins):

OTEL_RESOURCE_ATTRIBUTES and OTEL_SERVICE_NAME environment variables
Annotations with the resource.opentelemetry.io/ prefix
Labels (e.g. app.kubernetes.io/name) when defaults.useLabelsForResourceAttributes=true
Resource attributes calculated from the pod’s metadata (e.g. k8s.pod.name)
Resource attributes set in the Instrumentation CR under spec.resource.resourceAttributes

This priority is applied per attribute individually, so it is possible to set some attributes via annotations and others via labels.

How resource attributes are calculated from pod metadata

How `service.name` is calculated

The first value found in this order is used:

pod.annotation[resource.opentelemetry.io/service.name]
pod.label[app.kubernetes.io/name] (if useLabelsForResourceAttributes=true)
k8s.deployment.name
k8s.replicaset.name
k8s.statefulset.name
k8s.daemonset.name
k8s.cronjob.name
k8s.job.name
k8s.pod.name
k8s.container.name

How `service.version` is calculated

The first value found in this order is used:

pod.annotation[resource.opentelemetry.io/service.version]
pod.label[app.kubernetes.io/version] (if useLabelsForResourceAttributes=true)
Container Docker image tag (only if the tag does not contain a /)

How `service.instance.id` is calculated

The first value found in this order is used:

pod.annotation[resource.opentelemetry.io/service.instance.id]
Concatenation of k8s.namespace.name, k8s.pod.name, and k8s.container.name joined by .

How `service.namespace` is calculated

The first value found in this order is used:

pod.annotation[resource.opentelemetry.io/service.namespace]
k8s.namespace.name

Troubleshooting

If you run into problems trying to auto-instrument your code, here are a few things that you can try.

Did the Instrumentation resource install?

After installing the Instrumentation resource, verify that it installed correctly by running this command, where <namespace> is the namespace in which the Instrumentation resource is deployed:

kubectl describe otelinst -n <namespace>

Sample output:

Name:         python-instrumentation
Namespace:    application
Labels:       app.kubernetes.io/managed-by=opentelemetry-operator
Annotations:  instrumentation.opentelemetry.io/default-auto-instrumentation-apache-httpd-image:
               ghcr.io/open-telemetry/opentelemetry-operator/autoinstrumentation-apache-httpd:1.0.3
             instrumentation.opentelemetry.io/default-auto-instrumentation-dotnet-image:
               ghcr.io/open-telemetry/opentelemetry-operator/autoinstrumentation-dotnet:0.7.0
             instrumentation.opentelemetry.io/default-auto-instrumentation-go-image:
               ghcr.io/open-telemetry/opentelemetry-go-instrumentation/autoinstrumentation-go:v0.2.1-alpha
             instrumentation.opentelemetry.io/default-auto-instrumentation-java-image:
               ghcr.io/open-telemetry/opentelemetry-operator/autoinstrumentation-java:1.26.0
             instrumentation.opentelemetry.io/default-auto-instrumentation-nodejs-image:
               ghcr.io/open-telemetry/opentelemetry-operator/autoinstrumentation-nodejs:0.40.0
             instrumentation.opentelemetry.io/default-auto-instrumentation-python-image:
               ghcr.io/open-telemetry/opentelemetry-operator/autoinstrumentation-python:0.39b0
API Version:  opentelemetry.io/v1alpha1
Kind:         Instrumentation
Metadata:
 Creation Timestamp:  2023-07-28T03:42:12Z
 Generation:          1
 Resource Version:    3385
 UID:                 646661d5-a8fc-4b64-80b7-8587c9865f53
Spec:
...
 Exporter:
   Endpoint:  http://demo-collector.opentelemetry.svc.cluster.local:4318
...
 Propagators:
   tracecontext
   baggage
 Python:
   Image:  ghcr.io/open-telemetry/opentelemetry-operator/autoinstrumentation-python:0.39b0
   Resource Requirements:
     Limits:
       Cpu:     500m
       Memory:  32Mi
     Requests:
       Cpu:     50m
       Memory:  32Mi
 Resource:
 Sampler:
Events:  <none>

Do the OTel Operator logs show any auto-instrumentation errors?

Check the OTel Operator logs for any errors pertaining to auto-instrumentation by running this command:

kubectl logs -l app.kubernetes.io/name=opentelemetry-operator --container manager -n opentelemetry-operator-system --follow

Were the resources deployed in the right order?

Order matters! The Instrumentation resource needs to be deployed before deploying the application, otherwise the auto-instrumentation won’t work.

Recall the auto-instrumentation annotation:

annotations:
  instrumentation.opentelemetry.io/inject-python: 'true'

When the pod starts up, the annotation above tells the OTel Operator to look for an Instrumentation object in the pod’s namespace. It also tells the Operator to inject Python auto-instrumentation into the pod.

It adds an init-container to the application’s pod, called opentelemetry-auto-instrumentation, which is then used to inject the auto-instrumentation into the app container.

If the Instrumentation resource isn’t present by the time the application is deployed, however, the init-container can’t be created. Therefore, if the application is deployed before deploying the Instrumentation resource, the auto-instrumentation will fail.

To make sure that the opentelemetry-auto-instrumentation init-container has started up correctly (or has even started up at all), run the following command:

kubectl get events -n <your_app_namespace>

Which should output something like this:

53s         Normal   Created             pod/py-otel-server-7f54bf4cbc-p8wmj    Created container opentelemetry-auto-instrumentation
53s         Normal   Started             pod/py-otel-server-7f54bf4cbc-p8wmj    Started container opentelemetry-auto-instrumentation

If the output is missing Created and/or Started entries for opentelemetry-auto-instrumentation, then it means that there is an issue with your auto-instrumentation. This can be the result of any of the following:

The Instrumentation resource wasn’t installed (or wasn’t installed properly).
The Instrumentation resource was installed after the application was deployed.
There’s an error in the auto-instrumentation annotation, or the annotation in the wrong spot — see #4 below.

Be sure to check the output of kubectl get events for any errors, as these might help point to the issue.

Is the auto-instrumentation annotation correct?

Sometimes auto-instrumentation can fail due to errors in the auto-instrumentation annotation.

Here are a few things to check for:

Is the auto-instrumentation for the right language?
- For example, when instrumenting a Python application, make sure that the annotation doesn’t incorrectly say instrumentation.opentelemetry.io/inject-java: "true" instead.
- For Deno, make sure you are using the instrumentation.opentelemetry.io/inject-sdk: "true" annotation, rather than an annotation containing the string deno.
Is the auto-instrumentation annotation in the correct location? When defining a Deployment, annotations can be added in one of two locations: spec.metadata.annotations, and spec.template.metadata.annotations. The auto-instrumentation annotation needs to be added to spec.template.metadata.annotations, otherwise it won’t work.

Was the auto-instrumentation endpoint configured correctly?

The spec.exporter.endpoint attribute of the Instrumentation resource defines where to send data to. This can be an OTel Collector, or any OTLP endpoint. If this attribute is left out, it defaults to http://localhost:4317, which, most likely won’t send telemetry data anywhere.

When sending telemetry to an OTel Collector located in the same Kubernetes cluster, spec.exporter.endpoint should reference the name of the OTel Collector Service.

For example:

spec:
  exporter:
    endpoint: http://demo-collector.opentelemetry.svc.cluster.local:4317

Here, the Collector endpoint is set to http://demo-collector.opentelemetry.svc.cluster.local:4317, where demo-collector is the name of the OTel Collector Kubernetes Service. In the above example, the Collector is running in a different namespace from the application, which means that opentelemetry.svc.cluster.local must be appended to the Collector’s service name, where opentelemetry is the namespace in which the Collector resides.

মূল্যায়ন

এই পৃষ্ঠাটি কি সহায়ক ছিল?

Thank you. Your feedback is appreciated!

Please let us know how we can improve this page. Your feedback is appreciated!

সর্বশেষ পরিবর্তিত May 6, 2026: docs(operator): copy sampler and propagation config from operator README (#9684) (e6937ea7)

Injecting Auto-instrumentation

Installation

Create an OpenTelemetry Collector (Optional)

Configure Automatic Instrumentation

.NET

Excluding auto-instrumentation

Learn more

Deno

Configuration options

Learn more

Go

Java

Excluding auto-instrumentation

Learn more

Node.js

Excluding instrumentation libraries

Learn more

Python

Auto-instrumenting Python logs

Excluding auto-instrumentation

Learn more

Add annotations to existing deployments

Opt-in a Go Service

Auto-instrumenting a Python musl based container

Multi-container pods

Single instrumentation

Instrumenting init containers

Multiple instrumentations

Using customized or vendor instrumentation

Using Apache HTTPD auto-instrumentation

Using NGINX auto-instrumentation

Inject OpenTelemetry SDK environment variables only

Controlling Instrumentation Capabilities

Configure resource attributes

Configure resource attributes with annotations

Configure resource attributes with labels

Priority for setting resource attributes

How resource attributes are calculated from pod metadata

How service.name is calculated

How service.version is calculated

How service.instance.id is calculated

How service.namespace is calculated

Troubleshooting

Did the Instrumentation resource install?

Do the OTel Operator logs show any auto-instrumentation errors?

Were the resources deployed in the right order?

Is the auto-instrumentation annotation correct?

Was the auto-instrumentation endpoint configured correctly?

মূল্যায়ন

How `service.name` is calculated

How `service.version` is calculated

How `service.instance.id` is calculated

How `service.namespace` is calculated