Semantic conventions for RPC spans
Status: Stable, unless otherwise specified
This document defines how to describe remote procedure calls (also called “remote method invocations” / “RMI”) with spans.
Existing RPC instrumentations that are using v1.37.0 of this document (or prior):
- SHOULD NOT change the version of the RPC conventions that they emit by default in their existing major version. Conventions include, but are not limited to, attributes, metric and span names, span kind, and unit of measure.
- SHOULD introduce an environment variable
OTEL_SEMCONV_STABILITY_OPT_INin their existing major version as a comma-separated list of category-specific values (e.g.,http,database,rpc). The list of values includes:rpc- emit the stable RPC conventions, and stop emitting the experimental RPC conventions that the instrumentation emitted previously.rpc/dup- emit both the experimental and stable RPC conventions, allowing for a phased rollout of the stable semantic conventions.- The default behavior (in the absence of one of these values) is to continue emitting whatever version of the old experimental RPC conventions the instrumentation was emitting previously.
rpc/duptakes precedence overrpcif both values are present.
- SHOULD maintain (security patching at a minimum) their existing major version for at least six months after it starts emitting both sets of conventions.
- MAY drop the environment variable in their next major version and emit only the stable RPC conventions.
OTEL_SEMCONV_STABILITY_OPT_IN is only intended to be used when migrating
from experimental semantic conventions to their initial stable version.
Name
RPC spans MUST follow the overall guidelines for span names.
The span name SHOULD be {rpc.method} if it is available and not set to
_OTHER.
If rpc.method is unavailable or set to _OTHER, the span name SHOULD be
{rpc.system.name}.
Semantic conventions for individual RPC systems MAY specify different span name format.
RPC client span
Status:
This span represents an outgoing Remote Procedure Call (RPC).
RPC client spans SHOULD cover the entire client-side lifecycle of an RPC, starting when the RPC is initiated and ending when the response is received or the RPC is terminated due to an error or cancellation.
For streaming RPCs, the span covers the full lifetime of the request and/or response streams until they are closed or terminated.
If a transient issue happened and was retried within this RPC, the corresponding span SHOULD cover the duration of the logical call with all retries.
Span name: refer to the Span Name section.
Span kind MUST be CLIENT.
Span status: refer to the Recording Errors document for details on how to record span status.
Attributes:
| Key | Stability | Requirement Level | Value Type | Description | Example Values |
|---|---|---|---|---|---|
rpc.system.name |  | Required | string | The Remote Procedure Call (RPC) system. [1] | grpc; dubbo; connectrpc (see more) |
error.type | | Conditionally Required If and only if the operation failed. | string | Describes a class of error the operation ended with. [2] | DEADLINE_EXCEEDED; java.net.UnknownHostException; -32602 |
rpc.method | | Conditionally Required if available. | string | The fully-qualified logical name of the method from the RPC interface perspective. [3] | com.example.ExampleService/exampleMethod; EchoService/Echo; _OTHER |
rpc.method_original | | Conditionally Required If and only if it’s different than rpc.method. | string | The original name of the method used by the client. | com.myservice.EchoService/catchAll; com.myservice.EchoService/unknownMethod; InvalidMethod |
rpc.status_code | | Conditionally Required if available. | string | Status code of the RPC returned by the RPC server or generated by the client [4] | OK; DEADLINE_EXCEEDED; -32602 |
server.address | | Conditionally Required If available. | string | A string identifying a group of RPC server instances request is sent to. [5] | example.com; 10.1.2.80; /tmp/my.sock |
server.port | | Conditionally Required if applicable and if server.address is set. | int | Server port number. [6] | 80; 8080; 443 |
network.peer.address | | Recommended | string | Peer address of the network connection - IP address or UNIX domain socket name. [7] | 10.1.2.80; /tmp/my.sock |
network.peer.port | | Recommended If network.peer.address is set. | int | Peer port number of the network connection. | 65123 |
rpc.request.header.<key> |  | Opt-In | string[] | RPC request headers or metadata, <key> being the normalized header name (lowercase), the value being the header values. [8] | ["1.2.3.4", "1.2.3.5"] |
rpc.response.header.<key> | | Opt-In | string[] | RPC response headers or metadata sent before the response payload, <key> being the normalized header name (lowercase), the value being the header values. [9] | ["attribute_value"] |
rpc.response.trailer.<key> | | Opt-In | string[] | RPC response trailers or metadata sent after the response payload, <key> being the normalized trailer name (lowercase), the value being the trailer values. [10] | ["attribute_value"] |
[1] rpc.system.name: The client and server RPC systems may differ for the same RPC interaction. For example, a client may use Apache Dubbo or Connect RPC to communicate with a server that uses gRPC since both protocols provide compatibility with gRPC.
[2] error.type: If the RPC fails with an error before status code is returned,
error.type SHOULD be set to the exception type (its fully-qualified class name, if applicable)
or a component-specific, low cardinality error identifier.
If a status code is returned and it indicates an error,
error.type SHOULD be set to that status code. Check system-specific conventions
for the details on which values of rpc.status_code are considered errors.
The error.type value SHOULD be predictable and SHOULD have low cardinality.
Instrumentations SHOULD document the list of errors they report.
If the request has completed successfully, instrumentations SHOULD NOT set
error.type.
[3] rpc.method: The method name MAY have unbounded cardinality in edge or error cases.
Some RPC frameworks or libraries provide a fixed set of recognized methods for client stubs and server implementations. Instrumentations for such frameworks MUST set this attribute to the original method name only when the method is recognized by the framework or library.
When the method is not recognized, for example, when the server receives
a request for a method that is not predefined on the server, or when
instrumentation is not able to reliably detect if the method is predefined,
the attribute MUST be set to _OTHER.
If the RPC instrumentation could end up converting valid RPC methods to
_OTHER, then it SHOULD provide a way to configure the list of recognized
RPC methods.
The rpc.method can be different from the name of any implementing
method/function.
The code.function.name attribute may be used to record the fully-qualified
method actually executing the call on the server side, or the
RPC client stub method on the client side.
[4] rpc.status_code: Usually it represents an error code, but may also represent partial success, warning, or differentiate between various types of successful outcomes.
The value MAY use a standardized string representation of a numeric status code, if one exists, instead of the numeric code. The value MAY be a synthetic status provided by the client library rather than returned by the remote service.
Semantic conventions for individual RPC frameworks SHOULD document what rpc.status_code means in the context of that system and which values are considered to represent errors.
[5] server.address: May contain a DNS name, an endpoint and path in the service registry, local socket name or an IP address.
Semantic conventions for individual RPC systems SHOULD document how to populate this attribute.
When address is an IP address, instrumentations SHOULD NOT do a reverse DNS lookup to obtain a DNS name and SHOULD set server.address to the provided IP address.
[6] server.port: When observed from the client side, and when communicating through an intermediary, server.port SHOULD represent the server port behind any intermediaries, for example proxies, if it’s available.
[7] network.peer.address: If a RPC involved multiple network calls (for example retries), the last contacted address SHOULD be used.
[8] rpc.request.header.<key>: Instrumentations SHOULD require an explicit configuration of which headers are to be captured.
Including all request headers can be a security risk - explicit configuration helps avoid leaking sensitive information.
For example, a header my-custom-key with value ["1.2.3.4", "1.2.3.5"] SHOULD be recorded as
the rpc.request.header.my-custom-key attribute with value ["1.2.3.4", "1.2.3.5"]
The attribute value MUST consist of either multiple header values as an array of strings or a single-item array containing a possibly comma-concatenated string, depending on the way the RPC library provides access to headers.
Values that the RPC system exposes as binary data, or that are otherwise not valid UTF-8 strings, SHOULD be base64-encoded when recorded.
[9] rpc.response.header.<key>: Instrumentations SHOULD require an explicit configuration of which headers are to be captured.
Including all response headers can be a security risk - explicit configuration helps avoid leaking sensitive information.
For example, a header my-custom-key with value ["attribute_value"] SHOULD be recorded as
the rpc.response.header.my-custom-key attribute with value ["attribute_value"]
The attribute value MUST consist of either multiple header values as an array of strings or a single-item array containing a possibly comma-concatenated string, depending on the way the RPC library provides access to headers.
Values that the RPC system exposes as binary data, or that are otherwise not valid UTF-8 strings, SHOULD be base64-encoded when recorded.
[10] rpc.response.trailer.<key>: Instrumentations SHOULD require an explicit configuration of which trailers are to be captured.
Including all response trailers can be a security risk - explicit configuration helps avoid leaking sensitive information.
For example, a trailer my-custom-key with value ["attribute_value"] SHOULD be recorded as
the rpc.response.trailer.my-custom-key attribute with value ["attribute_value"]
The attribute value MUST consist of either multiple trailer values as an array of strings or a single-item array containing a possibly comma-concatenated string, depending on the way the RPC library provides access to trailers.
Values that the RPC system exposes as binary data, or that are otherwise not valid UTF-8 strings, SHOULD be base64-encoded when recorded.
The following attributes can be important for making sampling decisions and SHOULD be provided at span creation time (if provided at all):
rpc.system.name has the following list of well-known values. If one of them applies, then the respective value MUST be used; otherwise, a custom value MAY be used.
| Value | Description | Stability |
|---|---|---|
connectrpc | Connect RPC |  |
dubbo | Apache Dubbo | |
grpc | gRPC | |
jsonrpc | JSON-RPC | |
RPC server span
Status:
This span represents an incoming Remote Procedure Call (RPC).
RPC server spans SHOULD cover the entire server-side lifecycle of an RPC, starting when the request is received and ending when the response is sent or the RPC is terminated due to an error or cancellation.
For streaming RPCs, the span SHOULD cover the full lifetime of the request and/or response streams until they are closed or terminated.
Span name: refer to the Span Name section.
Span kind MUST be SERVER.
Span status: refer to the Recording Errors document for details on how to record span status.
Attributes:
| Key | Stability | Requirement Level | Value Type | Description | Example Values |
|---|---|---|---|---|---|
rpc.system.name | | Required | string | The Remote Procedure Call (RPC) system. [1] | grpc; dubbo; connectrpc (see more) |
error.type | | Conditionally Required If and only if the operation failed. | string | Describes a class of error the operation ended with. [2] | DEADLINE_EXCEEDED; java.net.UnknownHostException; -32602 |
rpc.method | | Conditionally Required if available. | string | The fully-qualified logical name of the method from the RPC interface perspective. [3] | com.example.ExampleService/exampleMethod; EchoService/Echo; _OTHER |
rpc.method_original | | Conditionally Required If and only if it’s different than rpc.method. | string | The original name of the method used by the client. | com.myservice.EchoService/catchAll; com.myservice.EchoService/unknownMethod; InvalidMethod |
rpc.status_code | | Conditionally Required if available. | string | Status code of the RPC returned by the RPC server or generated by the client [4] | OK; DEADLINE_EXCEEDED; -32602 |
server.port | | Conditionally Required if applicable and if server.address is set. | int | Server port number. [5] | 80; 8080; 443 |
network.peer.address | | Recommended | string | Peer address of the network connection - IP address or UNIX domain socket name. [6] | 10.1.2.80; /tmp/my.sock |
network.peer.port | | Recommended If network.peer.address is set. | int | Peer port number of the network connection. | 65123 |
server.address | | Recommended when available. | string | A string identifying a group of RPC server instances request is sent to. [7] | example.com; 10.1.2.80; /tmp/my.sock |
rpc.request.header.<key> | | Opt-In | string[] | RPC request headers or metadata, <key> being the normalized header name (lowercase), the value being the header values. [8] | ["1.2.3.4", "1.2.3.5"] |
rpc.response.header.<key> | | Opt-In | string[] | RPC response headers or metadata sent before the response payload, <key> being the normalized header name (lowercase), the value being the header values. [9] | ["attribute_value"] |
rpc.response.trailer.<key> | | Opt-In | string[] | RPC response trailers or metadata sent after the response payload, <key> being the normalized trailer name (lowercase), the value being the trailer values. [10] | ["attribute_value"] |
[1] rpc.system.name: The client and server RPC systems may differ for the same RPC interaction. For example, a client may use Apache Dubbo or Connect RPC to communicate with a server that uses gRPC since both protocols provide compatibility with gRPC.
[2] error.type: If the RPC fails with an error before status code is returned,
error.type SHOULD be set to the exception type (its fully-qualified class name, if applicable)
or a component-specific, low cardinality error identifier.
If a status code is returned and it indicates an error,
error.type SHOULD be set to that status code. Check system-specific conventions
for the details on which values of rpc.status_code are considered errors.
The error.type value SHOULD be predictable and SHOULD have low cardinality.
Instrumentations SHOULD document the list of errors they report.
If the request has completed successfully, instrumentations SHOULD NOT set
error.type.
[3] rpc.method: The method name MAY have unbounded cardinality in edge or error cases.
Some RPC frameworks or libraries provide a fixed set of recognized methods for client stubs and server implementations. Instrumentations for such frameworks MUST set this attribute to the original method name only when the method is recognized by the framework or library.
When the method is not recognized, for example, when the server receives
a request for a method that is not predefined on the server, or when
instrumentation is not able to reliably detect if the method is predefined,
the attribute MUST be set to _OTHER.
If the RPC instrumentation could end up converting valid RPC methods to
_OTHER, then it SHOULD provide a way to configure the list of recognized
RPC methods.
The rpc.method can be different from the name of any implementing
method/function.
The code.function.name attribute may be used to record the fully-qualified
method actually executing the call on the server side, or the
RPC client stub method on the client side.
[4] rpc.status_code: Usually it represents an error code, but may also represent partial success, warning, or differentiate between various types of successful outcomes.
The value MAY use a standardized string representation of a numeric status code, if one exists, instead of the numeric code. The value MAY be a synthetic status provided by the client library rather than returned by the remote service.
Semantic conventions for individual RPC frameworks SHOULD document what rpc.status_code means in the context of that system and which values are considered to represent errors.
[5] server.port: When observed from the client side, and when communicating through an intermediary, server.port SHOULD represent the server port behind any intermediaries, for example proxies, if it’s available.
[6] network.peer.address: If a RPC involved multiple network calls (for example retries), the last contacted address SHOULD be used.
[7] server.address: server.address and server.port describe the address the client used to reach this server, as reported by the transport or RPC framework. Instrumentations SHOULD NOT use actual network-level connection information to populate these attributes. If the address used by the client is unavailable, instrumentations SHOULD NOT set these attributes.
[8] rpc.request.header.<key>: Instrumentations SHOULD require an explicit configuration of which headers are to be captured.
Including all request headers can be a security risk - explicit configuration helps avoid leaking sensitive information.
For example, a header my-custom-key with value ["1.2.3.4", "1.2.3.5"] SHOULD be recorded as
the rpc.request.header.my-custom-key attribute with value ["1.2.3.4", "1.2.3.5"]
The attribute value MUST consist of either multiple header values as an array of strings or a single-item array containing a possibly comma-concatenated string, depending on the way the RPC library provides access to headers.
Values that the RPC system exposes as binary data, or that are otherwise not valid UTF-8 strings, SHOULD be base64-encoded when recorded.
[9] rpc.response.header.<key>: Instrumentations SHOULD require an explicit configuration of which headers are to be captured.
Including all response headers can be a security risk - explicit configuration helps avoid leaking sensitive information.
For example, a header my-custom-key with value ["attribute_value"] SHOULD be recorded as
the rpc.response.header.my-custom-key attribute with value ["attribute_value"]
The attribute value MUST consist of either multiple header values as an array of strings or a single-item array containing a possibly comma-concatenated string, depending on the way the RPC library provides access to headers.
Values that the RPC system exposes as binary data, or that are otherwise not valid UTF-8 strings, SHOULD be base64-encoded when recorded.
[10] rpc.response.trailer.<key>: Instrumentations SHOULD require an explicit configuration of which trailers are to be captured.
Including all response trailers can be a security risk - explicit configuration helps avoid leaking sensitive information.
For example, a trailer my-custom-key with value ["attribute_value"] SHOULD be recorded as
the rpc.response.trailer.my-custom-key attribute with value ["attribute_value"]
The attribute value MUST consist of either multiple trailer values as an array of strings or a single-item array containing a possibly comma-concatenated string, depending on the way the RPC library provides access to trailers.
Values that the RPC system exposes as binary data, or that are otherwise not valid UTF-8 strings, SHOULD be base64-encoded when recorded.
The following attributes can be important for making sampling decisions and SHOULD be provided at span creation time (if provided at all):
rpc.system.name has the following list of well-known values. If one of them applies, then the respective value MUST be used; otherwise, a custom value MAY be used.
| Value | Description | Stability |
|---|---|---|
connectrpc | Connect RPC | |
dubbo | Apache Dubbo | |
grpc | gRPC | |
jsonrpc | JSON-RPC | |
Feedback
Was this page helpful?
Thank you. Your feedback is appreciated!
Please let us know how we can improve this page. Your feedback is appreciated!