Microservices observability reference information

Stay organized with collections Save and categorize content based on your preferences.

Configuration data

The configuration data found in the environment variables is defined in the following table.

Fields Spec
project_ID String

The identifier of the project to which observability data is sent. If empty, the gRPC Observability plugin attempt to fetch the project ID from the environment variables, or from the default credentials.

If not found, the observability init functions return an error.
cloud_logging Object

The logging options are categorized in this table.
When absent, logging is disabled.
cloud_logging.client_rpc_events[] List

A list of client_rpc_events configs, representing the config for outgoing RPCs from the binary.

The client_rpc_events configs are evaluated in text order, the first one matched is used. If an RPC doesn't match an entry, it continues to the next entry in the list.
cloud_logging.client_rpc_events[].methods[] List [String]

A list of method identifiers.

By default, the list is empty, matching no methods.

The value of the method is in the form of [service]/[method].

* is accepted as a wildcard for:
  • The method name. If the value is [service]/*, it matches all methods in the specified service.
  • The whole value of the field which matches any [service]/[method]. It's not supported when client_rpc_events[].exclude is true.
  • The * wildcard cannot be used on the service name independently, */[method] is not supported.

The service name, when specified, must be the fully qualified service name, including the package name.

Examples:
  • goo.Foo/Bar selects only the method Bar from service goo.Foo, here goo is the package name.
  • goo.Foo/* selects all methods from service goo.Foo
  • * selects all methods from all services.
cloud_logging.client_rpc_events[].exclude Bool

Whether the methods denoted by the client_rpc_events[].methods[] should be excluded from logging.

The default value is false, meaning the methods denoted by the client_rpc_events[].methods[] are included in the log record.

If the value is true, the wildcard (*) cannot be used as the whole value in the client_rpc_events[].methods[].
cloud_logging.client_rpc_events[].max_metadata_bytes Int

Maximum number of bytes of metadata to log. If the size of the metadata is greater than the defined limit, key-value pairs that exceed the limit are not logged.

The default value is 0, meaning no metadata are logged.
cloud_logging.client_rpc_events[].max_message_bytes Int

Maximum number of bytes of each message to log. If the size of the message is greater than the defined limit, content that exceeds the limit is truncated.

The default value is 0, meaning no message payload is logged.
cloud_logging.server_rpc_events[] List

A list of server_rpc_events configs, represents the config for incoming RPCs to the binary.

The server_rpc_events configs are evaluated in text order, the first one matched is used. If an RPC doesn't match an entry, it continues to the next

entry in the list.
cloud_logging.server_rpc_events[].methods[] List [String]

A list of strings which can select a group of methods.

By default, the list is empty, matching no methods.

The value of the method is in the form of [service]/[method].

* is accepted as a wildcard for:
  • The method name. If the value is [service]/*, it matches all methods in the specified service.
  • The whole value of the method which matches any [service]/[method]. It's not supported when server_rpc_events[].exclude is true.
  • The * wildcard cannot be used on the service name independently, */[method] is not supported.

The service name, when specified, must be the fully qualified service name, including the package name.

Examples:
  • goo.Foo/Bar selects only the method Bar from service goo.Foo, here goo is the package name.
  • goo.Foo/* selects all methods from service goo.Foo
  • * selects all methods from all services.
cloud_logging.server_rpc_events[].exclude Bool

Whether the methods denoted by the server_rpc_events[].methods[] should be excluded from logging.

The default value is false, meaning that the methods denoted by the server_rpc_events[].methods[] are logged.

If the value is true, the wildcard (*) cannot be used as the whole value in any entry of server_rpc_events[].methods[].
cloud_logging.server_rpc_events[].max_metadata_bytes Int

Maximum number of bytes of metadata to log. If the size of the metadata is greater than the defined limit, key-value pairs that exceed the limit are not logged.

The default value is 0, meaning no metadata are logged.
cloud_logging.server_rpc_events[].max_message_bytes Int

Maximum number of bytes of each message to log. If the size of the message is greater than the defined limit, content that passes the limit is truncated.

The default value is 0, meaning no message payload is logged.
cloud_monitoring Object

Enables Monitoring. There are no configuration options. If you provide an empty configuration objection, monitoring is enabled. If you do not provide a configuration object, monitoring is disabled.

For example, when no other options are specified, an empty config section enables monitoring.
export GRPC_GCP_OBSERVABILITY_CONFIG='{
    "project_id": "your-project-here",
    "cloud_monitoring": {
    }
}'
cloud_trace
Object

An empty config section enables tracing with the default configuration options. If you do not provide a configuration object, tracing is disabled.

For example, an empty config section enables tracing with default configuration options.
export GRPC_GCP_OBSERVABILITY_CONFIG='{
    "project_id": "your-project-here",
    "cloud_trace": {
    }
}'


When tracing is enabled, even with a `0` sampling rate, the decision to sample a particular trace is propagated.
cloud_trace.sampling_rate Number

The global setting that controls the probability of an RPC being traced. For example:
  • The value 0.05 means that there's a 5% chance for an RPC to be traced.
  • The value 1.0 means that every call is traced.
  • The value 0 means don't start new traces.

By default, the sampling_rate is 0.

The plugin respects the sampling decision upstream. If an RPC is chosen for sampling upstream, the plugin collects spans and uploads the data to the backend, regardless of the sampling rate setting for the plugin.
labels Object

A JSON object containing a set of key-value pairs. Both key and value are strings.

Labels are applied on Logging, Monitoring, and Cloud Trace together.

Trace definitions

This section provides information on tracing.

Trace context propagation

For cross-service tracing to work, the service owner must support the propagation of trace context received from upstream (or started by itself) to downstream. Trace context is propagated among services through gRPC metadata. Make sure that you enable the Monitoring, Logging, Trace APIs, and Microservices APIs, which let services in this configuration to report their telemetry data to the appropriate service.

Without propagation support, downstream services can't generate spans for a trace. Existing spans are not affected. The Microservices observability plugins support the OpenCensus Binary Format for encoding and encoding trace context.

Spans

The name of a span is formatted as follows:

Type Example value Usage
RPC span
[Sent|Recv].helloworld.Greeter.SayHello
The span name is the full method name, connected by dots, with no prefix slash.
Span names are prefixed with Sent. for CLIENT RPC span and Recv. for SERVER RPC span in front of the full method name.
Attempt span
Attempt.helloworld.Greeter.SayHello
Attaching a prefix Attempt. in front of the full method name.

Attempt spans are not available to the Microservices observability Go integration.

Span labels

The integrations provide different span labels.

For attempt spans, two additional retry-related attributes (span labels) are attached:

Label Example value Usage
previous-rpc-attempts
0
The retry attempts count before this RPC.
transparent-retry
True/False
Whether this RPC is initiated by a transparent retry.

Metrics definitions

The following metrics are available and are displayed in a dashboard named Microservices (gRPC) for common user journeys:

Metric Name Notes
custom.googleapis.com/opencensus/grpc.io/client/started_rpcs
The total number of client RPCs ever opened, including RPCs that have not completed. is tagged with grpc_client_method.
custom.googleapis.com/opencensus/grpc.io/client/completed_rpcs
The total number of client RPCs completed, for example, when a response is received (or sent by the server), is tagged with grpc_client_method and grpc_client_status.

The following metrics from the gRPC server side are available:


Metric Name

Notes
custom.googleapis.com/opencensus/grpc.io/server/started_rpcs

The total number of RPCs ever received at the server, including RPCs that have not completed. is tagged with grpc_server_method.
custom.googleapis.com/opencensus/grpc.io/server/completed_rpcs

The total number of RPCs completed, for example, when a response is sent by the server, is tagged with grpc_server_method and grpc_server_status.

Location tags

Location tags identify the source that generates observability data. Each location tag is a key-value pair, where keys are predefined values that are specific to the source environment (for example, GKE or Compute Engine). For metrics and tracing on GKE deployments, location tags are populated by default except for the container name and namespace name. The missing values can be populated in the Deployment file using the Downward API.

The following are the environment variable keys:

  • CONTAINER_NAME
  • NAMESPACE

For example, the env section in the following includes a location tag:

apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    run: app1
  name: app1
spec:
  replicas: 2
  selector:
    matchLabels:
      run: app1
  template:
    metadata:
      labels:
        run: app1
    spec:
      containers:
        - image: 'o11y-examples:1.00'
          name: container1
          ports:
            - protocol: TCP
              containerPort: 50051
          env:
            - name: CONTAINER_NAME
              value: container1
            - name: NAMESPACE
              valueFrom:
                fieldRef:
                  fieldPath: metadata.namespace

gRPC log record definitions

The Microservices observability logs are uploaded to Logging using the log name (PROJECT_ID is the placeholder for the string representing your project):

logName=projects/[PROJECT_ID]/logs/microservices.googleapis.com%2Fobservability%2Fgrpc

The following is the JSON representation of the generated log record:

{
    "authority": string,
    "callId": string,
    "type": string,
    "logger": string,
    "serviceName": string,
    "methodName": string,
    "peer": {
        "type": string,
        "address": string,
        "ipPort": int
    },
    "payload": {
        "timeout": string,
        "metadata":
            {
                string: string,
                string: string
            },
        "status_code": int,
        "status_message": string,
        "status_details": string,
        "message": string,
        "messageLength": int,
    },
    "sequenceId": int
}

The following table describes the fields in the log entry:

Fields Spec
authority String

A single process may be used to run multiple virtual servers with different identities.

The authority is the name of such a server identity. It's typically a portion of the URI in the form of host or host:port
callId String

Uniquely identifies a [client/server] call that is an UUID. Each call can have several log entries. They all have the same callId.
type String

The type of the log event.

Types of event are:
EVENT_TYPE_UNKNOWN,
CLIENT_HEADER,
SERVER_HEADER,
CLIENT_MESSAGE,
SERVER_MESSAGE,
CLIENT_HALF_CLOSE,
SERVER_TRAILER,
CANCEL
logger String

The type of the event logger.

Types of event logger are:
LOGGER_UNKNOWN, CLIENT, SERVER
serviceName String

The name of the service.
methodName String

The name of the RPC method.
peer Object

Peer address information. On the client side, peer is logged on server header events and trailer events. On the server side, peer is always logged on the client header event.
peer.type String

The type of the address, whether it's IPv4, IPv6, or UNIX.
peer.address String

The content of the address.
peer.ip_port Int

The port number for the address. Only available for IPv4 and IPv6 addresses.
payload Object

Payload can include a combination of metadata, timeout, message, and status depending on the event.

For message events, payload is actual data passed as client/server messages and length of message.

For header events, payload includes the header name and value.

For trailer events, payload includes status details along with trailer metadata (if present).

For client header events, if timeout is set, payload includes timeout as well.
payload.timeout String

A string representing google.protobuf.Duration, like "1.2 s".

The RPC timeout value.
payload.metadata Mapping[String, String]

Used by header event or trailer event.
payload.message String (Bytes)

The message payload.
payload.messageLength Int

Size of the message, regardless of whether the full message is being logged (for example, they could be truncated or omitted).
payload.statusCode Int

The gRPC status code.
payload.statusMessage String

The gRPC status message.
payload.statusDetails String

The value of the grpc-status-details-bin metadata key, if any. This is always an encoded google.rpc.Status message.
payloadTruncated Bool

True if the message or metadata field is either truncated or omitted due to config options.
sequenceId Int

The message sequence ID for this call. The first message has a value of 1, to disambiguate from an unset value. The purpose of this field is to detect missing entries in environments where durability or ordering is not guaranteed.

Custom labels

Custom labels represent additional user-provided information in the observability data. Labels consist of a key and a value. The key-value pair is attached to tracing data as span labels, to metrics data as metrics labels, and to logging data as log entry labels. All custom labels are of the STRING type.

You can provide custom labels in the configuration by specifying a list of key-value pairs for labels. The implementation reads the configuration and creates a separate label for each key-value pair, then attaches the label to the observability data:

"labels": {
    "DATACENTER": "SAN_JOSE_DC",
    "APP_ID": "24512"
  }

Each log entry has the following additional labels in them:

{
   "DATACENTER": "SAN_JOSE_DC"
   "APP_ID": "24512"
}