Configuration data
The configuration data found in the environment variables is defined in the following table.
| Fields | Spec |
|---|---|
| project_ID | String The identifier of the project to which observability data is sent. If empty, the gRPC Observability plugin attempt to fetch the project ID from the environment variables, or from the default credentials. If not found, the observability init functions return an error. |
| cloud_logging | Object The logging options are categorized in this table. When absent, logging is disabled. |
| cloud_logging.client_rpc_events[] | List A list of client_rpc_events configs, representing the config for
outgoing RPCs from the binary.The client_rpc_events configs are evaluated in text order, the
first one matched is used. If an RPC doesn't match an entry, it
continues to the next entry in the list. |
| cloud_logging.client_rpc_events[].methods[] | List [String] A list of method identifiers. By default, the list is empty, matching no methods. The value of the method is in the form of [service]/[method].* is accepted as a wildcard for:
The service name, when specified, must be the fully qualified service name, including the package name. Examples:
|
| cloud_logging.client_rpc_events[].exclude |
Bool Whether the methods denoted by the client_rpc_events[].methods[] should be excluded from logging. The default value is false, meaning the methods denoted by the client_rpc_events[].methods[] are included in the log record. If the value is true, the wildcard (*) cannot be used as the whole value in the client_rpc_events[].methods[]. |
| cloud_logging.client_rpc_events[].max_metadata_bytes | Int Maximum number of bytes of metadata to log. If the size of the metadata is greater than the defined limit, key-value pairs that exceed the limit are not logged. The default value is 0, meaning no metadata are logged. |
| cloud_logging.client_rpc_events[].max_message_bytes | Int Maximum number of bytes of each message to log. If the size of the message is greater than the defined limit, content that exceeds the limit is truncated. The default value is 0, meaning no message payload is logged. |
| cloud_logging.server_rpc_events[] | List A list of server_rpc_events configs, represents the config for
incoming RPCs to the binary.The server_rpc_events configs are evaluated in text order, the first one matched is used. If an RPC doesn't match an entry, it continues to the next entry in the list. |
| cloud_logging.server_rpc_events[].methods[] | List [String] A list of strings which can select a group of methods. By default, the list is empty, matching no methods. The value of the method is in the form of [service]/[method].* is accepted as a wildcard for:
The service name, when specified, must be the fully qualified service name, including the package name. Examples:
|
| cloud_logging.server_rpc_events[].exclude | Bool Whether the methods denoted by the server_rpc_events[].methods[] should be excluded from logging. The default value is false, meaning that the methods denoted by the server_rpc_events[].methods[] are logged. If the value is true, the wildcard (*) cannot be used as the whole value in any entry of server_rpc_events[].methods[]. |
| cloud_logging.server_rpc_events[].max_metadata_bytes |
Int Maximum number of bytes of metadata to log. If the size of the metadata is greater than the defined limit, key-value pairs that exceed the limit are not logged. The default value is 0, meaning no metadata are logged. |
| cloud_logging.server_rpc_events[].max_message_bytes |
Int Maximum number of bytes of each message to log. If the size of the message is greater than the defined limit, content that passes the limit is truncated. The default value is 0, meaning no message payload is logged. |
| cloud_monitoring | Object Enables Monitoring. There are no configuration options. If you provide an empty configuration objection, monitoring is enabled. If you do not provide a configuration object, monitoring is disabled. For example, when no other options are specified, an empty config section enables monitoring.
export GRPC_GCP_OBSERVABILITY_CONFIG='{
"project_id": "your-project-here",
"cloud_monitoring": {
}
}'
|
| cloud_trace | Object An empty config section enables tracing with the default configuration options. If you do not provide a configuration object, tracing is disabled. For example, an empty config section enables tracing with default configuration options.
export GRPC_GCP_OBSERVABILITY_CONFIG='{
"project_id": "your-project-here",
"cloud_trace": {
}
}'
When tracing is enabled, even with a `0` sampling rate, the decision to sample a particular trace is propagated. |
| cloud_trace.sampling_rate | Number The global setting that controls the probability of an RPC being traced. For example:
By default, the sampling_rate is 0.The plugin respects the sampling decision upstream. If an RPC is chosen for sampling upstream, the plugin collects spans and uploads the data to the backend, regardless of the sampling rate setting for the plugin. |
| labels | Object A JSON object containing a set of key-value pairs. Both key and value are strings. Labels are applied on Logging, Monitoring, and Cloud Trace together. |
Trace definitions
This section provides information on tracing.
Trace context propagation
For cross-service tracing to work, the service owner must support the propagation of trace context received from upstream (or started by itself) to downstream. Trace context is propagated among services through gRPC metadata. Make sure that you enable the Monitoring, Logging, Trace APIs, and Microservices APIs, which let services in this configuration to report their telemetry data to the appropriate service.
Without propagation support, downstream services can't generate spans for a trace. Existing spans are not affected. The Microservices observability plugins support the OpenCensus Binary Format for encoding and encoding trace context.
Spans
The name of a span is formatted as follows:
| Type | Example value | Usage |
|---|---|---|
| RPC span | [Sent|Recv].helloworld.Greeter.SayHello |
The span name is the full method name, connected by dots, with no
prefix slash. Span names are prefixed with Sent. for CLIENT RPC span
and Recv. for SERVER RPC span in front of the full
method name. |
| Attempt span | Attempt.helloworld.Greeter.SayHello |
Attaching a prefix Attempt. in front of the full method
name. |
Attempt spans are not available to the Microservices observability Go integration.
Span labels
The integrations provide different span labels.
For attempt spans, two additional retry-related attributes (span labels) are attached:
| Label | Example value | Usage |
|---|---|---|
| previous-rpc-attempts | 0 |
The retry attempts count before this RPC. |
| transparent-retry | True/False |
Whether this RPC is initiated by a transparent retry. |
Metrics definitions
The following metrics are available and are displayed in a dashboard named Microservices (gRPC) for common user journeys:
| Metric Name | Notes |
|---|---|
custom.googleapis.com/opencensus/grpc.io/client/started_rpcs |
The total number of client RPCs ever opened, including RPCs that have
not completed. is tagged with grpc_client_method. |
custom.googleapis.com/opencensus/grpc.io/client/completed_rpcs |
The total number of client RPCs completed, for example, when a
response is received (or sent by the server), is tagged with
grpc_client_method and grpc_client_status. |
The following metrics from the gRPC server side are available:
Metric Name |
Notes |
|---|---|
custom.googleapis.com/opencensus/grpc.io/server/started_rpcs |
The total number of RPCs ever received at the server, including RPCs that have not completed. is tagged with grpc_server_method. |
custom.googleapis.com/opencensus/grpc.io/server/completed_rpcs |
The total number of RPCs completed, for example, when a response is sent by the server, is tagged with grpc_server_method
and grpc_server_status. |
Location tags
Location tags identify the source that generates observability data. Each location tag is a key-value pair, where keys are predefined values that are specific to the source environment (for example, GKE or Compute Engine). For metrics and tracing on GKE deployments, location tags are populated by default except for the container name and namespace name. The missing values can be populated in the Deployment file using the Downward API.
The following are the environment variable keys:
- CONTAINER_NAME
- NAMESPACE
For example, the env section in the following includes a location tag:
apiVersion: apps/v1
kind: Deployment
metadata:
labels:
run: app1
name: app1
spec:
replicas: 2
selector:
matchLabels:
run: app1
template:
metadata:
labels:
run: app1
spec:
containers:
- image: 'o11y-examples:1.00'
name: container1
ports:
- protocol: TCP
containerPort: 50051
env:
- name: CONTAINER_NAME
value: container1
- name: NAMESPACE
valueFrom:
fieldRef:
fieldPath: metadata.namespace
gRPC log record definitions
The Microservices observability logs are uploaded to Logging
using the log name (PROJECT_ID is the placeholder for the string representing
your project):
logName=projects/[PROJECT_ID]/logs/microservices.googleapis.com%2Fobservability%2Fgrpc
The following is the JSON representation of the generated log record:
{
"authority": string,
"callId": string,
"type": string,
"logger": string,
"serviceName": string,
"methodName": string,
"peer": {
"type": string,
"address": string,
"ipPort": int
},
"payload": {
"timeout": string,
"metadata":
{
string: string,
string: string
},
"status_code": int,
"status_message": string,
"status_details": string,
"message": string,
"messageLength": int,
},
"sequenceId": int
}
The following table describes the fields in the log entry:
| Fields | Spec |
|---|---|
| authority | String A single process may be used to run multiple virtual servers with different identities. The authority is the name of such a server identity. It's typically a portion of the URI in the form of host or host:port |
| callId | String Uniquely identifies a [client/server] call that is an UUID. Each call can have several log entries. They all have the same callId. |
| type | String The type of the log event. Types of event are: EVENT_TYPE_UNKNOWN, CLIENT_HEADER, SERVER_HEADER, CLIENT_MESSAGE, SERVER_MESSAGE, CLIENT_HALF_CLOSE, SERVER_TRAILER, CANCEL |
| logger | String The type of the event logger. Types of event logger are: LOGGER_UNKNOWN, CLIENT, SERVER |
| serviceName | String The name of the service. |
| methodName | String The name of the RPC method. |
| peer | Object Peer address information. On the client side, peer is logged on server header events and trailer events. On the server side, peer is always logged on the client header event. |
| peer.type | String The type of the address, whether it's IPv4, IPv6, or UNIX. |
| peer.address | String The content of the address. |
| peer.ip_port | Int The port number for the address. Only available for IPv4 and IPv6 addresses. |
| payload | Object Payload can include a combination of metadata, timeout, message, and status depending on the event. For message events, payload is actual data passed as client/server messages and length of message. For header events, payload includes the header name and value. For trailer events, payload includes status details along with trailer metadata (if present). For client header events, if timeout is set, payload includes timeout as well. |
| payload.timeout | String A string representing google.protobuf.Duration, like
"1.2 s".The RPC timeout value. |
| payload.metadata | Mapping[String, String] Used by header event or trailer event. |
| payload.message | String (Bytes) The message payload. |
| payload.messageLength | Int Size of the message, regardless of whether the full message is being logged (for example, they could be truncated or omitted). |
| payload.statusCode | Int The gRPC status code. |
| payload.statusMessage | String The gRPC status message. |
| payload.statusDetails | String The value of the grpc-status-details-bin metadata key,
if any. This is always an encoded google.rpc.Status
message. |
| payloadTruncated | Bool True if the message or metadata field is either truncated or omitted due to config options. |
| sequenceId | Int The message sequence ID for this call. The first message has a value of 1, to disambiguate from an unset value. The purpose of this field is to detect missing entries in environments where durability or ordering is not guaranteed. |
Custom labels
Custom labels represent additional user-provided information in the observability data. Labels consist of a key and a value. The key-value pair is attached to tracing data as span labels, to metrics data as metrics labels, and to logging data as log entry labels. All custom labels are of the STRING type.
You can provide custom labels in the configuration by specifying a list of
key-value pairs for labels. The implementation reads the configuration
and creates a separate label for each key-value pair, then attaches the label to the
observability data:
"labels": {
"DATACENTER": "SAN_JOSE_DC",
"APP_ID": "24512"
}
Each log entry has the following additional labels in them:
{
"DATACENTER": "SAN_JOSE_DC"
"APP_ID": "24512"
}