Observability with proxyless gRPC applications

This guide provides instructions for using gRPC's observability features with services using Traffic Director. These features give you access to monitoring and tracing metrics that help you solve issues with your deployment.

This guide also shows you how to export the information to Cloud Trace and Cloud Monitoring.


To create the examples in this document, you need to do the following:

Some of the observability features require additional command-line parameters to the gRPC applications. The parameters are described in this document. The features and functionality described in this guide are all part of the gRPC library. You do not have to further configure Traffic Director.

Metrics and tracing

gRPC clients and servers are integrated with Open Census to export metrics and traces to a variety of backends, including Trace and Monitoring. This document provides examples in the following gRPC languages:

  • C++
  • Go
  • Java

OpenCensus implementations in other languages might support some of the features described in this document, but we do not guarantee that those features work. See the OpenCensus documentation for details.

When metrics and tracing are enabled, gRPC clients and servers export the views, which are aggregations over a given metric, that are specified in the OpenCensus gRPC metrics specification:

  • grpc.io/{client,server}/sent_bytes_per_rpc
  • grpc.io/{client,server}/received_bytes_per_rpc
  • grpc.io/{client,server}/sent_messages_per_rpc
  • grpc.io/{client,server}/received_messages_per_rpc
  • grpc.io/{client,server}/started_rpcs
  • grpc.io/client/roundtrip_latency
  • grpc.io/server/server_latency

Enabling the Cloud Monitoring and Cloud Trace APIs

gRPC's OpenCensus integration supports a broad range of backends for exporting metrics and traces, and the examples below demonstrate how you export this information to Monitoring and Trace. These are collectively referred to as stackdriver in the dependencies elsewhere in this document.

You must follow the instructions to enable the Monitoring API and the instructions to enable the Trace API for these examples to work and to allow you to visualize metrics and traces on your project's Cloud Console.

Instrumenting gRPC applications

The gRPC wallet example code is already instrumented for metrics and tracing. To enable metrics and tracing, run the gRPC client and servers with the command-line flag --gcp_client_project=YOUR_PROJECT_ID, replacing YOUR_PROJECT_ID with the project ID of your project. If you use the create_service.sh script from the gRPC wallet example repository, as documented in Setting up proxyless gRPC services with advanced traffic management, add this flag as the last argument provided to the shell script.

For example:

./create_service.sh go stats 50052 stats \
    --account_server="xds:///account.grpcwallet.io" \

The code changes required to add instrumentation to an existing gRPC application are below. As a best practice in your own applications, we recommend gating the instrumentation code behind a command-line flag or environment variable, which allows proxyless gRPC deployments to dynamically choose to enable or disable metrics and tracing without requiring additional code changes.

The example code demonstrates how to configure metrics and tracing at the same time. You can also configure them separately, or use one but not the other. The code shows how to configure them together, because the steps for both are similar.


The OpenCensus gRPC C++ plugin requires building with Bazel. You must modify two build files to include the OpenCensus dependencies.

First, add the following to your WORKSPACE file:

load("@com_google_googleapis//:repository_rules.bzl", "switched_rules_by_language")

    name = "com_google_googleapis",
    sha256 = "150be57ff83646e5652e03683c949f0830d9a0e73ef787786864210e45537fe0",
    strip_prefix = "googleapis-6e3b55e26bf5a9f7874b6ba1411a0cc50cb87a48",
   urls =       ["https://github.com/googleapis/googleapis/archive/6e3b55e26bf5a9f7874b6ba1411a0cc50cb87a48.zip"],

    name = "com_google_googleapis_imports",
    cc = True,
    grpc = True,

Add the following dependencies to the build target in your BUILD file:

    name = "client",
    srcs = ["client.cc"],
    defines = ["BAZEL_BUILD"],
    deps = [
        # New dependencies:

Lastly, instrument your C++ application. Clients and servers are both instrumented globally within the process, as follows:

#include <grpcpp/opencensus.h>
#include "opencensus/exporters/stats/stackdriver/stackdriver_exporter.h"
include "opencensus/exporters/trace/stackdriver/stackdriver_exporter.h"

ABSL_FLAG(std::string, gcp_client_project, "", "GCP project for metric/trace uploads");

int main(int argc, char** argv) {
  // Optionally configure OpenCensus before using gRPC
  if (!absl::GetFlag(FLAGS_gcp_client_project).empty()) {
    // For demo purposes, always sample (probability=1.0)
        {128, 128, 128, 128, opencensus::trace::ProbabilitySampler(1.0)});
    opencensus::exporters::trace::StackdriverOptions trace_opts;
    trace_opts.project_id = absl::GetFlag(FLAGS_gcp_client_project);
    opencensus::exporters::stats::StackdriverOptions stats_opts;
    stats_opts.project_id = absl::GetFlag(FLAGS_gcp_client_project);

  // Now use gRPC as usual.


For Go, you instrument a client using a ClientHandler:

import (

if gcpClientProject != "" {
    sd, err := stackdriver.NewExporter(stackdriver.Options{
        ProjectID:    gcpClientProject,
    defer sd.Flush()
    defer sd.StopMetricsExporter()
    trace.ApplyConfig(trace.Config{DefaultSampler: trace.AlwaysSample()})

    channel, err := grpc.Dial(serverAddr, grpc.WithInsecure(),

You instrument a server using a ServerHandler:

import (

if gcpClientProject != "" {
    sd, err := stackdriver.NewExporter(stackdriver.Options{
        ProjectID:    gcpClientProject,
    defer sd.Flush()
    defer sd.StopMetricsExporter()
    trace.ApplyConfig(trace.Config{DefaultSampler: trace.AlwaysSample()})

    srv := grpc.NewServer(grpc.StatsHandler(&ocgrpc.ServerHandler{}))


For Java, add the following dependencies to your build.gradle file:

def opencensusVersion = "0.28.0"
def grpcVersion = "1.35.0"

dependencies {
  // New dependencies:
  compile "io.grpc:grpc-census:{$grpcVersion}",
 runtime "io.opencensus:opencensus-impl:${opencensusVersion}"

For Java clients and servers, adding the dependency on io.grpc:grpc-census automatically adds metric and trace instrumentation to your RPCs. However, you need to specify an OpenCensus exporter so that you get data:

import io.opencensus.contrib.grpc.metrics.RpcViews;
import io.opencensus.exporter.stats.stackdriver.StackdriverStatsConfiguration;
import io.opencensus.exporter.stats.stackdriver.StackdriverStatsExporter;
import io.opencensus.exporter.trace.stackdriver.StackdriverTraceConfiguration;
import io.opencensus.exporter.trace.stackdriver.StackdriverTraceExporter;
import io.opencensus.trace.Tracing;
import io.opencensus.trace.config.TraceConfig;
import io.opencensus.trace.samplers.Samplers;

if (gcpClientProject != "") {
  // Configure export to Monitoring and Trace
  TraceConfig traceConfig = Tracing.getTraceConfig();


Viewing metrics on the Monitoring dashboard

After your applications are instrumented, gRPC metrics are exported to Monitoring and are viewable in Cloud Console's Metrics Explorer.

To view metrics, do the following:

  1. In the Cloud Console, click Go to Monitoring.
  2. In the left-hand tray, click Metrics explorer.
  3. Under Build Your Query, choose VM Instance or GKE Container as the resource type.
  4. In the Metric field, choose any of the metric names listed in Metrics and tracing, such as OpenCensus/grpc.io/client/roundtrip_latency.
  5. In the Aggregator drop-down list, select Mean as the type. You see a graph similar to this:
Metrics explorer (click to enlarge)
Metrics explorer (click to enlarge)

In addition to the standard Monitoring detailed views, such as filtering to view only statistics from a specific instance group, you can break down the stats by grpc_client_method or grpc_server_method, because each metric sent by OpenCensus is annotated with the corresponding gRPC method name.

You can incorporate these metric views into Monitoring dashboards and charts, and you can use them as the basis for automated alerting.

Viewing traces on Trace

After you complete the setup process, your instrumented gRPC clients and servers send traces to Trace. The Trace Overview page in the Cloud Console show you a list of recent traces, and you can select an individual trace to see a breakdown of your traffic, similar to the following:

Trace compatibility with the Envoy proxy

Exporting traces to Trace with Traffic Director and the Envoy proxy, as described in Observability with Envoy, makes use of Envoy's OpenCensus tracer configuration, which allows traces exported from proxyless gRPC applications and Envoy proxies to be fully compatible within a service mesh. For compatibility with proxyless gRPC, the Envoy bootstrap needs to configure the trace context to include the GRPC_TRACE_BIN trace format in its OpenCensusConfig, as shown below:

      name: envoy.tracers.opencensus
        "@type": type.googleapis.com/envoy.config.trace.v2.OpenCensusConfig
        stackdriver_exporter_enabled: "true"
        stackdriver_project_id: "PROJECT_ID"
        incoming_trace_context: ["CLOUD_TRACE_CONTEXT", "GRPC_TRACE_BIN"]
        outgoing_trace_context: ["CLOUD_TRACE_CONTEXT", "GRPC_TRACE_BIN"]

Exposing the admin interface

In some cases, metrics and tracing data are not sufficient for resolving an issue. You might need to look at the configuration or the runtime state of the gRPC library in your application, such as resolver information, the state of connectivity to peers, RPC statistics on a channel, and the configuration received from Traffic Director. To obtain such information, gRPC applications can expose the admin interface on a particular port. You can then query the application to understand how the services are configured and how they are running. In this section, you can find instructions on how to configure the admin interface for applications written in each supported language.

We recommend that you build a separate gRPC server in your application that listens on a port reserved for this purpose. This allows you to access your gRPC applications even when the data ports are inaccessible because of misconfiguration or network issues. We recommend that you expose the admin interface only on localhost or on a Unix domain socket. The admin interface is supported in C++, Java, and Go.

The following code snippets show how to create an admin interface.


In C++, use this code to create an admin interface:

#include <grpcpp/ext/admin_services.h>

grpc::ServerBuilder builder;
builder.AddListeningPort(":50051", grpc::ServerCredentials(...));
std::unique_ptr<grpc::Server> server(builder.BuildAndStart());


In Go, use this code to create an admin interface:

import "google.golang.org/grpc/admin"

lis, err := net.Listen("tcp", ":50051")
if err != nil {
        log.Fatalf("failed to listen: %v", err)
defer lis.Close()
grpcServer := grpc.NewServer(...opts)
cleanup, err := admin.Register(grpcServer)
if err != nil {
        log.Fatalf("failed to register admin services: %v", err)
defer cleanup()

if err := grpcServer.Serve(lis); err != nil {
        log.Fatalf("failed to serve: %v", err)


In Java, use this code to create an admin interface:

import io.grpc.services.AdminInterface;

server = ServerBuilder.forPort(50051)
        .useTransportSecurity(certChainFile, privateKeyFile)

SSH to a VM

The gRPC wallet example already enables the admin interface. You can change the admin interface port by providing the flag --admin-port=[PORT]. The default admin port is localhost:28881.

To debug your gRPC application, you can SSH into one of the VMs that serves the wallet service. This gives you access to the localhost.

# List the Wallet VMs
$ gcloud compute instances list --filter="zone:(us-central1-a)" --filter="name~'grpcwallet-wallet-v2'"
NAME                                       ZONE           MACHINE_TYPE   PREEMPTIBLE  INTERNAL_IP   EXTERNAL_IP    STATUS
grpcwallet-wallet-v2-mig-us-central1-ccl1  us-central1-a  n1-standard-1        RUNNING
grpcwallet-wallet-v2-mig-us-central1-k623  us-central1-a  n1-standard-1       RUNNING
# Pick one of the Wallet VMs to debug
$ gcloud compute ssh grpcwallet-wallet-v2-mig-us-central1-ccl1 --zone=us-central1-a

Installing and using grpcdebug

To access the admin interface, you need a gRPC client that can communicate with the admin services in your gRPC application. In the following examples, you use a tool called grpcdebug that you can download and install on the VM or Pod where your gRPC application is running. The Github repository for grpcdebug is located at grpc-ecosystem/grpcdebug.

The minimum support Golang version is 1.12. The official Golang installation guide is at the Golang site. If you are following the guide to create a Linux VM for the wallet service, you can install Golang 1.16 using these commands:

sudo apt update && sudo apt install -y wget
wget https://golang.org/dl/go1.16.3.linux-amd64.tar.gz
sudo rm -rf /usr/local/go
sudo tar -C /usr/local -xzf go1.16.3.linux-amd64.tar.gz
export PATH=$PATH:/usr/local/go/bin
sudo ln -sf /usr/local/go/bin/go /usr/bin/go
go version
# go version go1.16.3 linux/amd64

You install the grpcdebug tool with the following commands:

go install -v github.com/grpc-ecosystem/grpcdebug@latest
export PATH=$PATH:$(go env GOPATH)/bin

You now how access to the grpcdebug CLI. The help output contains information on supported commands:

$ grpcdebug -h
grpcdebug is a gRPC service admin CLI

  grpcdebug <target address> [flags]  <command>

Available Commands:
  channelz    Display gRPC states in human readable way.
  health      Check health status of the target service (default "").
  help        Help about any command
  xds         Fetch xDS related information.

      --credential_file string        Sets the path of the credential file; used in [tls] mode
  -h, --help                          help for grpcdebug
      --security string               Defines the type of credentials to use [tls, google-default, insecure] (default "insecure")
      --server_name_override string   Overrides the peer server name if non empty; used in [tls] mode
  -t, --timestamp                     Print timestamp as RFC 3339 instead of human readable strings
  -v, --verbose                       Print verbose information for debugging

Use grpcdebug <target address> [command] --help to obtain more information about a particular command.

Using the grpcdebug tool to debug your applications

The following sections describe the services exposed by the admin interface and how to access them.

The grpcdebug tool provides an ssh_config-like config that supports aliasing, hostname rewriting, and connection security setting (insecure/TLS). See grpcdebug/Connect&Security for more information about this advanced feature.


The Channelz service provides access to runtime information about the connections at different levels in the gRPC library of your application. You can use this for live analysis of applications that might have configuration- or network-related issues. The examples below assume that you deployed the gRPC wallet example using the instructions in Setting up proxyless gRPC services with advanced traffic management and that you provided the --admin-port=[PORT] flag.

After you send some RPCs from a test client, as shown in Verifying the configuration, use the following commands to access the Channelz data for gRPC services.

  1. SSH into a VM that is running the Wallet service.
  2. Set up grpcdebug to connect to the running gRPC application.

Note that the default output of grpcdebug is in a console-friendly table format. If you supply the --json flag, the output is encoded as JSON.

The grpcdebug channelz command is used to fetch and present debugging information from the Channelz service. The command works for both gRPC clients and gRPC servers.

For gRPC clients, the commandgrpcdebug channelz channels provides a list of existing channels and some basic information.

$ grpcdebug localhost:28881 channelz channels
Channel ID   Target                               State     Calls(Started/Succeeded/Failed)   Created Time
1            xds:///account.grpcwallet.io:10080   READY     0/0/0                             59 seconds ago
2            trafficdirector.googleapis.com:443   READY     2/0/0                             59 seconds ago
4            xds:///stats.grpcwallet.io:10080     READY     0/0/0                             59 seconds ago

If you need additional information about a particular channel, you can use grpcdebug channelz channel [CHANNEL_ID] to inspect detailed information for that channel. The channel identifier can be the channel ID or the target address, if there is only one target address. A gRPC channel can contain multiple subchannels, which is gRPC's abstraction on top of a TCP connection.

$ grpcdebug localhost:28881 channelz channel 2
Channel ID:        2
Target:            trafficdirector.googleapis.com:443
State:             READY
Calls Started:     2
Calls Succeeded:   0
Calls Failed:      0
Created Time:      10 minutes ago
Subchannel ID   Target                               State     Calls(Started/Succeeded/Failed)   CreatedTime
3               trafficdirector.googleapis.com:443   READY     2/0/0                             10 minutes ago
Severity   Time             Child Ref                      Description
CT_INFO    10 minutes ago                                  Channel Created
CT_INFO    10 minutes ago                                  parsed scheme: ""
CT_INFO    10 minutes ago                                  scheme "" not registered, fallback to default scheme
CT_INFO    10 minutes ago                                  ccResolverWrapper: sending update to cc: {[{trafficdirector.googleapis.com:443  <nil> 0 <nil>}] <nil> <nil>}
CT_INFO    10 minutes ago                                  Resolver state updated: {Addresses:[{Addr:trafficdirector.googleapis.com:443 ServerName: Attributes:<nil> Type:0 Metadata:<nil>}] ServiceConfig:<nil> Attributes:<nil>} (resolver returned new addresses)
CT_INFO    10 minutes ago                                  ClientConn switching balancer to "pick_first"
CT_INFO    10 minutes ago                                  Channel switches to new LB policy "pick_first"
CT_INFO    10 minutes ago   subchannel(subchannel_id:3 )   Subchannel(id:3) created
CT_INFO    10 minutes ago                                  Channel Connectivity change to CONNECTING
CT_INFO    10 minutes ago                                  Channel Connectivity change to READY

You can also inspect detailed information for a subchannel.

$ grpcdebug localhost:28881 channelz subchannel 3
Subchannel ID:     3
Target:            trafficdirector.googleapis.com:443
State:             READY
Calls Started:     2
Calls Succeeded:   0
Calls Failed:      0
Created Time:      12 minutes ago
Socket ID   Local->Remote                           Streams(Started/Succeeded/Failed)   Messages(Sent/Received)
9 >   2/0/0                               214/132

You can retrieve information about TCP sockets.

$ grpcdebug localhost:28881 channelz socket 9
Socket ID:                       9
Address:               >
Streams Started:                 2
Streams Succeeded:               0
Streams Failed:                  0
Messages Sent:                   226
Messages Received:               141
Keep Alives Sent:                0
Last Local Stream Created:       12 minutes ago
Last Remote Stream Created:      a long while ago
Last Message Sent Created:       8 seconds ago
Last Message Received Created:   8 seconds ago
Local Flow Control Window:       65535
Remote Flow Control Window:      966515
Socket Options Name   Value
SO_LINGER             [type.googleapis.com/grpc.channelz.v1.SocketOptionLinger]:{duration:{}}
SO_RCVTIMEO           [type.googleapis.com/grpc.channelz.v1.SocketOptionTimeout]:{duration:{}}
SO_SNDTIMEO           [type.googleapis.com/grpc.channelz.v1.SocketOptionTimeout]:{duration:{}}
TCP_INFO              [type.googleapis.com/grpc.channelz.v1.SocketOptionTcpInfo]:{tcpi_state:1  tcpi_options:7  tcpi_rto:204000  tcpi_ato:40000  tcpi_snd_mss:1408  tcpi_rcv_mss:1408  tcpi_last_data_sent:8212  tcpi_last_data_recv:8212  tcpi_last_ack_recv:8212  tcpi_pmtu:1460  tcpi_rcv_ssthresh:88288  tcpi_rtt:2400  tcpi_rttvar:3012  tcpi_snd_ssthresh:2147483647  tcpi_snd_cwnd:10  tcpi_advmss:1408  tcpi_reordering:3}
Security Model:   TLS
Standard Name:    TLS_AES_128_GCM_SHA256

On the server side, you can use Channelz to inspect your server application's status. For example, you can get the list of servers using the grpcdebug channelz servers command:

$ grpcdebug localhost:28881 channelz servers
Server ID   Listen Addresses    Calls(Started/Succeeded/Failed)   Last Call Started
5           []   9/8/0                             now
6           [[::]:50051]        159/159/0                         4 seconds ago

To obtain more information about a specific server, use the grpcdebug channelz server command. Note that you can inspect server sockets the same way that you inspect client sockets.

$ grpcdebug localhost:28881 channelz server 6
Server Id:           6
Listen Addresses:    [[::]:50051]
Calls Started:       174
Calls Succeeded:     174
Calls Failed:        0
Last Call Started:   now
Socket ID   Local->Remote                            Streams(Started/Succeeded/Failed)   Messages(Sent/Received)
25>    68/68/0                             68/68
26>   54/54/0                             54/54
27>    52/52/0                             52/52

Client Status Discovery Service

The Client Status Discovery Service (CSDS) API is part of the xDS APIs. In a gRPC application, the CSDS service provides access to the configuration (also called the xDS configuration) that it receives from Traffic Director. This allows you to identify and resolve configuration-related issues in your mesh.

The examples below assume that you deployed the gRPC wallet example using the instructions in Setting up proxyless gRPC serviceswith advanced traffic management.

To use CSDS to examine the configuration:

  1. SSH into a VM that is running the Wallet service. Use the instructions in SSH to a VM.
  2. Run the grpcdebug client to show the usage of CSDS. Show all the relevant resources (Listener, Route, Cluster and endpoints) and explain how these relate to the configuration and the channels this application is creating.

To get an overview of config status, run the following command:

grpcdebug localhost:28881 xds status

You see results similar to the following:

Name                                                                    Status    Version               Type                                                                 LastUpdated
account.grpcwallet.io:10080                                             ACKED     1618529574783547920   type.googleapis.com/envoy.config.listener.v3.Listener                3 seconds ago
stats.grpcwallet.io:10080                                               ACKED     1618529574783547920   type.googleapis.com/envoy.config.listener.v3.Listener                3 seconds ago
URL_MAP/830293263384_grpcwallet-url-map_0_account.grpcwallet.io:10080   ACKED     1618529574783547920   type.googleapis.com/envoy.config.route.v3.RouteConfiguration         3 seconds ago
URL_MAP/830293263384_grpcwallet-url-map_1_stats.grpcwallet.io:10080     ACKED     1618529574783547920   type.googleapis.com/envoy.config.route.v3.RouteConfiguration         3 seconds ago
cloud-internal-istio:cloud_mp_830293263384_3566964729007423588          ACKED     1618529574783547920   type.googleapis.com/envoy.config.cluster.v3.Cluster                  3 seconds ago
cloud-internal-istio:cloud_mp_830293263384_7383783194368524341          ACKED     1618529574783547920   type.googleapis.com/envoy.config.cluster.v3.Cluster                  3 seconds ago
cloud-internal-istio:cloud_mp_830293263384_3363366193797120473          ACKED     1618529574783547920   type.googleapis.com/envoy.config.cluster.v3.Cluster                  3 seconds ago
cloud-internal-istio:cloud_mp_830293263384_3566964729007423588          ACKED     86                    type.googleapis.com/envoy.config.endpoint.v3.ClusterLoadAssignment   2 seconds ago
cloud-internal-istio:cloud_mp_830293263384_3363366193797120473          ACKED     86                    type.googleapis.com/envoy.config.endpoint.v3.ClusterLoadAssignment   2 seconds ago
cloud-internal-istio:cloud_mp_830293263384_7383783194368524341          ACKED     86                    type.googleapis.com/envoy.config.endpoint.v3.ClusterLoadAssignment   2 seconds ago

You can find the definition of config status in documentation for the Envoy proxy. Briefly, the status of an xDS resource is one of REQUESTED, DOES_NOT_EXIST, ACKED, or NACKED.

To obtain a raw xDS configuration dump, run the following command:

grpcdebug localhost:28881 xds config

You see a JSON list of the PerXdsConfig object:

  "config":  [
      "node":  {
        "id":  "projects/830293263384/networks/default/nodes/6e98b038-6d75-4a4c-8d35-b0c7a8c9cdde",
        "cluster":  "cluster",
        "metadata":  {
          "INSTANCE_IP":  "",
          "TRAFFICDIRECTOR_GCP_PROJECT_NUMBER":  "830293263384",
          "TRAFFICDIRECTOR_NETWORK_NAME":  "default"
        "locality":  {
          "zone":  "us-central1-a"
        "userAgentName":  "gRPC Go",
        "userAgentVersion":  "1.37.0",
        "clientFeatures":  [
      "xdsConfig":  [
          "listenerConfig":  {
            "versionInfo":  "1618529930989701137",
            "dynamicListeners":  [

If the raw configuration output is too verbose, grpcdebug lets you filter based on specific xDS types. for example:

$ grpcdebug localhost:28881 xds config --type=cds
  "versionInfo":  "1618530076226619310",
  "dynamicActiveClusters":  [
      "versionInfo":  "1618530076226619310",
      "cluster":  {
        "@type":  "type.googleapis.com/envoy.config.cluster.v3.Cluster",
        "name":  "cloud-internal-istio:cloud_mp_830293263384_7383783194368524341",
        "altStatName":  "/projects/830293263384/global/backendServices/grpcwallet-stats-service",
        "type":  "EDS",
        "edsClusterConfig":  {
          "edsConfig":  {
            "ads":  {},
            "initialFetchTimeout":  "15s",

You can also dump the configuration of seberal xDS types at the same time:

$ grpcdebug localhost:28881 xds config --type=lds,eds
  "versionInfo":  "1618530076226619310",
  "dynamicListeners":  [...]
  "dynamicEndpointConfigs":  [...]

What's next