Observability with proxyless gRPC applications
This document provides instructions for using gRPC's observability features with services that use Traffic Director. These features give you access to monitoring and tracing metrics that help you solve issues with your deployment.
It also shows you how to export the information to Cloud Trace and Cloud Monitoring.
Prerequisites
To create the examples in this document, do the following:
- Set up Traffic Director with Compute Engine VMs and proxyless gRPC services.
- Follow the instructions in Configure advanced traffic management with proxyless gRPC services to deploy the gRPC Wallet application.
Some of the observability features require additional command-line parameters to the gRPC applications; these parameters are described in this document. The features and functionality described in this document are all part of the gRPC library. You do not have to further configure Traffic Director.
Metrics and tracing
gRPC clients and servers are integrated with OpenCensus to export metrics and traces to various backends, including Trace and Monitoring. This document provides examples in the following gRPC languages:
- C++
- Go
- Java
OpenCensus implementations in other languages might support some of the features described in this document, but we do not guarantee that those features work. For details, see the OpenCensus documentation.
When metrics and tracing are enabled, gRPC clients and servers export the views—aggregations over a given metric—that are specified in the OpenCensus gRPC metrics specification:
grpc.io/{client,server}/sent_bytes_per_rpcgrpc.io/{client,server}/received_bytes_per_rpcgrpc.io/{client,server}/sent_messages_per_rpcgrpc.io/{client,server}/received_messages_per_rpcgrpc.io/{client,server}/started_rpcsgrpc.io/client/roundtrip_latencygrpc.io/server/server_latency
Enable the Cloud Monitoring and Cloud Trace APIs
gRPC's OpenCensus integration supports a broad range of backends for exporting
metrics and traces. The following examples demonstrate how you export
this information to Monitoring and Trace. These
are collectively referred to as stackdriver in the dependencies elsewhere in
this document.
For these examples to work and to allow you to visualize metrics and traces on your project's Google Cloud console, follow the instructions to enable the Monitoring API and the Cloud Trace API.
Instrument gRPC applications
The gRPC Wallet example code is already instrumented for metrics and tracing. To enable metrics and tracing, run the gRPC client and servers with the following command-line flag:
--gcp_client_project=PROJECT_ID
Replace PROJECT_ID with the project ID of your project.
If you use the create_service.sh script from the
gRPC Wallet example repository as documented in
Configure advanced traffic management with proxyless gRPC services,
add this flag as the last argument provided to the shell script.
For example:
./create_service.sh go stats 50052 stats \
--account_server="xds:///account.grpcwallet.io" \
--gcp_client_project=PROJECT_ID'
The code changes required to add instrumentation to an existing gRPC application follow. As a best practice in your own applications, we recommend gating the instrumentation code behind a command-line flag or environment variable, which allows proxyless gRPC deployments to dynamically choose to enable or disable metrics and tracing without requiring additional code changes.
The example code demonstrates how to configure metrics and tracing at the same time. You can also configure them separately, or use one but not the other. The code shows how to configure them together because the steps for both are similar.
C++
The OpenCensus gRPC C++ plugin requires building with Bazel. You must modify two build files to include the OpenCensus dependencies.
Add the following to your
WORKSPACEfile:load("@com_google_googleapis//:repository_rules.bzl", "switched_rules_by_language") http_archive( name = "com_google_googleapis", sha256 = "150be57ff83646e5652e03683c949f0830d9a0e73ef787786864210e45537fe0", strip_prefix = "googleapis-6e3b55e26bf5a9f7874b6ba1411a0cc50cb87a48", urls = ["https://github.com/googleapis/googleapis/archive/6e3b55e26bf5a9f7874b6ba1411a0cc50cb87a48.zip"], ) switched_rules_by_language( name = "com_google_googleapis_imports", cc = True, grpc = True, )Add the following dependencies to the build target in your
BUILDfile:cc_binary( name = "client", srcs = ["client.cc"], defines = ["BAZEL_BUILD"], deps = [ # New dependencies: "@com_github_grpc_grpc//:grpc_opencensus_plugin", "@io_opencensus_cpp//opencensus/trace:trace", "@io_opencensus_cpp//opencensus/stats:stats", "@io_opencensus_cpp//opencensus/exporters/trace/stackdriver:stackdriver_exporter", "@io_opencensus_cpp//opencensus/exporters/stats/stackdriver:stackdriver_exporter", ], )Instrument your C++ application. Clients and servers are both instrumented globally within the process as follows:
#include <grpcpp/opencensus.h> #include "opencensus/exporters/stats/stackdriver/stackdriver_exporter.h" #include "opencensus/exporters/trace/stackdriver/stackdriver_exporter.h" ABSL_FLAG(std::string, gcp_client_project, "", "Google Cloud project for metric/trace uploads"); int main(int argc, char** argv) { // Optionally configure OpenCensus before using gRPC if (!absl::GetFlag(FLAGS_gcp_client_project).empty()) { grpc::RegisterOpenCensusPlugin(); grpc::RegisterOpenCensusViewsForExport(); // For demo purposes, always sample (probability=1.0) opencensus::trace::TraceConfig::SetCurrentTraceParams( {128, 128, 128, 128, opencensus::trace::ProbabilitySampler(1.0)}); opencensus::exporters::trace::StackdriverOptions trace_opts; trace_opts.project_id = absl::GetFlag(FLAGS_gcp_client_project); opencensus::exporters::trace::StackdriverExporter::Register(std::move(trace_opts)); opencensus::exporters::stats::StackdriverOptions stats_opts; stats_opts.project_id = absl::GetFlag(FLAGS_gcp_client_project); // This must be unique among all processes exporting to Stackdriver stats_opts.opencensus_task = "wallet-server-" + std::to_string(getpid()); opencensus::exporters::stats::StackdriverExporter::Register(std::move(stats_opts)); } // Now use gRPC as usual. }
Go
For Go, follow these steps.
Instrument a client by using a
ClientHandler:import ( "go.opencensus.io/plugin/ocgrpc" "go.opencensus.io/stats/view" "go.opencensus.io/trace" "contrib.go.opencensus.io/exporter/stackdriver" ) if gcpClientProject != "" { view.Register(ocgrpc.DefaultClientViews...) sd, err := stackdriver.NewExporter(stackdriver.Options{ ProjectID: gcpClientProject, }) defer sd.Flush() trace.RegisterExporter(sd) sd.StartMetricsExporter() defer sd.StopMetricsExporter() trace.ApplyConfig(trace.Config{DefaultSampler: trace.AlwaysSample()}) channel, err := grpc.Dial(serverAddr, grpc.WithInsecure(), grpc.WithStatsHandler(new(ocgrpc.ClientHandler))) }Instrument a server by using a
ServerHandler:import ( "go.opencensus.io/plugin/ocgrpc" "go.opencensus.io/stats/view" "go.opencensus.io/trace" "contrib.go.opencensus.io/exporter/stackdriver" ) if gcpClientProject != "" { view.Register(ocgrpc.DefaultServerViews...) sd, err := stackdriver.NewExporter(stackdriver.Options{ ProjectID: gcpClientProject, }) defer sd.Flush() trace.RegisterExporter(sd) sd.StartMetricsExporter() defer sd.StopMetricsExporter() trace.ApplyConfig(trace.Config{DefaultSampler: trace.AlwaysSample()}) srv := grpc.NewServer(grpc.StatsHandler(&ocgrpc.ServerHandler{})) }
Java
For Java, add the following dependencies to your build.gradle file:
def opencensusVersion = "0.28.0"
def grpcVersion = "1.35.0"
dependencies {
// New dependencies:
compile "io.grpc:grpc-census:{$grpcVersion}",
"io.opencensus:opencensus-api:${opencensusVersion}",
"io.opencensus:opencensus-contrib-grpc-metrics:${opencensusVersion}",
"io.opencensus:opencensus-exporter-stats-stackdriver:${opencensusVersion}",
"io.opencensus:opencensus-exporter-trace-stackdriver:${opencensusVersion}"
runtime "io.opencensus:opencensus-impl:${opencensusVersion}"
}
For Java clients and servers, adding the dependency on io.grpc:grpc-census
automatically adds metric and trace instrumentation to your RPCs. However,
you need to specify an OpenCensus exporter so that you get data:
import io.opencensus.contrib.grpc.metrics.RpcViews;
import io.opencensus.exporter.stats.stackdriver.StackdriverStatsConfiguration;
import io.opencensus.exporter.stats.stackdriver.StackdriverStatsExporter;
import io.opencensus.exporter.trace.stackdriver.StackdriverTraceConfiguration;
import io.opencensus.exporter.trace.stackdriver.StackdriverTraceExporter;
import io.opencensus.trace.Tracing;
import io.opencensus.trace.config.TraceConfig;
import io.opencensus.trace.samplers.Samplers;
if (gcpClientProject != "") {
// Configure export to Monitoring and Trace
RpcViews.registerAllViews();
TraceConfig traceConfig = Tracing.getTraceConfig();
traceConfig.updateActiveTraceParams(
traceConfig.getActiveTraceParams().toBuilder()
.setSampler(Samplers.alwaysSample())
.build());
StackdriverStatsExporter.createAndRegister(
StackdriverStatsConfiguration.builder().setProjectId(gcpClientProject).build());
StackdriverTraceExporter.createAndRegister(
StackdriverTraceConfiguration.builder().setProjectId(gcpClientProject).build());
}
View metrics on the Monitoring dashboard
After your applications are instrumented, gRPC metrics are exported to Monitoring and are viewable in the Google Cloud console's Metrics explorer.
Console
To view metrics, do the following:
In the Google Cloud console, go to the Monitoring page.
In the navigation on the left, click Metrics explorer.
Under Build Your Query, choose VM Instance or GKE Container as the resource type.
In the Metric field, choose any of the metric names listed in Metrics and tracing, such as
OpenCensus/grpc.io/client/roundtrip_latency.In the Aggregator drop-down list, select Mean as the type. You see a graph similar to the following:
Metrics explorer (click to enlarge) In addition to the standard Monitoring detailed views, such as filtering to view only statistics from a specific instance group, you can break down the stats by
grpc_client_methodorgrpc_server_methodbecause each metric sent by OpenCensus is annotated with the corresponding gRPC method name.You can incorporate these metric views into Monitoring dashboards and charts, and you can use them as the basis for automated alerting.
View traces on Trace
After you complete the setup process, your instrumented gRPC clients and servers send traces to Trace. The Trace Overview page in the Google Cloud console shows you a list of recent traces. You can select an individual trace to see a breakdown of your traffic, similar to what's described in the following section.
Trace compatibility with the Envoy proxy
Exporting traces to Trace with Traffic Director and the Envoy
proxy, as described in
Observability with Envoy, uses
Envoy's OpenCensus tracer configuration, which allows traces exported from
proxyless gRPC applications and Envoy proxies to be fully compatible within a
service mesh. For compatibility with proxyless gRPC, the Envoy bootstrap needs
to configure the trace context to include the GRPC_TRACE_BIN trace format
in its OpenCensusConfig. For example:
tracing:
http:
name: envoy.tracers.opencensus
typed_config:
"@type": type.googleapis.com/envoy.config.trace.v2.OpenCensusConfig
stackdriver_exporter_enabled: "true"
stackdriver_project_id: "PROJECT_ID"
incoming_trace_context: ["CLOUD_TRACE_CONTEXT", "GRPC_TRACE_BIN"]
outgoing_trace_context: ["CLOUD_TRACE_CONTEXT", "GRPC_TRACE_BIN"]
Expose the admin interface
Sometimes, metrics and tracing data are not sufficient for resolving an issue. You might need to look at the configuration or the runtime state of the gRPC library in your application. This information includes resolver information, the state of connectivity to peers, RPC statistics on a channel, and the configuration received from Traffic Director.
To obtain such information, gRPC applications can expose the admin interface on a particular port. You can then query the application to understand how the services are configured and how they are running. In this section, you can find instructions about how to configure the admin interface for applications written in each supported language.
We recommend that you build a separate gRPC server in your application that
listens on a port reserved for this purpose. This lets you access your gRPC
applications even when the data ports are inaccessible because of
misconfiguration or network issues. We recommend that you expose the admin
interface only on localhost or on a Unix domain socket.
The following code snippets show how to create an admin interface.
C++
In C++, use this code to create an admin interface:
#include <grpcpp/ext/admin_services.h>
grpc::ServerBuilder builder;
grpc::AddAdminServices(&builder);
builder.AddListeningPort(":50051", grpc::ServerCredentials(...));
std::unique_ptr<grpc::Server> server(builder.BuildAndStart());
Go
In Go, use this code to create an admin interface:
import "google.golang.org/grpc/admin"
lis, err := net.Listen("tcp", ":50051")
if err != nil {
log.Fatalf("failed to listen: %v", err)
}
defer lis.Close()
grpcServer := grpc.NewServer(...opts)
cleanup, err := admin.Register(grpcServer)
if err != nil {
log.Fatalf("failed to register admin services: %v", err)
}
defer cleanup()
if err := grpcServer.Serve(lis); err != nil {
log.Fatalf("failed to serve: %v", err)
}
Java
In Java, use this code to create an admin interface:
import io.grpc.services.AdminInterface;
server = ServerBuilder.forPort(50051)
.useTransportSecurity(certChainFile, privateKeyFile)
.addServices(AdminInterface.getStandardServices())
.build()
.start();
server.awaitTermination();
Python
In Python, use this code to create an admin interface:
import grpc_admin
server = grpc.server(futures.ThreadPoolExecutor())
grpc_admin.add_admin_servicers(server)
server.add_insecure_port('[::]:50051')
server.start()
server.wait_for_termination()
Use SSH to connect to a VM
The gRPC Wallet example already enables the admin interface. You can change the admin interface port by providing the following flag:
--admin-port=PORT
The default admin port is localhost:28881.
To debug your gRPC application, you can use SSH to connect to one of the VMs
that serves the wallet-service. This gives you access to the localhost.
# List the Wallet VMs $ gcloud compute instances list --filter="zone:(us-central1-a)" --filter="name~'grpcwallet-wallet-v2'" NAME ZONE MACHINE_TYPE PREEMPTIBLE INTERNAL_IP EXTERNAL_IP STATUS grpcwallet-wallet-v2-mig-us-central1-ccl1 us-central1-a n1-standard-1 10.240.0.38 35.223.42.98 RUNNING grpcwallet-wallet-v2-mig-us-central1-k623 us-central1-a n1-standard-1 10.240.0.112 35.188.133.75 RUNNING # Pick one of the Wallet VMs to debug $ gcloud compute ssh grpcwallet-wallet-v2-mig-us-central1-ccl1 --zone=us-central1-a
Install the grpcdebug tool
To access the admin interface, you need a gRPC client that can communicate with
the admin services in your gRPC application. In the following examples, you use
a tool called grpcdebug that you can download and install on the VM or Pod
where your gRPC application is running. The repository for grpcdebug is
located at
grpc-ecosystem/grpcdebug.
The minimum support Golang version is 1.12. The official Golang installation
guide is at the Golang site.
If you are following the guide to create a Linux VM for the wallet-service,
you can install Golang 1.16 by using these commands:
sudo apt update && sudo apt install -y wget wget https://golang.org/dl/go1.16.3.linux-amd64.tar.gz sudo rm -rf /usr/local/go sudo tar -C /usr/local -xzf go1.16.3.linux-amd64.tar.gz export PATH=$PATH:/usr/local/go/bin sudo ln -sf /usr/local/go/bin/go /usr/bin/go go version # go version go1.16.3 linux/amd64
You install the grpcdebug tool with the following commands:
go install -v github.com/grpc-ecosystem/grpcdebug@latest export PATH=$PATH:$(go env GOPATH)/bin
You now have access to the grpcdebug command-line interface. The help output
contains information about supported commands:
$ grpcdebug -h
grpcdebug is a gRPC service admin command-line interface
Usage:
grpcdebug <target address> [flags] <command>
Available Commands:
channelz Display gRPC states in human readable way.
health Check health status of the target service (default "").
help Help about any command
xds Fetch xDS related information.
Flags:
--credential_file string Sets the path of the credential file; used in [tls] mode
-h, --help Help for grpcdebug
--security string Defines the type of credentials to use [tls, google-default, insecure] (default "insecure")
--server_name_override string Overrides the peer server name if non empty; used in [tls] mode
-t, --timestamp Print timestamp as RFC 3339 instead of human readable strings
-v, --verbose Print verbose information for debugging
To obtain more information about a particular command, use the following:
grpcdebug <target address> [command] --help
Use the grpcdebug tool to debug your applications
You can use the grpcdebug tool to debug your applications.
The grpcdebug tool provides an ssh_config-like configuration that supports
aliasing, hostname rewriting, and connection security setting (insecure/TLS).
For more information about this advanced feature, see
grpcdebug/Connect&Security.
The following sections describe the services exposed by the admin interface and how to access them.
Use Channelz
The Channelz service provides access to runtime information about the connections at different levels in the gRPC library of your application. You can use this for live analysis of applications that might have configuration- or network-related issues. The following examples assume that you deployed the gRPC Wallet example by using the instructions in Configure advanced traffic management with proxyless gRPC services and that you provided the following flag:
--admin-port=PORT
After you send some RPCs from a test client, as shown in Verifying the configuration, use the following commands to access the Channelz data for gRPC services:
- Use SSH to connect to a VM that is running the
wallet-service. - Set up
grpcdebugto connect to the running gRPC application.
The default output of grpcdebug is in a console-friendly table
format. If you supply the --json flag, the output is encoded as JSON.
The grpcdebug channelz command is used to fetch and present debugging
information from the Channelz service. The command works for both gRPC clients
and gRPC servers.
For gRPC clients, the command grpcdebug channelz channels provides a list of
existing channels and some basic information:
$ grpcdebug localhost:28881 channelz channels Channel ID Target State Calls(Started/Succeeded/Failed) Created Time 1 xds:///account.grpcwallet.io:10080 READY 0/0/0 59 seconds ago 2 trafficdirector.googleapis.com:443 READY 2/0/0 59 seconds ago 4 xds:///stats.grpcwallet.io:10080 READY 0/0/0 59 seconds ago
If you need additional information about a particular channel, you can use
grpcdebug channelz channel [CHANNEL_ID] to inspect detailed
information for that channel. The channel identifier can be the channel ID or
the target address, if there is only one target address. A gRPC channel can
contain multiple subchannels, which is gRPC's abstraction on top of a TCP
connection.
$ grpcdebug localhost:28881 channelz channel 2
Channel ID: 2
Target: trafficdirector.googleapis.com:443
State: READY
Calls Started: 2
Calls Succeeded: 0
Calls Failed: 0
Created Time: 10 minutes ago
---
Subchannel ID Target State Calls(Started/Succeeded/Failed) CreatedTime
3 trafficdirector.googleapis.com:443 READY 2/0/0 10 minutes ago
---
Severity Time Child Ref Description
CT_INFO 10 minutes ago Channel Created
CT_INFO 10 minutes ago parsed scheme: ""
CT_INFO 10 minutes ago scheme "" not registered, fallback to default scheme
CT_INFO 10 minutes ago ccResolverWrapper: sending update to cc: {[{trafficdirector.googleapis.com:443 <nil> 0 <nil>}] <nil> <nil>}
CT_INFO 10 minutes ago Resolver state updated: {Addresses:[{Addr:trafficdirector.googleapis.com:443 ServerName: Attributes:<nil> Type:0 Metadata:<nil>}] ServiceConfig:<nil> Attributes:<nil>} (resolver returned new addresses)
CT_INFO 10 minutes ago ClientConn switching balancer to "pick_first"
CT_INFO 10 minutes ago Channel switches to new LB policy "pick_first"
CT_INFO 10 minutes ago subchannel(subchannel_id:3 ) Subchannel(id:3) created
CT_INFO 10 minutes ago Channel Connectivity change to CONNECTING
CT_INFO 10 minutes ago Channel Connectivity change to READY
You can also inspect detailed information for a subchannel:
$ grpcdebug localhost:28881 channelz subchannel 3 Subchannel ID: 3 Target: trafficdirector.googleapis.com:443 State: READY Calls Started: 2 Calls Succeeded: 0 Calls Failed: 0 Created Time: 12 minutes ago --- Socket ID Local->Remote Streams(Started/Succeeded/Failed) Messages(Sent/Received) 9 10.240.0.38:60338->142.250.125.95:443 2/0/0 214/132
You can retrieve information about TCP sockets:
$ grpcdebug localhost:28881 channelz socket 9
Socket ID: 9
Address: 10.240.0.38:60338->142.250.125.95:443
Streams Started: 2
Streams Succeeded: 0
Streams Failed: 0
Messages Sent: 226
Messages Received: 141
Keep Alives Sent: 0
Last Local Stream Created: 12 minutes ago
Last Remote Stream Created: a long while ago
Last Message Sent Created: 8 seconds ago
Last Message Received Created: 8 seconds ago
Local Flow Control Window: 65535
Remote Flow Control Window: 966515
---
Socket Options Name Value
SO_LINGER [type.googleapis.com/grpc.channelz.v1.SocketOptionLinger]:{duration:{}}
SO_RCVTIMEO [type.googleapis.com/grpc.channelz.v1.SocketOptionTimeout]:{duration:{}}
SO_SNDTIMEO [type.googleapis.com/grpc.channelz.v1.SocketOptionTimeout]:{duration:{}}
TCP_INFO [type.googleapis.com/grpc.channelz.v1.SocketOptionTcpInfo]:{tcpi_state:1 tcpi_options:7 tcpi_rto:204000 tcpi_ato:40000 tcpi_snd_mss:1408 tcpi_rcv_mss:1408 tcpi_last_data_sent:8212 tcpi_last_data_recv:8212 tcpi_last_ack_recv:8212 tcpi_pmtu:1460 tcpi_rcv_ssthresh:88288 tcpi_rtt:2400 tcpi_rttvar:3012 tcpi_snd_ssthresh:2147483647 tcpi_snd_cwnd:10 tcpi_advmss:1408 tcpi_reordering:3}
---
Security Model: TLS
Standard Name: TLS_AES_128_GCM_SHA256
On the server side, you can use Channelz to inspect your server application's
status. For example, you can get the list of servers by using the grpcdebug
channelz servers command:
$ grpcdebug localhost:28881 channelz servers Server ID Listen Addresses Calls(Started/Succeeded/Failed) Last Call Started 5 [127.0.0.1:28881] 9/8/0 now 6 [[::]:50051] 159/159/0 4 seconds ago
To obtain more information about a specific server, use the grpcdebug channelz
server command. You can inspect server sockets the same way that you
inspect client sockets.
$ grpcdebug localhost:28881 channelz server 6 Server Id: 6 Listen Addresses: [[::]:50051] Calls Started: 174 Calls Succeeded: 174 Calls Failed: 0 Last Call Started: now --- Socket ID Local->Remote Streams(Started/Succeeded/Failed) Messages(Sent/Received) 25 10.240.0.38:50051->130.211.1.39:44904 68/68/0 68/68 26 10.240.0.38:50051->130.211.0.167:32768 54/54/0 54/54 27 10.240.0.38:50051->130.211.0.22:32768 52/52/0 52/52
Use the Client Status Discovery Service
The Client Status Discovery Service (CSDS) API is part of the xDS APIs. In a gRPC application, the CSDS service provides access to the configuration (also called the xDS configuration) that it receives from Traffic Director. This lets you identify and resolve configuration-related issues in your mesh.
The following examples assume that you deployed the gRPC Wallet example by using the instructions in Configure advanced traffic management with proxyless gRPC services.
To use CSDS to examine the configuration:
- Use SSH to connect to a VM that is running the
wallet-service. Use the instructions in Use SSH to connect to a VM. - Run the
grpcdebugclient.
To get an overview of configuration status, run the following command:
grpcdebug localhost:28881 xds status
You see results similar to the following:
Name Status Version Type LastUpdated account.grpcwallet.io:10080 ACKED 1618529574783547920 type.googleapis.com/envoy.config.listener.v3.Listener 3 seconds ago stats.grpcwallet.io:10080 ACKED 1618529574783547920 type.googleapis.com/envoy.config.listener.v3.Listener 3 seconds ago URL_MAP/830293263384_grpcwallet-url-map_0_account.grpcwallet.io:10080 ACKED 1618529574783547920 type.googleapis.com/envoy.config.route.v3.RouteConfiguration 3 seconds ago URL_MAP/830293263384_grpcwallet-url-map_1_stats.grpcwallet.io:10080 ACKED 1618529574783547920 type.googleapis.com/envoy.config.route.v3.RouteConfiguration 3 seconds ago cloud-internal-istio:cloud_mp_830293263384_3566964729007423588 ACKED 1618529574783547920 type.googleapis.com/envoy.config.cluster.v3.Cluster 3 seconds ago cloud-internal-istio:cloud_mp_830293263384_7383783194368524341 ACKED 1618529574783547920 type.googleapis.com/envoy.config.cluster.v3.Cluster 3 seconds ago cloud-internal-istio:cloud_mp_830293263384_3363366193797120473 ACKED 1618529574783547920 type.googleapis.com/envoy.config.cluster.v3.Cluster 3 seconds ago cloud-internal-istio:cloud_mp_830293263384_3566964729007423588 ACKED 86 type.googleapis.com/envoy.config.endpoint.v3.ClusterLoadAssignment 2 seconds ago cloud-internal-istio:cloud_mp_830293263384_3363366193797120473 ACKED 86 type.googleapis.com/envoy.config.endpoint.v3.ClusterLoadAssignment 2 seconds ago cloud-internal-istio:cloud_mp_830293263384_7383783194368524341 ACKED 86 type.googleapis.com/envoy.config.endpoint.v3.ClusterLoadAssignment 2 seconds ago
You can find the definition of configuration status in
documentation for the Envoy proxy.
Briefly, the status of an xDS resource is one of REQUESTED, DOES_NOT_EXIST,
ACKED, or NACKED.
To obtain a raw xDS configuration dump, run the following command:
grpcdebug localhost:28881 xds config
You see a JSON list of the
PerXdsConfig
object:
{
"config": [
{
"node": {
"id": "projects/830293263384/networks/default/nodes/6e98b038-6d75-4a4c-8d35-b0c7a8c9cdde",
"cluster": "cluster",
"metadata": {
"INSTANCE_IP": "10.240.0.38",
"TRAFFICDIRECTOR_GCP_PROJECT_NUMBER": "830293263384",
"TRAFFICDIRECTOR_NETWORK_NAME": "default"
},
"locality": {
"zone": "us-central1-a"
},
"userAgentName": "gRPC Go",
"userAgentVersion": "1.37.0",
"clientFeatures": [
"envoy.lb.does_not_support_overprovisioning"
]
},
"xdsConfig": [
{
"listenerConfig": {
"versionInfo": "1618529930989701137",
"dynamicListeners": [
{
...
If the raw configuration output is too verbose, grpcdebug lets you filter
based on specific xDS types. For example:
$ grpcdebug localhost:28881 xds config --type=cds
{
"versionInfo": "1618530076226619310",
"dynamicActiveClusters": [
{
"versionInfo": "1618530076226619310",
"cluster": {
"@type": "type.googleapis.com/envoy.config.cluster.v3.Cluster",
"name": "cloud-internal-istio:cloud_mp_830293263384_7383783194368524341",
"altStatName": "/projects/830293263384/global/backendServices/grpcwallet-stats-service",
"type": "EDS",
"edsClusterConfig": {
"edsConfig": {
"ads": {},
"initialFetchTimeout": "15s",
...
You can also dump the configuration of seberal xDS types at the same time:
$ grpcdebug localhost:28881 xds config --type=lds,eds
{
"versionInfo": "1618530076226619310",
"dynamicListeners": [...]
}
{
"dynamicEndpointConfigs": [...]
}
What's next
- To find related information, see Observability with Envoy.
- To resolve configuration issues when you deploy proxyless gRPC services, see Troubleshooting deployments that use proxyless gRPC.