Understand Traffic Director client status

When you use Traffic Director to handle your application networking, consider the following two primary components:

The infrastructure layer. The infrastructure layer, such as Envoy sidecar proxies or proxyless gRPC libraries, is configured to handle networking on behalf of your applications.
The control plane, Traffic Director. The control plane generates configuration for, and distributes configuration to, the infrastructure layer.

When an Envoy proxy or the gRPC library is initialized, it uses xDS APIs to connect to Traffic Director. The proxy or the library acts as a client to Traffic Director. After a connection is established between the client and Traffic Director, Traffic Director sends configuration information back to the client and updates the configuration as needed.

It is sometimes helpful to understand which clients are connected to Traffic Director or to inspect the configuration that Traffic Director generates for its clients. For example, you might want to debug an issue, or you might want to understand how the actions that you took when configuring Traffic Director affect the configuration that your clients see.

Traffic Director supports the Client Status Discovery Service (CSDS) API. You use a CSDS client to query this API. This enables you to see which clients are connected to Traffic Director and to inspect the configuration that Traffic Director generates for its clients.

The CSDS client is an open source tool that you can obtain from the Envoy repository. The following diagram illustrates how the CSDS client queries Traffic Director for information about Traffic Director's CSDS API.

Using the CSDS API to obtain configuration information about Traffic Director clients. — Using the CSDS API to obtain configuration information about Traffic Director clients (click to enlarge)

The CSDS client connects to Traffic Director and presents a project number and network name, along with a set of credentials. Traffic Director can then respond with information about the various Traffic Director clients to which it is connected.

For more information about the CSDS client, see the README file.

Prerequisites

To connect to the CSDS API, you need a CSDS client. You can obtain the client in one of two ways:

You can build the client by using Cloud Shell.
You can build the client on a local development machine.

Build the CSDS client by using Cloud Shell

To use Cloud Shell to build the CSDS client, do the following:

Reset Cloud Shell by using the instructions in Disabling or resetting Cloud Shell. This ensures that existing configurations do not interfere with your build.
In the Google Cloud console, open a new Cloud Shell session.
Run the following command to obtain the source code that you use to build the CSDS client:
```
git clone https://github.com/envoyproxy/envoy-tools.git
```
Navigate to the CSDS client directory and run the following commands:
```
cd envoy-tools/csds-client/
make init
make build
```
After the build is complete, test it with the following command:
```
csds-client -help
```

If the build is successful, you see the help text for the client.

Build the CSDS client on a local development machine

You can download and build a CSDS client on a local machine by following the instructions in the README file in the open source repository. To do this, you must also have Go and the make tool set up in your environment. If you prefer not to do this, use the preceding instructions for Cloud Shell, in which Go and the make tool are provided for you.

Additional prerequisites

Make sure that the node ID of each client is unique within the service mesh. If multiple clients share the same node ID, only one configuration is returned—the configuration for the client that most recently connected to Traffic Director.

If you use Google's reference packages, you do not need to manually set the node ID in your bootstrap files; a node ID is generated for you. If you do not use the reference packages, you must manually set the node ID in each of your bootstrap files.
Ensure that you have access to a user account that has the Identity and Access Management (IAM) permissions required to configure Traffic Director. The following instructions use the Google Cloud CLI to generate and automatically supply the credentials needed by the CSDS client. Alternatively, you can use the CSDS client and supply the credentials directly.

Determine which clients are currently connected to Traffic Director

Before you complete these steps, you need the following information:

The project number of the project from which the credentials were generated.
The network name that you specified when you configured Traffic Director. This is the network name from the forwarding rule of the routing rule map.

To determine which clients are connected to Traffic Director, follow these steps:

From an account that has the correct permissions, run this command:

gcloud auth application-default login \
   --billing-project=BILLING_PROJECT_ID

Create a new file in the YAML format with the following content. Replace PROJECT_NUMBER and NETWORK_NAME with the correct values for your deployment:

  node_matchers:
    - node_metadatas:
        - path:
            - key: TRAFFICDIRECTOR_GCP_PROJECT_NUMBER
          value:
            string_match:
              exact: "PROJECT_NUMBER" # for example: "314053285323"
        - path:
            - key: TRAFFICDIRECTOR_NETWORK_NAME
          value:
            string_match:
              exact: "NETWORK_NAME" # for example: "default"

Run the CSDS client, which uses the credentials generated by the gcloud CLI. Replace PATH_TO_CSDS_REQUEST_YAML_FILE with the path to the YAML file that you created in the previous step.

csds-client \
    -service_uri trafficdirector.googleapis.com:443 \
    -platform gcp \
    -authn_mode auto \
    -api_version v2 \
    -request_file PATH_TO_CSDS_REQUEST_YAML_FILE

You should see output similar to the following:

Client ID                                          xDS stream type    Config status
603e3524-d1d6-4a9e-9b26-39bcd633a7cb~10.128.0.5    ADS                N/A
603e3524-d1d6-4a9e-9b26-39bcd633a7cb~10.128.0.5    LRS                N/A
8576d4bf-8f10-40b2-920b-bb6a7cf9f34a~10.168.0.3    ADS                N/A
8576d4bf-8f10-40b2-920b-bb6a7cf9f34a~10.168.0.3    LRS                N/A
d9577b61-fa3a-41d6-90bd-11c4fdd2f8c0~10.128.0.4    ADS                N/A
d9577b61-fa3a-41d6-90bd-11c4fdd2f8c0~10.128.0.4    LRS                N/A
f38a59c1-4428-42f1-be81-e02eb994f9dd~10.128.0.6    ADS                N/A
f38a59c1-4428-42f1-be81-e02eb994f9dd~10.128.0.6    LRS                N/A

The Client ID column displays the client IDs of the clients that are connected to Traffic Director. These client IDs are provided by using the node_id field in the bootstrap file used by Envoy or proxyless gRPC when they connect to Traffic Director.

Inspect the configuration for a specific Traffic Director client

You can inspect the configuration that Traffic Director sends to a particular client by using the client's ID, which you obtained in the previous section.

To inspect the configuration, follow these steps:

Update the YAML file that you previously created and set the value of node_id to the client ID:

  node_matchers:
    - node_id:
        exact: "CLIENT_ID" # for example: "f38a59c1-4428-42f1-be81-e02eb994f9dd~10.128.0.6"
      node_metadatas:
        - path:
            - key: TRAFFICDIRECTOR_GCP_PROJECT_NUMBER
          value:
            string_match:
              exact: "PROJECT_NUMBER" # for example: "314053285323"
        - path:
            - key: TRAFFICDIRECTOR_NETWORK_NAME
          value:
            string_match:
              exact: "NETWORK_NAME" # for example: "default"

Run the CSDS client. This generates a JSON file that contains the configuration that is sent to the Traffic Director client.

csds-client \
   -service_uri trafficdirector.googleapis.com:443 \
   -platform gcp \
   -authn_mode auto \
   -api_version v2 \
   -request_file PATH_TO_CSDS_REQUEST_YAML_FILE \
   -output_file FILENAME.JSON

You should see output similar to the following:

Client ID                                          xDS stream type    Config status
8576d4bf-8f10-40b2-920b-bb6a7cf9f34a~10.168.0.3    ADS                LDS  SYNCED
                                                                      RDS  SYNCED
                                                                      CDS  STALE
Config has been saved to FILENAME.JSON

You can inspect a detailed xDS configuration by looking at the JSON file. This output contains the status of individual xDS configurations sent by Traffic Director to the client by using an aggregated gRPC stream (ADS).

Status values

The following table lists the xDS configuration status values that you might see.

Value	Description
`UNKNOWN`	(Default) ⁣Status information is not available or is unknown.
`SYNCED`	Traffic Director sent the configuration to the client and received an `ACK` from the client.
`ERROR`	⁣Traffic Director sent the configuration to the client and received a `NACK` from the client.
`STALE`	Traffic Director sent the configuration to the client but did not receive an `ACK` or a `NACK` from the client.
`NOT_SENT`	The configuration was not sent.
`N/A`	The CSDS client did not include the node ID. All of the connected streams are returned, but the configuration status is not available.

Visualization and monitoring

The CSDS client open source tool has additional features that you might want to use, such as visualization and continued monitoring. For more information about these features, see the README file in the open source repository.

Error message

You might see the following error message from the CSDS client when you enable the Traffic Director API in only your project:

rpc error: code = NotFound desc = Requested entity was not found.

This is expected behavior. Traffic Director configuration is scoped per network. If you haven't created a network yet and you run the CSDS client, you see this error message.

Limitations

Traffic Director only supports the CSDS v2 API.
Endpoint information is not included in the CSDS response because this information is not available in the CSDS v2 API.
The node ID of each client must be unique within the service mesh. If multiple clients share the same node ID, only one configuration is returned—the configuration for the client that most recently connected to Traffic Director.
You might sometimes see a backslash (\) in the node ID field in the YAML file. If this happens, escape the backslash by using an additional backslash when you query the CSDS API for configuration information. This is a known issue.

What's next

To find general Traffic Director troubleshooting information, see Troubleshooting deployments that use Envoy.
To resolve configuration issues when you deploy proxyless gRPC services, see Troubleshooting deployments that use proxyless gRPC.