Observability

Using a service mesh gives you the ability to observe traffic to and from services, which allows for richer monitoring and debugging without code changes in the service itself. In the sidecar proxy architecture that Traffic Director uses, the proxy is the component that processes requests and provides the necessary telemetry information. Telemetry information must be collected and stored in a centralized location for further use, such as data analysis, alerting, and troubleshooting.

This guide demonstrates how to generate tracing and logging for the Envoy proxy. This guide also shows you how to export the information to Cloud Trace and Cloud Logging.

Demonstration setup

This guide uses the following configuration to demonstrate tracing and logging:

  • A single application that listens on the HTTP port and returns the hostname of the VM that served the request. In the diagram, this application is in the upper-right-hand corner, labeled HTTP service(s) (10.10.10.10:80). One or more VMs can provide this service.
  • A single Compute Engine VM running a consumer of this service. In the diagram, this is labeled Demo Compute Engine VM.
  • An Envoy sidecar proxy installed and configured by Traffic Director. In the diagram, this is labeled Envoy.
  • A service consumer application, shown in the gray box, is the consumer of the HTTP service running on 10.10.10.10:80.
Demonstration application for Stackdriver logging and monitoring for Envoy (click to enlarge)
Demonstration application for Stackdriver logging and monitoring for Envoy (click to enlarge)
  1. Traffic Director configures the Envoy proxy to load balance traffic for the 10.10.10.10:80 service, to store access log information for each request issued for this service, and to generate tracing information for the service.
  2. After the consumer sends a request to 10.10.10.10, the sidecar proxy routes the request to the correct destination.
  3. The Sidecar proxy also generates the necessary telemetry information:
    1. Adds an entry to the access log on the local disk with additional information about the request
    2. Generates a trace entry and sends it to Trace using OpenCensus Envoy tracing.
  4. The logging agent exports this data to the Cloud Logging API, so that the data becomes available in the Cloud Logging interface.

Prerequisites

Ensure that:

  1. The Traffic Director API is enabled and other prerequisites are met, as described in Preparing for Traffic Director setup.
  2. The service account that the Compute Engine VM uses has the Cloud Trace Agent role configured. See the Trace Access Control page for more information.
  3. The service account that the Compute Engine VM uses has the Logs Writer role configured. See the Logging Access Control page for more information.

Setting up the demonstration service and Traffic Director

This guide uses several shell scripts to perform the steps required to configure the demonstration service. Review the scripts to understand the specific steps they perform.

  1. Start a Compute Engine VM and configure the HTTP service on the VM.

    curl -sSO https://storage.googleapis.com/traffic-director/demo/observability/setup_demo_service.sh
    chmod 755 setup_demo_service.sh && ./setup_demo_service.sh
    

    The setup_demo_service.sh script creates a VM template that launches apache2 when a VM starts and a managed instance group that uses this template. The script launches a single instance without autoscaling enabled.

  2. Configure routing for the 10.10.10.10 service using Traffic Director

    curl -sSO https://storage.googleapis.com/traffic-director/demo/observability/setup_demo_trafficdirector.sh
    chmod 755 setup_demo_trafficdirector.sh && ./setup_demo_trafficdirector.sh
    

    The setup_demo_trafficdirector.sh script configures the necessary parameters for the Traffic Director managed service, similar to the configuration described in the Setting up Traffic Director for Compute Engine with VMs.

  3. Start a Compute Engine VM that runs a consumer of the HTTP service, with the sidecar proxy installed and configured on the VM. In the command below, replace the variable gcp_project_id with the project ID to which Trace information should be sent. This is typically the same project to which your VM belongs.

    curl -sSO https://storage.googleapis.com/traffic-director/demo/observability/setup_demo_client.sh
    chmod 755 setup_demo_client.sh && ./setup_demo_client.sh my_gcp_project_id
    

    The setup_demo_client.sh script creates a Compute Engine VM that has an Envoy proxy preconfigured to use Traffic Director. This is similar to the configuration described in Setting up Traffic Director for Compute Engine with VMs.

The following additional configuration settings enable tracing and logging:

  • The TRAFFICDIRECTOR_ACCESS_LOG_PATH and TRAFFICDIRECTOR_ENABLE_TRACING bootstrap node metadata variables enable logging and tracing, as described in the Configuring additional attributes for sidecar proxies.
  • Static bootstrap configuration, enabling export of trace information to Trace using OpenCensus.

NOTE: You can add and modify additional tracing parameters through the Envoy runtime configuration.

After running these scripts, you can log in to the td-observability-demo-client VM and access the HTTP service available at 10.10.10.10.

curl http://10.10.10.10

At this point, Envoy generates access log and tracing information. The following section describes how to export logs and tracing information.

Setting up tracer export to Cloud Trace

The Envoy bootstrap configuration that you created when you ran the setup-demo-client.sh script is sufficient to generate tracing information. All other configuration is optional. If you want to configure additional parameters, see the OpenCensus Envoy configuration page and modify the tracing options in the Envoy bootstrap configuration.

After you issue a sample request to the demonstration server (curl 10.10.10.10), go to the Trace interface (Trace > Trace list) in the Google Cloud Console. You see a trace record corresponding to the request that you issued.

For more information on how to use Trace, see its documentation.

Setting up access log export to Logging

At this stage, Envoy should be recording access log information to the local disk of the VM where it is running. To export these records to Logging, you must install the Logging agent locally. This requires installing and configuring the Logging agent.

Installing the Logging agent

Install the Logging agent on the VM from which logging information is exported. For this example configuration, the VM is td-observability-demo-vm.

curl -sSO https://dl.google.com/cloudagents/install-logging-agent.sh
sudo bash install-logging-agent.sh

For more information on Logging agent installation, see the guide Installing the agent.

Configuring the Logging agent

You can export the Envoy logs as either unstructured or structured text.

Exporting the Envoy logs as unstructured text

This option exports log records from the access log to Cloud Logging as raw text. Each entry in the access log is exported as a single entry to Logging. This configuration is easier to install, because it relies on a parser that is distributed with the current version of the Logging agent, but it is more difficult to filter and process raw text log entries using this option.

  1. Download and install the Envoy access log unstructured export config file.

    curl -sSO https://storage.googleapis.com/traffic-director/demo/observability/envoy_access_fluentd_unstructured.conf
    sudo cp envoy_access_fluentd_unstructured.conf /etc/google-fluentd/config.d/envoy_access.conf
    
  2. Restart the agent. The changes take effect when the agent starts up.

    sudo service google-fluentd restart
    

Exporting the Envoy logs in a structured format

  1. Install the Envoy access log parser from GitHub.

    sudo /opt/google-fluentd/embedded/bin/gem install fluent-plugin-envoy-parser
    
  2. Download and install the configuration file for exporting Envoy access logs in a structured format:

    curl -sSO https://storage.googleapis.com/traffic-director/demo/observability/envoy_access_fluentd_structured.conf
    sudo cp envoy_access_fluentd_structured.conf /etc/google-fluentd/config.d/envoy_access.conf
    
  3. Restart the agent. The changes take effect when the agent starts up.

    sudo service google-fluentd restart
    

For more information on Logging agent configuration, see Configuring the agent.

Verification

  1. From the sidecar proxy VM, generate a request to the demonstration service. This creates a new local log record. For example, you can run curl 10.10.10.10.
  2. Go to Logging > Logs Viewer in the Cloud Console. In the drop-down menu, select the envoy-access log type. You see a log entry for the most recent request in the unstructured or structured format, depending on the configuration type you chose earlier.

Troubleshooting

If configuration is complete, but you do not see trace or logging entries available, verify the following:

  1. The service accounts for the Compute Engine VM have the necessary Trace and Logging IAM permissions, as specified in the prerequisites. For information on Trace IAM permissions, see Access control. For information on Logging permissions, see Access control.
  2. For logging: Ensure that there are no errors in /var/log/google-fluentd/google-fluentd.log.
  3. For logging: Ensure that new entries appear in the local access log file when requests are issued.