View app latency

Learn how to use Cloud Trace by doing the following:

  1. Deploy a sample application to a Google Kubernetes Engine (GKE) cluster.

  2. Create a trace by sending an HTTP request to the sample application.

  3. Use the Cloud Trace interface to view the latency information of the trace you created.

Before you begin

Security constraints defined by your organization might prevent you from completing the following steps. For troubleshooting information, see Develop applications in a constrained Google Cloud environment.

Create a project with billing enabled:

  1. Sign in to your Google Cloud account. If you're new to Google Cloud, create an account to evaluate how our products perform in real-world scenarios. New customers also get $300 in free credits to run, test, and deploy workloads.
  2. In the Google Cloud console, on the project selector page, select or create a Google Cloud project.

    Go to project selector

  3. Make sure that billing is enabled for your Google Cloud project.

  4. In the Google Cloud console, on the project selector page, select or create a Google Cloud project.

    Go to project selector

  5. Make sure that billing is enabled for your Google Cloud project.

Download and deploy your application

To download and deploy the application, do the following:

  1. To open the Cloud Shell, click Activate Cloud Shell in the Google Cloud console toolbar:

    Activate Cloud Shell.

    After a few moments, a Cloud Shell session opens inside the Google Cloud console.

  2. To download the source code from GitHub, run the following command:

    git clone
  3. Enable the Google Kubernetes Engine API by using the following command:

    gcloud services enable
  4. To create the GKE cluster named cloud-trace-demo in the us-central1-c zone, run the following commands in the Cloud Shell:

    gcloud container clusters create cloud-trace-demo \
       --zone $ZONE

    This command takes a few minutes to complete. After it completes successfully, your Google Cloud project contains the GKE cluster named cloud-trace-demo. You must have permission to create clusters that have external access in your Google Cloud project.

  5. Update your GKE cluster credentials by running the following command:

    gcloud container clusters get-credentials cloud-trace-demo --zone $ZONE
  6. Verify access to the cluster by running the following command:

    kubectl get nodes

    A sample output of this command is:

    NAME                                              STATUS   ROLES    AGE   VERSION
    gke-cloud-trace-demo-default-pool-063c0416-113s   Ready    <none>   78s   v1.22.12-gke.2300
    gke-cloud-trace-demo-default-pool-063c0416-1n27   Ready    <none>   79s   v1.22.12-gke.2300
    gke-cloud-trace-demo-default-pool-063c0416-frkd   Ready    <none>   78s   v1.22.12-gke.2300
  7. Deploy the sample application by running the following command:

    cd python-docs-samples/trace/cloud-trace-demo-app-opentelemetry && ./

    The script configures three services of the application using a pre-built image. The workloads are named cloud-trace-demo-a, cloud-trace-demo-b, and cloud-trace-demo-c. The setup script waits for all resources to be provisioned, so the configuration might take several minutes to complete.

    A sample output of this command is:

    deployment.apps/cloud-trace-demo-a is created
    service/cloud-trace-demo-a is created
    deployment.apps/cloud-trace-demo-b is created
    service/cloud-trace-demo-b is created
    deployment.apps/cloud-trace-demo-c is created
    service/cloud-trace-demo-c is created
    Wait for load balancer initialization complete......
    Completed. You can access the demo at

Create a trace

A trace describes the time it takes an application to complete a single operation. Each trace consists of one or more spans. A span describes how long it takes to perform a complete sub-operation. For example, a trace might describe how long it takes to process an incoming request from a user and return a response. A span might describe how long a particular RPC call requires. For more information, see Cloud Trace's Data model.

To create a trace by sending a curl request to cloud-trace-demo-a, use the following command:

curl $(kubectl get svc -o=jsonpath='{.items[?("cloud-trace-demo-a")].status.loadBalancer.ingress[0].ip}')

You can execute the curl command multiple times to generate multiple traces.

The output looks like the following:

Hello, I am service A
And I am service B
Hello, I am service C

View the trace data

In the Google Cloud console, select Trace, and then select  Overview, or click the following button:

Go to Trace Overview

Overview window

The Overview window is the default view in Trace. This window displays latency data and summary information, including an analysis report. If you created a new project, the most interesting pane of the Overview window is the pane labeled Recent traces:

Recent traces pane that displays the most recent traces and their latency.

This pane lists the most recent traces and their latency. To view the details of a trace, click its link.

Trace explorer window

In the Trace navigation pane, click Trace explorer:

Trace explorer window for the quickstart.

This window displays a graph and a table. Each dot on the graph represents a trace. Each dot also corresponds to a row in the table. In the previous screenshot, multiple traces are listed, indicating that the curl command was executed multiple times.

To view a trace in detail, select a dot in the graph or a row in the table:

Waterfall display that shows five spans.

After you select a dot, the scatter plot is refreshed and the dot you selected is highlighted with a circle drawn around the dot. The dots that represent all other traces are dimmed. The Trace details pane displays information about the trace. The first row of the table is for the trace, and there exists one row for each span in the trace. To view details about a span, click the latency bar:

Additional details about each span are shown in the details pane.

Analysis Reports window

Trace automatically creates daily reports. For this project, there isn't enough data to create a new report.

About the application

The sample application used in this quickstart is available in a GitHub repository. This repository contains information on how to use the application in environments other than the Cloud Shell. The sample application is written in Python, uses the Flask framework and OpenTelemetry packages, and executes on a Google Kubernetes Engine cluster.


The file in the GitHub repository, contains the instrumentation necessary to capture and send trace data to your Google Cloud project:

  • The application imports several OpenTelemetry packages:

    from opentelemetry import trace
    from opentelemetry.exporter.cloud_trace import CloudTraceSpanExporter
    from opentelemetry.instrumentation.flask import FlaskInstrumentor
    from opentelemetry.instrumentation.requests import RequestsInstrumentor
    from opentelemetry.propagate import set_global_textmap
    from opentelemetry.propagators.cloud_trace_propagator import CloudTraceFormatPropagator
    from opentelemetry.sdk.trace import TracerProvider
    from opentelemetry.sdk.trace.export import BatchSpanProcessor
  • The application instruments web requests with trace context and automatically traces Flask handlers and requests to other services:

    app = flask.Flask(__name__)
  • The application configures the Cloud Trace exporter as a trace provider, which propagates trace context in the Cloud Trace format:

    def configure_exporter(exporter):
        """Configures OpenTelemetry context propagation to use Cloud Trace context
            exporter: exporter instance to be configured in the OpenTelemetry tracer provider
        tracer_provider = TracerProvider()
    tracer = trace.get_tracer(__name__)
  • The following code snippet shows how to send requests in Python. OpenTelemetry implicitly propagates the trace context for you with your outgoing requests:

    if endpoint is not None and endpoint != "":
        data = {"body": keyword}
        response = requests.get(
        return keyword + "\n" + response.text
        return keyword, 200

How the application works

For clarity, in this section, cloud-trace-demo is omitted from the service names. For example, the service cloud-trace-demo-c is referenced as c.

This application creates three services named a, b, and c. Service a is configured to call service b, service b is configured to call service c. For details on the configuration of the services, see the YAML files in the GitHub repository.

When you issued a HTTP request to service a in this quickstart, you used the following curl command:

curl $(kubectl get svc -o=jsonpath='{.items[?("cloud-trace-demo-a")].status.loadBalancer.ingress[0].ip}')

The curl command works as follows:

  1. kubectl fetches the IP address of the service named cloud-trace-demo-a.
  2. The curl command then sends the HTTP request to service a.
  3. Service a receives the HTTP request and sends a request to service b.
  4. Service b receives the HTTP request and sends a request to service c.
  5. Service c receives the HTTP request from service b and returns the string Hello, I am service C to service b.
  6. Service b receives the response from service c, appends it to the string And I am service B, and returns the result to service a.
  7. Service a receives the response from service b and appends it to the string Hello, I am service A.
  8. The response from service a is printed in the Cloud Shell.

Clean up

To avoid incurring charges to your Google Cloud account for the resources used on this page, follow these steps.

  • If you created a new Google Cloud project for this quickstart, then delete the project to stop accruing charges. To delete your project, do the following:

    1. In the Google Cloud console, click Navigation Menu, and select Home.
    2. In the Project info pane, click Go to project settings.
    3. In the Settings window, click Shut down, and complete the remaining steps.
  • If you didn't create a new Google Cloud project for this quickstart, then delete the Google Kubernetes Engine cluster named cloud-trace-demo by running the following command:

    gcloud container clusters delete cloud-trace-demo --zone $ZONE

What's next