Trace app latency by using Cloud Trace

Learn how to use Cloud Trace by doing the following:

  1. Deploy a sample application to a Google Kubernetes Engine (GKE) cluster.

  2. Create a trace by sending an HTTP request to the sample application.

  3. Use the Cloud Trace interface to view the latency information of the trace you created.

For step-by-step guidance on this task directly in console, click Guide me:

Guide me

The following sections take you through the same steps as clicking Guide me.

Before you begin

Some of the steps in this document might not work correctly if your organization applies constraints to your Google Cloud environment. In that case, you might not be able to complete tasks like creating public IP addresses or service account keys. If you make a request that returns an error about constraints, see how to Develop applications in a constrained Google Cloud environment.

Create a project with billing enabled:

  1. Sign in to your Google Cloud account. If you're new to Google Cloud, create an account to evaluate how our products perform in real-world scenarios. New customers also get $300 in free credits to run, test, and deploy workloads.

  2. In the Google Cloud console, on the project selector page, select or create a Google Cloud project.

    Go to project selector

  3. Make sure that billing is enabled for your Cloud project. Learn how to check if billing is enabled on a project.

  4. In the Google Cloud console, on the project selector page, select or create a Google Cloud project.

    Go to project selector

  5. Make sure that billing is enabled for your Cloud project. Learn how to check if billing is enabled on a project.

Download and deploy your application

To download and deploy the application, do the following:

  1. To open the Cloud Shell, click Activate Cloud Shell in the Google Cloud console toolbar:

    Activate Cloud Shell.

    After a few moments, a Cloud Shell session opens inside the Google Cloud console:

    Cloud Shell session.

  2. To download the source code from GitHub, run the following command:

    git clone https://github.com/GoogleCloudPlatform/python-docs-samples.git

  3. To enable the Google Kubernetes Engine API, issue the following command in the Cloud Shell:

    gcloud services enable container.googleapis.com

  4. To create the GKE cluster named demo, run the following command in the Cloud Shell:

    gcloud container clusters create demo --zone us-west1-b

    This command requires 3 to 5 minutes to complete. After it completes successfully, your Google Cloud project contains the GKE cluster named demo. You must have permission to create clusters that have external access in your Google Cloud project. If your Google Cloud is in an organization, or in a folder, you might not have these permissions even if you are the project owner. If this command fails, contact your system administrator.

  5. To verify successful creation, execute the following kubectl command:

    kubectl get nodes

    A sample output of this command is:

    NAME                                  STATUS   ROLES    AGE   VERSION
gke-demo-default-pool-24680f3d-bzvg   Ready    <none>   26m   v1.14.10-gke.27
gke-demo-default-pool-24680f3d-hfnq   Ready    <none>   26m   v1.14.10-gke.27
gke-demo-default-pool-24680f3d-qnh6   Ready    <none>   26m   v1.14.10-gke.27

  6. To set the cluster context, run the following command:

    gcloud container clusters get-credentials demo --zone us-west1-b

  7. Deploy the sample application by running the following command:

    cd python-docs-samples/trace/cloud-trace-demo-app && ./setup.sh

    The script setup.sh, which is part of this project, configures three different services, labeled cloud-trace-demo-a, cloud-trace-demo-b, and cloud-trace-demo-c. Because the set-up script configures one service at a time, the configuration requires several minutes to complete. The following is an example of what the script prints to the Cloud Shell during its execution:

     Creating service a
 deployment.apps/cloud-trace-demo-a unchanged
 service/cloud-trace-demo-a unchanged
 Fetching the external IP of service a
 Passing external IP for the first service 34.82.132.95 to the second service template
 deployment.apps/cloud-trace-demo-b created
 service/cloud-trace-demo-b created
 Fetching the external IP of service b
 Passing external IP for the service b 35.230.0.131 to the service c
 deployment.apps/cloud-trace-demo-c created
 service/cloud-trace-demo-c created

Create a trace

A trace describes the time it takes an application to complete a single operation. Each trace consists of one of more spans. A span describes how long it takes to perform a complete sub-operation. For example, a trace might describe how long it takes to process an incoming request from a user and return a response. A span might describe how long a particular RPC call requires. For more information, see Cloud Trace's Data model.

To create a trace, execute the following command:

curl $(kubectl get svc cloud-trace-demo-c -ojsonpath='{.status.loadBalancer.ingress[0].ip}')

The curl command generates a HTTP GET request and issues the request to the service named cloud-trace-demo-c. When this request completes, Helloworld! is printed to the shell.

You can execute the curl command multiple times to generate multiple traces.

View the trace data

To open the Cloud Trace interface, in the Google Cloud console, click Navigation Menu and select Trace, or click the following button:

Go to Trace

Overview window

The Overview window is the default view in Trace. This window displays latency data and summary information, including an analysis report. If you created a new project, the most interesting pane of the Overview window is the pane labeled Recent traces:

Recent traces pane that displays the most recent traces and their latency.

This pane lists the most recent traces and their latency. To view the details of a trace, click its link.

Trace list window

In the Trace navigation pane, click Trace list:

Trace list window for the quickstart.

This window displays a graph and a table. Each dot on the graph represents a trace. Each dot also corresponds to a row in the table. In the previous screenshot, multiple traces are listed, indicating that the curl command was executed multiple times.

To view a trace in detail, select a dot in the graph or a row in the table. This action results in two panes opening that display the details of the selected trace:

Waterfall display for the quickstart.

One pane displays the trace in a waterfall graph and the pane shows its details. Each row in the waterfall graph corresponds to a span. The details of the span, such as its trace labels, method, and summary information about the command latency are displayed in the details table. To view details about a span, click the corresponding row in the waterfall graph. Notice that the spans are for three different IP addresses; these correspond to the IP addresses for the three services. From this trace, you can see that a HTTP request received by the service name cloud-trace-demo-c is passed on to the service cloud-trace-demo-band then to cloud-trace-demo-a.

The table of details might include a link to data that was sent to Cloud Logging. To view the information in the Cloud Logging interface, click View. The following screenshot illustrates an example of a log that was generated by trace data:

LogEntry for a trace from the quickstart.

Note that the IP addresses displayed in the waterfall view correspond to the IP addresses of the services named cloud-trace-demo-a and cloud-trace-demo-b.

Analysis Reports window

To view or create a report, in the Trace navigation pane, click Analysis Reports. Trace automatically creates daily reports. For this project, there isn't enough data to create a new report.

About the application

The sample application used in this quickstart is available in a GitHub repository. This repository contains information on how to use the application in environments other than the Cloud Shell. The sample application is written in Python, uses the Flask framework and OpenCensus packages, and executes on a Google Kubernetes Engine cluster.

The application authors chose to use the Flask framework because its usage simplifies application development and because they wanted to use the OpenCensus Flask middleware. You don't have to use the Flask framework if you are using Python.

Instrumentation

The file app.py in the GitHub repository, contains the instrumentation necessary to capture and send trace data to your Google Cloud project:

  • The application import statements include a statement for Flask and for several OpenCensus packages. The StackdriverExporter is the object that sends trace data to your Google Cloud project:

    from flask import Flask
from opencensus.ext.flask.flask_middleware import FlaskMiddleware
from opencensus.ext.stackdriver.trace_exporter import StackdriverExporter
from opencensus.trace import execution_context
from opencensus.trace.propagation import google_cloud_format
from opencensus.trace.samplers import AlwaysOnSampler

  • The application creates a middleware component that uses Flask as the HTTP framework:

    propagator = google_cloud_format.GoogleCloudFormatPropagator()


def createMiddleWare(exporter):
    # Configure a flask middleware that listens for each request and applies automatic tracing.
    # This needs to be set up before the application starts.
    middleware = FlaskMiddleware(
        app,
        exporter=exporter,
        propagator=propagator,
        sampler=AlwaysOnSampler())
    return middleware

    In this application, the field sampler is set by the OpenCensus method AlwaysOnSampler(). This method returns True for every sampling decision and ensures 100% of requests are traced. Sampling all requests isn't recommended in a production environment. For more information, see Sampling rate.

  • In the application's main function, construct the Flask middleware that uses the StackdriverExporter():

    createMiddleWare(StackdriverExporter())

  • The application contains one more modification that isn't required. Instead, this modification is illustrative. In the application, the response to the route / includes the X-Cloud-Trace-Context header:

    trace_context_header = propagator.to_header(execution_context.get_opencensus_tracer().span_context)
response = requests.get(
    url,
    params=data,
    headers={
      'X-Cloud-Trace-Context' : trace_context_header}
)

    The X-Cloud-Trace-Context header is an HTTP header that contains information about the current trace, including the trace identifier. For different services to be able to add span information to the same trace, these services must be able to identify the trace identifier. By default, the OpenCensus package automatically includes this context in the response headers.

How the application works

For clarity, in this section, cloud-trace-demo is omitted from the service names. For example, the service cloud-trace-demo-c is referenced as c.

This application creates three services named a, b, and c. It configures service c to call service b, and for service b to call service a. For details on the configuration of the services, see the YAML files in the GitHub repository.

When you issued a HTTP request to service c in this quickstart, you used the following curl command:

curl $(kubectl get svc cloud-trace-demo-c -ojsonpath='{.status.loadBalancer.ingress[0].ip}')

The curl command works as follows:

  1. kubectl fetches the IP address of the service named cloud-trace-demo-c.
  2. The curl command then sends the HTTP request to service c.
  3. Service c receives the HTTP request and sends a request to service b.
  4. Service b receives the HTTP request and sends a request to service a.
  5. Service a receives the request and returns the string Hello. The string Hello is a keyword passed as a default argument to this service.
  6. Service b receives the response from service a, appends the string world, and then returns Helloworld. The string world is a keyword passed as a default argument to this service.
  7. Service c receives the response from service b, appends !, and then returns Helloworld!.
  8. The response from service c is printed in the Cloud Shell.

Clean up

To avoid incurring charges to your Google Cloud account for the resources used on this page, follow these steps.

  • If you created a new Google Cloud project for this quickstart, then delete the project to stop accruing charges. To delete your project, do the following:

    1. In the Google Cloud console, click Navigation Menu, and select Home.
    2. In the Project info pane, click Go to project settings.
    3. In the Settings window, click Shut down, and complete the remaining steps.

  • If you didn't create a new Google Cloud project for this quickstart, then delete the Google Kubernetes Engine cluster named demo by running the following command:

    gcloud container clusters delete demo --zone us-west1-b

What's next