Tracing your API

After you deploy the Extensible Service Proxy (ESP) and your API's backend code, ESP intercepts all requests and performs any necessary checks before forwarding the request to the API backend. When the backend responds, ESP gathers and reports telemetry. One piece of telemetry data that ESP captures is tracing, by using Stackdriver Trace.

This page explains how to:

  • View traces in the Google Cloud Platform Console.
  • Estimate your cost for Trace.
  • Configure ESP to disable trace sampling.

Viewing traces for your API

To view traces for your API:

  1. Go to the Endpoints > Services page for your project:

    Go to the Endpoints Services page

  2. If you have more than one API, click the API that you want to see traces for.

  3. Ensure you are on the Overview tab.

  4. Scroll down and click View traces. The Stackdriver Trace List page is displayed.

A trace tracks an incoming request to your API and the various events, such as RPC calls or instrumented sections of code that took place, along with precise timings. These events are represented as spans in the trace.

On the Trace List page, you can drill down on an individual trace and see the spans that ESP creates in a trace.

Example trace with spans

For more information on the Trace List page, see Viewing traces in the GCP Console.

Spans created by ESP

At a minimum, ESP creates 4 spans per trace:

  • A span for the entire request and response.
  • A CheckServiceControl span for the call to the Service Control's services.check method to get the configuration for your API.
  • A QuotaControl span to check if there is a quota configured on your API.
  • A Backend span that tracks the time spent in your API's backend code.

Depending on the configuration for your API, ESP creates additional spans:

  • If your API requires authentication, ESP creates a CheckAuth span in every trace. To authenticate a request, ESP caches the public key it needs to authenticate for 5 minutes. If the public key isn't in the cache, ESP retrieves and caches the public key and creates a HttpFetch span.
  • If your API requires an API key, ESP creates a CheckServiceControlCache span in every trace. ESP caches the information that it needs to validate the API key. If the information isn't in the cache, ESP calls Service Control and creates a Call ServiceControl server span.
  • If you have a quota set for your API, ESP creates a QuotaServiceControlCache span in every trace. ESP caches the information that it needs to check the quota. If the information isn't in the cache, ESP calls Service Control and creates a Call ServiceControl server span.

Trace sampling rate

ESP samples a small number of requests to your API to get trace data. To control the sampling rate, ESP maintains a request counter and a timer. The number of requests per second to your API determines the sampling rate. If there are no requests within a second, ESP doesn't send a trace.

If the number of requests in a second is:

  • Less than or equal to 999, ESP sends 1 trace.
  • Between 1000 and 1999, ESP sends 2 traces.
  • Between 2000 and 2999, ESP sends 3 traces.
  • And so on.

In summary, you can estimate the sampling rate with the ceiling function: ceiling(requests per second/1000)

Estimating the cost of Trace

To estimate the cost of Trace, you need to estimate the number of spans that ESP sends to Trace in a month.

To estimate the number of spans per month:

  1. Estimate the number of requests per second to your API. To get this estimate, you can use the Requests graph on the Endpoints > Services page or Stackdriver Logging. See Monitoring your API for more information.
  2. Calculate the number of traces that ESP sends to Trace per second: ceiling(requests per second/1000)
  3. Estimate the number of spans in a trace. To get this estimate, you can use the information in Spans created by ESP or view the Trace List page to see traces for your API.
  4. Estimate the number of seconds in a month that your API gets traffic. For example, some APIs get requests only during certain times of the day, and other APIs get requests sporadically.
  5. Multiply the number of seconds in the month by the number of spans.

For example:

  1. Assume that the maximum number of requests per second for an API is 5.
  2. The trace sampling rate is ceiling (5/1000) = 1
  3. The API doesn't have a quota configured, doesn't require an API key, and doesn't require authentication. Therefore, the number of spans that ESP creates per trace is 4.
  4. This API gets requests only during business hours, Monday through Friday. The number of seconds in a month that the API gets traffic is approximately: 3600 X 8 X 20 = 576,000
  5. The number of spans per month is approximately 576,000 x 4 = 2,304,000

After you know the approximate number of spans in a month, refer to the Stackdriver pricing page for detailed pricing information.

Disabling trace sampling

If you want to stop ESP from sampling requests and sending traces, you can set an ESP startup option and restart ESP. The traces ESP sends to Stackdriver Trace are independent of the graphs displayed on the Endpoints > Services page. The graphs continue to be available if you disable trace sampling.

The following section assumes you have already deployed your API and ESP, or you are familiar with the deployment process. For more information, see Deploying the API backend.

App Engine

To disable ESP trace sampling in the App Engine flexible environment:

  1. Edit the app.yaml file. In the endpoints_api_service section, add the trace_sampling option and set its value to false. For example:
    endpoints_api_service:
      name: example-project-12345.appspot.com
      rollout_strategy: managed
      trace_sampling: false
    

    If your application is based on microservices, you must include trace_sampling: false in every app.yaml file.

  2. If you haven't updated the Cloud SDK recently, run the following command:
    gcloud components update
    
  3. Save the app.yaml file (or files).
  4. Deploy your backend code and ESP to App Engine:
    gcloud app deploy
    

To re-enable trace sampling:

  1. Remove the trace_sampling option from app.yaml.
  2. Deploy your backend code and ESP to App Engine:
    gcloud app deploy
    

Compute Engine

To disable ESP trace sampling in Compute Engine with Docker:

  1. Connect to your VM instance:
    gcloud compute ssh [INSTANCE_NAME]
  2. In the ESP flags for the docker run command, add the option --disable_trace_sampling=true:
    sudo docker run \
        --name=esp \
        --detach \
        --publish=80:8080 \
        --net=esp_net \
        gcr.io/endpoints-release/endpoints-runtime:1 \
        --service=[SERVICE_NAME] \
        --rollout_strategy=managed \
        --backend=[YOUR_API_CONTAINER_NAME]:8080 \
        --disable_trace_sampling=true
  3. Issue the docker run command to restart ESP.

To re-enable trace sampling:

  1. Either remove the --disable_trace_sampling option or set it to false.
  2. Issue the docker run command to restart ESP.

GKE

To disable ESP trace sampling in GKE:

  1. Open your deployment manifest file, referred to as deployment.yaml, and add the following to the containers section:
    containers:
    - name: esp
      image: gcr.io/endpoints-release/endpoints-runtime:1
      args: [
        "--http_port=8081",
        "--backend=127.0.0.1:8080",
        "--service=[SERVICE_NAME]",
        "--rollout_strategy=managed",
        "--disable_trace_sampling=true"
      ]
  2. Start the Kubernetes service by using the kubectl create command:
    kubectl create -f deployment.yaml

To re-enable trace sampling:

  1. Either remove the --disable_trace_sampling option or set it to false.
  2. Start the Kubernetes service:
    kubectl create -f deployment.yaml

If you are running ESP on a Compute Engine VM instance without a Docker container, there is no equivalent VM instance metadata key-value pair for the --disable_trace_sampling option. If you want to disable trace sampling, you must run ESP in a container.

A client can force a request to be traced by adding the X-Cloud-Trace-Context header to the request, as described in Trace troubleshooting. If a request contains the X-Cloud-Trace-Context header, ESP sends the trace data to Trace even if you have disabled trace sampling.

Was this page helpful? Let us know how we did:

Send feedback about...

Cloud Endpoints with OpenAPI
Need help? Visit our support page.