This document provides a brief overview on how to instrument your application for Cloud Trace. For detailed instructions on setting up Cloud Trace, see the language-specific setup pages.
Cloud Trace provides distributed tracing data for your applications. After instrumenting your application, you can inspect latency data for a single request and view the aggregate latency for an entire application in the Cloud Trace console.
Cloud Trace recommends using OpenTelemetry. OpenTelemetry is an open-source product from the merger between OpenCensus and OpenTracing.
When to instrument your application
For your application to submit traces to Cloud Trace, it must be instrumented. You can instrument your code by using the Google client libraries or by using OpenTelemetry, which is an open source tracing package. We recommend that you use OpenTelemetry.
How to instrument applications
There are multiple ways to implement tracing for your applications:
Use OpenTelemetry and the associated Cloud Trace exporter. This is the recommended way to instrument your applications.
If you are writing applications that run on Compute Engine, then you can also use the Ops Agent and the OpenTelemetry Protocol (OTLP) receiver to collect both traces and metrics from your application. For more information, see Use the Ops Agent and OTLP.
Use the Cloud Trace API and write custom methods to send tracing data to Cloud Trace.
Configure a Zipkin server to receive traces from Zipkin clients and then forward those traces to Cloud Trace for analysis. For information about this approach, see Using Cloud Trace with Zipkin.
Configure Spring Boot applications to forward the trace data it collects to Cloud Trace. For information about this procedure, see Spring Cloud for Google Cloud: Cloud Trace.
Recommended client libraries
The following table lists the recommended instrumentation for each programming language:
Language | Recommended instrumentation |
---|---|
C# .NET | Cloud Trace client library |
Go | OpenTelemetry Go |
Java | OpenTelemetry Java |
Node.js | OpenTelemetry JavaScript |
PHP | Cloud Trace client library |
Python | OpenTelemetry Python |
Ruby | OpenTelemetry Ruby |
When to create spans
The Cloud Trace client libraries typically maintain a global trace context that holds information about the current span, including its trace ID and whether the trace is sampled. These libraries usually create spans on RPC boundaries. However, you might need to create spans if the default creation algorithm isn't sufficient for your needs.
The current active span can be accessed by the global trace context, which is sometimes wrapped in a Tracer object. You can add information relevant to your application by using custom annotations and tags to existing spans, or you can create new child spans with their own annotations and tags to trace the application's behavior with finer granularity. Because the context is global, multi-threaded applications that update the context must use appropriate isolation.
When to provide authentication credentials
You generally don't need to provide authentication credentials to your application or specify your Google Cloud project ID in your application when you are running on Google Cloud. For some languages, you do need to specify your Google Cloud project ID even if you are running on Google Cloud. Also, if you use Autopilot mode for Google Kubernetes Engine, or if you enable Workload Identity, then you must configure your application to use Workload Identity.
If you are running outside of Google Cloud, you need to provide authentication credentials to your application. You also need to specify your Google Cloud project ID in your application.
For details, go to the language-specific setup pages.
How to force a request to be traced
Cloud Trace doesn't sample every request. For example, if you use Java and OpenCensus, then only 1 request out of every 10,000 is traced. If you are using App Engine, requests are sampled at a rate of 0.1 requests per second for each App Engine instance. If you use the Cloud Trace API, then you can configure custom rates. Some packages, such as OpenTelemetry, support configuring the sampling rate.
If you configure a microservice with a sampling rate, then that rate applies only to requests that start at that microservice. If you don't configure a sampling rate for a microservice, then the sampling rate of the parent context determines the sampling rate for the microservice.
To force a specific request to be traced, add an X-Cloud-Trace-Context
header to the request. The header specification is:
"X-Cloud-Trace-Context: TRACE_ID/SPAN_ID;o=TRACE_TRUE"
Where:
TRACE_ID
is a 32-character hexadecimal value representing a 128-bit number. It should be unique between your requests, unless you intentionally want to bundle the requests together.SPAN_ID
is the decimal representation of the (unsigned) span ID. It should be randomly generated and unique in your trace. For subsequent requests, setSPAN_ID
to the span ID of the parent request. See the description ofTraceSpan
(REST, RPC) for more information about nested traces.TRACE_TRUE
must be1
to trace this request. Specify0
to not trace the request.
For example, to force a trace with curl:
curl "http://www.example.com" --header "X-Cloud-Trace-Context:
105445aa7843bc8bf206b12000100000/1;o=1"
How to create Cloud Trace exemplars
OpenCensus can create Cloud Trace exemplars
when it writes time-series data for gRPC services. Exemplars are example
points that can be attached to distribution measurements. For example,
you might want a latency metric to include a label for a URL. You can't
add a URL label to the metric because the metric would then have
a high cardinality, and that results in performance issues.
However, you can create an exemplar
that records the URL. For more information, see Exemplar
.
To view exemplars, do the following:
- Chart a distribution-value metric and set the chart type to Heatmap chart. You can create a chart on a dashboard or use Metrics Explorer.
- In the chart's toolbar, select
Trace exemplars
in the chart's toolbar. For examples, see The chart toolbar.
To create exemplars, you can write your own custom metrics or you can use libraries like OpenCensus. For general information about custom metrics, see the following:
- Introduction to custom metrics
- Create custom metrics with OpenCensus
- Create custom metrics with the Cloud Monitoring API
The following code sample shows how to write an exemplar:
Create the additional information to include with the bucket counts:
The
SpanContext
object stores a project ID, a trace ID, and a span ID. Ensure that these fields match values for a trace that was sent to Cloud Trace. Cloud Monitoring uses these fields to identify the trace to display.The
DroppedLabels
object defines the additional labels. The following code sample adds one label—the key is "Label" and the value is "Dropped". In the latency metric example previously described, a label with the key of "URL" might be added.
Write a
Point
to the time series:- The
TimeInterval
field defines the start and end times for the point. - The
DistributionValue
field specifies the data. The data includes the definition of theDistribution
buckets and the bucket values. This field also includes any exemplars to be written with the data. The example writes two exemplars, one of which includes the span context and dropped labels objects.
- The
Configuring your Google Cloud project
To use Cloud Trace, your Google Cloud project must have the Cloud Trace API enabled. This setting lets your Google Cloud project receive trace data from authenticated sources.
By default, Google Cloud projects have the Cloud Trace API enabled and you don't need to take any action. If you have modified the access scopes of your Google Cloud project and want to verify your settings, do the following:
-
In the navigation panel of the Google Cloud console, select APIs & Services, click Enable APIs and Services, and then enable the Cloud Trace API:
If API enabled is displayed, then the API is already enabled. If not, click the Enable button.
What's next
For detailed configuration information, samples, and links to GitHub and other open source repositories, go to the setup page for your language: