This document provides a brief overview on how to instrument your application for Cloud Trace. For detailed instructions on setting up Cloud Trace, see the language-specific setup pages.
Cloud Trace provides distributed tracing data for your applications. After instrumenting your application, you can inspect latency data for a single request and view the aggregate latency for an entire application in the Cloud Trace console.
When to instrument your application
When trace data isn't automatically captured, you need to instrument your application to collect this data.
You can instrument your application so that it collects application-specific information. Several open-source instrumentation frameworks let you collect metrics, logs, and traces from your application and send that data to any vendor, including Google Cloud. To instrument your application, we recommend that you use a vendor-neutral instrumentation framework that is open source, such as OpenTelemetry, instead of vendor- and product-specific APIs or client libraries.
For information about instrumenting your applications by using vendor-neutral instrumentation frameworks, see Instrumentation and observability.
How to instrument applications
To instrument your applications to collect trace data, you can do any of the following:
You can use OpenTelemetry and the associated Cloud Trace exporter for the following programming languages:
OpenTelemetry SDK Example Go SDK Trace and metrics example for Go Java SDK Trace and metrics example for Java Node.js SDK Trace and metrics example for Node.js Python SDK Trace and metrics example for Python C++ SDK Trace example for C++ Ruby SDK See OpenTelemetry documentation. If you are writing applications that run on Compute Engine, then you can use the Ops Agent and the OpenTelemetry Protocol (OTLP) receiver to collect traces and metrics from your application. The Ops Agent can also collect logs, but not by using OTLP. For more information, see Use the Ops Agent and OTLP and Ops Agent overview.
You can use the client libraries or directly call the Cloud Trace API to send tracing data to Cloud Trace. However, we recommend that you use OpenTelemetry when your language is supported by that library.
You can configure a Zipkin server to receive traces from Zipkin clients and then forward those traces to Cloud Trace for analysis. For information about this approach, see Using Cloud Trace with Zipkin.
You can configure Spring Boot applications to forward the trace data it collects to Cloud Trace. For information about this procedure, see Spring Cloud for Google Cloud: Cloud Trace.
When to create spans
The Cloud Trace client libraries typically maintain a global trace context that holds information about the current span, including its trace ID and whether the trace is sampled. These libraries usually create spans on RPC boundaries. However, you might need to create spans if the default creation algorithm isn't sufficient for your needs.
The current active span can be accessed by the global trace context, which is sometimes wrapped in a Tracer object. You can add information relevant to your application by using custom annotations and tags to existing spans, or you can create new child spans with their own annotations and tags to trace the application's behavior with finer granularity. Because the context is global, multi-threaded applications that update the context must use appropriate isolation.
When to provide authentication credentials
You generally don't need to provide authentication credentials to your application or specify your Google Cloud project ID in your application when you are running on Google Cloud. For some languages, you do need to specify your Google Cloud project ID even if you are running on Google Cloud. Also, if you use Autopilot mode for Google Kubernetes Engine, or if you enable Workload Identity Federation for GKE, then you must configure your application to use Workload Identity Federation for GKE.
If you are running outside of Google Cloud, you need to provide authentication credentials to your application. You also need to specify your Google Cloud project ID in your application.
For details, go to the language-specific setup pages.
How to force a request to be traced
Unless your application always samples every span,
it isn't possible, in general, to force a request to be traced end-to-end
because each component in an end-to-end request makes its own
sampling decision. However, you can influence the
decision by adding to the trace header a sampled
flag,
with this flag set to true
. This setting is a hint to child components
to sample the request.
For more information about trace headers, see
Protocols for context propagation.
For downstream components whose code you own, you must determine whether
your instrumentation logic respects the sampled
flag.
For example, when using OpenTelemetry
for instrumentation, you can use the ParentBased
sampler
to ensure that the parent sampled flag is respected.
Google Cloud services that record tracing information to Cloud Trace typically accept the parent sampling flag as a hint; however, most services also rate-limit sampling. Each Google Cloud service determines whether it supports tracing, how the parent sampling flag is utilized, and the rate limit on sampling.
How to correlate metric and trace data
You can correlate distribution-valued metric data with traces by attaching exemplars to the metric data points. Provided you complete the necessary configuration steps, OpenTelemetry, which is the recommended instrumentation library, automatically adds these exemplars. For more information, see Correlate metrics and traces by using exemplars.
Configure your project and platform
Ensure that the Cloud Trace API is enabled.
By default, Google Cloud projects have the Cloud Trace API enabled and you don't need to take any action. However, security constraints defined by your organization might have disabled the API. For troubleshooting information, see Develop applications in a constrained Google Cloud environment.
Enable the Cloud Trace API.
Configure your platform.
You can use Cloud Trace on Google Cloud and other platforms.
Google Cloud: When your application is running on Google Cloud, you don't need to provide authentication credentials in the form of a service account to the client library. However, must ensure that your Google Cloud platform has the Cloud Trace API access scope enabled.
For the following configurations, the default access-scope settings include the Cloud Trace API access scope:
If you use custom access scopes, then you must ensure that Cloud Trace API access scope is enabled. For example, if you use the Google Cloud CLI to create a GKE cluster and if you specify the
--scopes
flag, then ensure the scope includestrace.append
. The following command illustrates setting the--scopes
flag:gcloud container clusters create example-cluster-name --scopes=https://www.googleapis.com/auth/trace.append
Running locally and elsewhere: If your application is running outside of Google Cloud, then you must provide authentication credentials in the form of a service account to the client library. The service account must be granted the Cloud Trace agent (
roles/cloudtrace.agent
) role. For information about roles, see Control access with IAM.Google Cloud client libraries use Application Default Credentials (ADC) to find your application's credentials. You can provide these credentials in one of three ways:
Run
gcloud auth application-default login
Place the service account in a default path for your operating system. The following lists the default paths for Windows and Linux:
Windows:
%APPDATA%/gcloud/application_default_credentials.json
Linux:
$HOME/.config/gcloud/application_default_credentials.json
Set the
GOOGLE_APPLICATION_CREDENTIALS
environment variable to the path to your service account:Linux/macOS
export GOOGLE_APPLICATION_CREDENTIALS=path-to-your-service-accounts-private-key
Windows
set GOOGLE_APPLICATION_CREDENTIALS=path-to-your-service-accounts-private-key
PowerShell:
$env:GOOGLE_APPLICATION_CREDENTIALS="path-to-your-service-accounts-private-key"
What's next
For detailed configuration information, samples, and links to GitHub and other open source repositories, go to the setup page for your language.
OpenTelemetry examples:
Client library examples: