Instrument for Cloud Trace

This document provides a brief overview on how to instrument your application for Cloud Trace. For detailed instructions on setting up Cloud Trace, see the language-specific setup pages.

Cloud Trace provides distributed tracing data for your applications. After instrumenting your application, you can inspect latency data for a single request and view the aggregate latency for an entire application in the Cloud Trace console.

Cloud Trace recommends using OpenTelemetry. OpenTelemetry is an open-source product from the merger between OpenCensus and OpenTracing.

When to instrument your application

For your application to submit traces to Cloud Trace, it must be instrumented. You can instrument your code by using the Google client libraries or by using OpenTelemetry, which is an open source tracing package. We recommend that you use OpenTelemetry.

Note: Trace data is automatically captured for some configurations, so you might not need to instrument your application. For more information, see Configurations with automatic tracing.

How to instrument applications

There are multiple ways to implement tracing for your applications:

• Use OpenTelemetry and the associated Cloud Trace client library. This is the recommended way to instrument your applications.

• Use the Cloud Trace API and write custom methods to send tracing data to Cloud Trace.

• Configure a Zipkin server to receive traces from Zipkin clients and then forward those traces to Cloud Trace for analysis. For information about this approach, see Using Cloud Trace with Zipkin.

• Configure Spring Boot applications to forward the trace data it collects to Cloud Trace. For information about this procedure, see Spring Cloud for Google Cloud: Cloud Trace.

Recommended client libraries

The following table lists the recommended instrumentation for each programming language:

Language	Recommended instrumentation
C# .NET	OpenTelemetry .NET
Go	OpenTelemetry Go
Java	OpenTelemetry Java
Node.js	OpenTelemetry JavaScript
PHP	OpenTelemetry PHP
Python	OpenTelemetry Python
Ruby	OpenTelemetry Ruby

When to create spans

The Cloud Trace client libraries typically maintain a global trace context that holds information about the current span, including its trace ID and whether the trace is sampled. These libraries usually create spans on RPC boundaries. However, you might need to create spans if the default creation algorithm isn't sufficient for your needs.

The current active span can be accessed by the global trace context, which is sometimes wrapped in a Tracer object. You can add information relevant to your application by using custom annotations and tags to existing spans, or you can create new child spans with their own annotations and tags to trace the application's behavior with finer granularity. Because the context is global, multi-threaded applications that update the context must use appropriate isolation.

When to provide authentication credentials

You generally don't need to provide authentication credentials to your application or specify your Google Cloud project ID in your application when you are running on Google Cloud. For some languages, you do need to specify your Google Cloud project ID even if you are running on Google Cloud. Also, if you use Autopilot mode for Google Kubernetes Engine, or if you enable Workload Identity, then you must configure your application to use Workload Identity.

If you are running outside of Google Cloud, you need to provide authentication credentials to your application. You also need to specify your Google Cloud project ID in your application.

For details, go to the language-specific setup pages.

How to force a request to be traced

Cloud Trace doesn't sample every request. For example, if you use Java and OpenCensus, then only 1 request out of every 10,000 is traced. If you are using App Engine, requests are sampled at a rate of 0.1 requests per second for each App Engine instance. If you use the Cloud Trace API, then you can configure custom rates. Some packages, such as OpenTelemetry, support configuring the sampling rate.

If you configure a microservice with a sampling rate, then that rate applies only to requests that start at that microservice. If you don't configure a sampling rate for a microservice, then the sampling rate of the parent context determines the sampling rate for the microservice.

To force a specific request to be traced, add an X-Cloud-Trace-Context header to the request. The header specification is:

"X-Cloud-Trace-Context: TRACE_ID/SPAN_ID;o=TRACE_TRUE"

Where:

• TRACE_ID is a 32-character hexadecimal value representing a 128-bit number. It should be unique between your requests, unless you intentionally want to bundle the requests together.

• SPAN_ID is the decimal representation of the (unsigned) span ID. It should be randomly generated and unique in your trace. For subsequent requests, set SPAN_ID to the span ID of the parent request. See the description of TraceSpan (REST, RPC) for more information about nested traces.

• TRACE_TRUE must be 1 to trace this request. Specify 0 to not trace the request.

For example, to force a trace with curl:

curl "http://www.example.com" --header "X-Cloud-Trace-Context:
  105445aa7843bc8bf206b12000100000/1;o=1"

How to create Cloud Trace exemplars

OpenCensus can create Cloud Trace exemplars when it writes time-series data for gRPC services. Exemplars are example points that can be attached to distribution measurements. For example, you might want a latency metric to include a label for a URL. You can't add a URL label to the metric because the metric would then have a high cardinality, and that results in performance issues. However, you can create an exemplar that records the URL. For more information, see Exemplar.

To view exemplars, do the following:

1. Chart a distribution-value metric and set the chart type to Heatmap chart. You can create a chart on a dashboard or use Metrics Explorer.
2. In the chart's toolbar, select Trace exemplars in the chart's toolbar. For examples, see The chart toolbar.

To create exemplars, you can write your own custom metrics or you can use libraries like OpenCensus. For general information about custom metrics, see the following:

• Introduction to custom metrics
• Create custom metrics with OpenCensus
• Create custom metrics with the Cloud Monitoring API

The following code sample shows how to write an exemplar:

1. Create the additional information to include with the bucket counts:

   • The SpanContext object stores a project ID, a trace ID, and a span ID. Ensure that these fields match values for a trace that was sent to Cloud Trace. Cloud Monitoring uses these fields to identify the trace to display.

   • The DroppedLabels object defines the additional labels. The following code sample adds one label—the key is "Label" and the value is "Dropped". In the latency metric example previously described, a label with the key of "URL" might be added.

2. Write a Point to the time series:

   • The TimeInterval field defines the start and end times for the point.
   • The DistributionValue field specifies the data. The data includes the definition of the Distribution buckets and the bucket values. This field also includes any exemplars to be written with the data. The example writes two exemplars, one of which includes the span context and dropped labels objects.

View on GitHub Feedback

import (
	"fmt"
	"time"

	"cloud.google.com/go/monitoring/apiv3/v2/monitoringpb"
	googlepb "github.com/golang/protobuf/ptypes/timestamp"
	distributionpb "google.golang.org/genproto/googleapis/api/distribution"
	"google.golang.org/protobuf/types/known/anypb"
)

// Generates metric TimeSeries points containing Exemplars with attached tracing span.
func createDataPointWithExemplar(projectID string) (*monitoringpb.Point, error) {
	// projectID := "my-cloud-project-id"
	end := time.Now().Unix()
	traceId := "0000000000000001"
	spanId := "00000001"
	spanCtx, err := anypb.New(&monitoringpb.SpanContext{
		SpanName: fmt.Sprintf("projects/%s/traces/%s/spans/%s", projectID, traceId, spanId),
	})
	if err != nil {
		return nil, err
	}
	droppedLabels, err := anypb.New(&monitoringpb.DroppedLabels{
		Label: map[string]string{"Label": "Dropped"},
	})
	if err != nil {
		return nil, err
	}
	dataPoint := &monitoringpb.Point{
		Interval: &monitoringpb.TimeInterval{
			StartTime: &googlepb.Timestamp{Seconds: end - 60},
			EndTime:   &googlepb.Timestamp{Seconds: end},
		},
		Value: &monitoringpb.TypedValue{Value: &monitoringpb.TypedValue_DistributionValue{
			DistributionValue: &distributionpb.Distribution{
				Count: 14,
				BucketOptions: &distributionpb.Distribution_BucketOptions{Options: &distributionpb.Distribution_BucketOptions_LinearBuckets{
					LinearBuckets: &distributionpb.Distribution_BucketOptions_Linear{NumFiniteBuckets: 2, Width: 3, Offset: 0},
				}},
				BucketCounts: []int64{5, 6, 3},
				Exemplars: []*distributionpb.Distribution_Exemplar{
					{Value: 1, Timestamp: &googlepb.Timestamp{Seconds: end - 30}, Attachments: []*anypb.Any{spanCtx, droppedLabels}},
					{Value: 4, Timestamp: &googlepb.Timestamp{Seconds: end - 30}},
				},
			},
		}},
	}
	return dataPoint, nil
}

Configuring your Google Cloud project

To use Cloud Trace, your Google Cloud project must have the Cloud Trace API enabled. This setting lets your Google Cloud project receive trace data from authenticated sources.

By default, Google Cloud projects have the Cloud Trace API enabled and you don't need to take any action. If you have modified the access scopes of your Google Cloud project and want to verify your settings, do the following:

1. From the Google Cloud console, go to APIs and Services:

   Go to APIs and Services

2. Click Enable APIs and Services

3. In the search bar, enter Trace API.

4. If API enabled is displayed, this API is already enabled and there is nothing for you to do. Otherwise, click Enable.

What's next

For detailed configuration information, samples, and links to GitHub and other open source repositories, go to the setup page for your language:

• Python

• Java

• Node.js

• Go

• PHP

• Ruby

• ASP.NET core

• ASP.NET