Set up trace collection using OpenTelemetry

This document shows you how to set up client-side and end-to-end tracing using OpenTelemetry. You need to set up client-side tracing before you can opt in for end-to-end tracing. For more information, see Trace collection overview.

Before you begin

• To ensure that the service account your application uses has the necessary permissions to set up trace collection, ask your administrator to grant the service account your application uses the Cloud Trace Agent (roles/cloudtrace.agent) IAM role on your project.

Configure client-side tracing

To configure client-side tracing, you need to export traces. You can export traces to a collector or directly to an observability backend. You can configure tracing using OpenTelemetry APIs.

Export traces to a collector using OpenTelemetry APIs

To export traces to a collector using OpenTelemetry APIs, configure the OpenTelemetry SDK and OLTP exporter:

1. Add the necessary dependencies to your application using the following code:

   Java

   <dependency>
     <groupId>com.google.cloud</groupId>
     <artifactId>google-cloud-spanner</artifactId>
   </dependency>
   <dependency>
     <groupId>io.opentelemetry</groupId>
     <artifactId>opentelemetry-sdk</artifactId>
   </dependency>
   <dependency>
     <groupId>io.opentelemetry</groupId>
     <artifactId>opentelemetry-sdk-trace</artifactId>
   </dependency>
   <dependency>
     <groupId>io.opentelemetry</groupId>
     <artifactId>opentelemetry-exporter-otlp</artifactId>
   </dependency>

   Go

   go.opentelemetry.io/otel v1.28.0
   go.opentelemetry.io/otel/sdk v1.28.0
   go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracegrpc v1.28.0

2. Configure the OpenTelemetry object and enable tracing.

   Java

   Resource resource = Resource
       .getDefault().merge(Resource.builder().put("service.name", "My App").build());
   
   OtlpGrpcSpanExporter otlpGrpcSpanExporter =
       OtlpGrpcSpanExporter
           .builder()
           .setEndpoint(otlpEndpoint) // Replace with your OTLP endpoint
           .build();
   
   // Using a batch span processor
   // You can use `.setScheduleDelay()`, `.setExporterTimeout()`,
   // `.setMaxQueueSize`(), and `.setMaxExportBatchSize()` to further customize.
   BatchSpanProcessor otlpGrpcSpanProcessor =
       BatchSpanProcessor.builder(otlpGrpcSpanExporter).build();
   
   // Create a new tracer provider
   sdkTracerProvider = SdkTracerProvider.builder()
       // Use Otlp exporter or any other exporter of your choice.
       .addSpanProcessor(otlpGrpcSpanProcessor)
       .setResource(resource)
       .setSampler(Sampler.traceIdRatioBased(0.1))
       .build();
   
   // Export to a collector that is expecting OTLP using gRPC.
   OpenTelemetry openTelemetry = OpenTelemetrySdk.builder()
       .setTracerProvider(sdkTracerProvider).build();
   
   // Enable OpenTelemetry traces before Injecting OpenTelemetry
   SpannerOptions.enableOpenTelemetryTraces();
   
   // Inject OpenTelemetry object via Spanner options or register as GlobalOpenTelemetry.
   SpannerOptions options = SpannerOptions.newBuilder()
       .setOpenTelemetry(openTelemetry)
       .build();
   Spanner spanner = options.getService();

   Go

   
   // Ensure that your Go runtime version is supported by the OpenTelemetry-Go
   // compatibility policy before enabling OpenTelemetry instrumentation.
   
   // Enable OpenTelemetry traces by setting environment variable GOOGLE_API_GO_EXPERIMENTAL_TELEMETRY_PLATFORM_TRACING
   // to the case-insensitive value "opentelemetry" before loading the client library.
   
   ctx := context.Background()
   
   // Create a new resource to uniquely identify the application
   res, err := resource.Merge(resource.Default(),
   	resource.NewWithAttributes(semconv.SchemaURL,
   		semconv.ServiceName("My App"),
   		semconv.ServiceVersion("0.1.0"),
   	))
   if err != nil {
   	log.Fatal(err)
   }
   
   // Create a new OTLP exporter.
   defaultOtlpEndpoint := "http://localhost:4317" // Replace with the endpoint on which OTLP collector is running
   traceExporter, err := otlptracegrpc.New(ctx, otlptracegrpc.WithEndpoint(defaultOtlpEndpoint))
   if err != nil {
   	log.Fatal(err)
   }
   
   // Create a new tracer provider
   tracerProvider := sdktrace.NewTracerProvider(
   	sdktrace.WithResource(res),
   	sdktrace.WithBatcher(traceExporter),
   	sdktrace.WithSampler(sdktrace.TraceIDRatioBased(0.1)),
   )
   
   // Register tracer provider as global
   otel.SetTracerProvider(tracerProvider)

Export directly to an observability backend using OpenTelemetry APIs

To configure Spanner client libraries to directly export trace spans to Cloud Trace or another observability service provider backend, follow these steps:

1. Add the necessary dependencies to your application using the following code:

   Java

   <dependency>
   <groupId>com.google.cloud</groupId>
   <artifactId>google-cloud-spanner</artifactId>
   </dependency>
   <dependency>
   <groupId>io.opentelemetry</groupId>
   <artifactId>opentelemetry-api</artifactId>
   </dependency>
   <dependency>
   <groupId>io.opentelemetry</groupId>
   <artifactId>opentelemetry-sdk</artifactId>
   </dependency>
   <dependency>
   <groupId>io.opentelemetry</groupId>
   <artifactId>opentelemetry-sdk-common</artifactId>
   </dependency>
   <dependency>
   <groupId>io.opentelemetry</groupId>
   <artifactId>opentelemetry-sdk-trace</artifactId>
   </dependency>
   <dependency>
   <groupId>com.google.cloud.opentelemetry</groupId>
   <artifactId>exporter-trace</artifactId>
   <version>0.30.0</version>
   </dependency>

   Go

   go.opentelemetry.io/otel v1.28.0
   go.opentelemetry.io/otel/sdk v1.28.0
   github.com/GoogleCloudPlatform/opentelemetry-operations-go/exporter/trace v1.24.1

2. Configure the OpenTelemetry object and enable tracing.

   Java

   Resource resource = Resource
       .getDefault().merge(Resource.builder().put("service.name", "My App").build());
   
   SpanExporter traceExporter = TraceExporter.createWithConfiguration(
       TraceConfiguration.builder().setProjectId(projectId).build()
   );
   
   // Using a batch span processor
   // You can use `.setScheduleDelay()`, `.setExporterTimeout()`,
   // `.setMaxQueueSize`(), and `.setMaxExportBatchSize()` to further customize.
   BatchSpanProcessor otlpGrpcSpanProcessor =
       BatchSpanProcessor.builder(traceExporter).build();
   
   // Create a new tracer provider
   sdkTracerProvider = SdkTracerProvider.builder()
       // Use Otlp exporter or any other exporter of your choice.
       .addSpanProcessor(otlpGrpcSpanProcessor)
       .setResource(resource)
       .setSampler(Sampler.traceIdRatioBased(0.1))
       .build();
   
   // Export to a collector that is expecting OTLP using gRPC.
   OpenTelemetry openTelemetry = OpenTelemetrySdk.builder()
       .setTracerProvider(sdkTracerProvider).build();
   
   // Enable OpenTelemetry traces before Injecting OpenTelemetry
   SpannerOptions.enableOpenTelemetryTraces();
   
   // Inject OpenTelemetry object via Spanner options or register it as global object.
   // To register as the global OpenTelemetry object,
   // use "OpenTelemetrySdk.builder()....buildAndRegisterGlobal()".
   SpannerOptions options = SpannerOptions.newBuilder()
       .setOpenTelemetry(openTelemetry)
       .build();
   Spanner spanner = options.getService();

   Go

   
   // Ensure that your Go runtime version is supported by the OpenTelemetry-Go
   // compatibility policy before enabling OpenTelemetry instrumentation.
   
   // Enable OpenTelemetry traces by setting environment variable GOOGLE_API_GO_EXPERIMENTAL_TELEMETRY_PLATFORM_TRACING
   // to the case-insensitive value "opentelemetry" before loading the client library.
   
   // Create a new resource to uniquely identify the application
   res, err := resource.Merge(resource.Default(),
   	resource.NewWithAttributes(semconv.SchemaURL,
   		semconv.ServiceName("My App"),
   		semconv.ServiceVersion("0.1.0"),
   	))
   if err != nil {
   	log.Fatal(err)
   }
   
   // Create a new cloud trace exporter
   exporter, err := traceExporter.New(traceExporter.WithProjectID(projectID))
   if err != nil {
   	log.Fatal(err)
   }
   
   // Create a new tracer provider
   tracerProvider := sdktrace.NewTracerProvider(
   	sdktrace.WithResource(res),
   	sdktrace.WithBatcher(exporter),
   	sdktrace.WithSampler(sdktrace.TraceIDRatioBased(0.1)),
   )
   
   // Register tracer provider as global
   otel.SetTracerProvider(tracerProvider)

Configure end-to-end tracing

This section provides instructions for configuring end-to-end tracing (Preview) on Spanner client libraries:

1. Add the necessary dependencies to your application using the following code:

   Java

   The existing client-side tracing dependencies are sufficient for configuring end-to-end tracing. You don't need any additional dependencies.

   Go

   In addition to the dependencies you need for client-side tracing, you also need the following dependency:

   go.opentelemetry.io/otel/propagation v1.28.0

2. Opt in for end-to-end tracing.

   Java

   SpannerOptions options = SpannerOptions.newBuilder()
     .setOpenTelemetry(openTelemetry)
     .setEnableEndToEndTracing(/* enableEndtoEndTracing= */ true)
     .build();

   Go

   Use the EnableEndToEndTracing option in the client configuration to opt in.

   client, _ := spanner.NewClientWithConfig(ctx, "projects/test-project/instances/test-instance/databases/test-db", spanner.ClientConfig{
   SessionPoolConfig: spanner.DefaultSessionPoolConfig,
   EnableEndToEndTracing:      true,
   }, clientOptions...)

3. Set the trace context propagation in OpenTelemetry.

   Java

   OpenTelemetry openTelemetry = OpenTelemetrySdk.builder()
     .setTracerProvider(sdkTracerProvider)
     .setPropagators(ContextPropagators.create(W3CTraceContextPropagator.getInstance()))
     .buildAndRegisterGlobal();

   Go

   // Register the TraceContext propagator globally.
   otel.SetTextMapPropagator(propagation.TraceContext{})

End-to-end tracing attributes

End-to-end traces can include the following information:

Attribute name	Description
service.name	The attribute value is always spanner_api_frontend.
cloud.region	The Google Cloud cloud region of the Spanner API frontend that serves your application request.
gcp.spanner.query.fingerprint	The attribute value is the query fingerprint. To debug this query further, see the TEXT_FINGERPRINT column in the Query statistics tables.
gcp.spanner.participants.count	The number of participants involved in the transaction. For more information, see Life of Spanner Reads & Writes.

Sample trace

An end-to-end trace lets you view the following details:

• The latency between your application and Spanner. You can calculate network latency to see if you have any network issues.
• The Spanner API frontend cloud region where your application requests are being served from. You can use this to check for cross-region calls between your application and Spanner.

In the following example, your application request is being served by the Spanner API frontend in the us-west1 region and the network latency is 8.542ms (55.47ms - 46.928ms).

What's next

• For more information about OpenTelemetry, see OpenTelemetry documentation.