Tracing with OpenTelemetry

This library uses OpenTelemetry to automatically generate traces providing insight on calls to Cloud Spanner. For information on the benefits and utility of tracing, see the Cloud Trace docs.

To take advantage of these traces, we first need to install OpenTelemetry:

pip install opentelemetry-api opentelemetry-sdk
pip install opentelemetry-exporter-gcp-trace

We also need to tell OpenTelemetry which exporter to use. To export Spanner traces to Cloud Tracing, add the following lines to your application:

from opentelemetry import trace
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.sampling import TraceIdRatioBased
from opentelemetry.exporter.cloud_trace import CloudTraceSpanExporter
# BatchSpanProcessor exports spans to Cloud Trace
# in a seperate thread to not block on the main thread
from opentelemetry.sdk.trace.export import BatchSpanProcessor

# Create and export one trace every 1000 requests
sampler = TraceIdRatioBased(1/1000)
tracer_provider = TracerProvider(sampler=sampler)
tracer_provider.add_span_processor(
    # Initialize the cloud tracing exporter
    BatchSpanProcessor(CloudTraceSpanExporter())
)
observability_options = dict(
    tracer_provider=tracer_provider,

    # By default extended_tracing is set to True due
    # to legacy reasons to avoid breaking changes, you
    # can modify it though using the environment variable
    # SPANNER_ENABLE_EXTENDED_TRACING=false.
    enable_extended_tracing=False,
)
spanner = spanner.NewClient(project_id, observability_options=observability_options)

To get more fine-grained traces from gRPC, you can enable the gRPC instrumentation by the following

pip install opentelemetry-instrumentation opentelemetry-instrumentation-grpc

and then in your Python code, please add the following lines:

from opentelemetry.instrumentation.grpc import GrpcInstrumentorClient
grpc_client_instrumentor = GrpcInstrumentorClient()
grpc_client_instrumentor.instrument()

Generated spanner traces should now be available on Cloud Trace.

Tracing is most effective when many libraries are instrumented to provide insight over the entire lifespan of a request. For a list of libraries that can be instrumented, see the OpenTelemetry Integrations section of the OpenTelemetry Python docs

Annotating spans with SQL

By default your spans will be annotated with SQL statements where appropriate, but that can be a PII (Personally Identifiable Information) leak. Sadly due to legacy behavior, we cannot simply turn off this behavior by default. However you can control this behavior by setting

SPANNER_ENABLE_EXTENDED_TRACING=false

to turn it off globally or when creating each SpannerClient, please set observability_options.enable_extended_tracing=false