Troubleshoot

This page contains troubleshooting information for Trace.

No data in the Trace interface

If you can't view any traces in your Google Cloud project when you expect trace data to be present, then try the following steps:

  1. In the navigation panel of the Google Cloud console, select APIs & Services, click Enable APIs and Services, and then enable the Cloud Trace API:

    Go to Cloud Trace API settings

  2. If API enabled is displayed, then the API is already enabled. If not, click the Enable button.

  3. In the toolbar, click Navigation menu and select APIs & Services. If the row for the Cloud Trace API doesn't list any numeric values for the Requests column, Errors column, or for other columns, then no trace requests are reaching your Google Cloud project. Check your applications and proxies to ensure they are configured to send the traces to the correct project.

  4. To view details about the API usage, click Cloud Trace API, and then select the Metrics tab.

    To narrow down the source of the errors, use the Credentials and Methods filters and view the effect your selections have on the Errors by API method chart:

    • If all writes are failing, ensure that your service account has access to the Cloud Trace Agent role. This role includes the permission cloudtrace.traces.patch. For more information, see Cloud Trace IAM roles.

    • If reads are failing, ensure that you have access to the Cloud Trace User role for this Google Cloud project. For a full list of permissions for this role, see Cloud Trace IAM roles.

No trace data after updating Go app to use OpenTelemetry

Your application relies on the client library to capture traces, and after updating your application to use OpenTelemetry, you no longer see Cloud Trace data.

Because some Google client libraries for Go are integrated with OpenCensus, you must use an OpenCensus Bridge. For more information about the problem solved by the bridge, see OpenCensus Bridge.

For information about the update of the Google client library for Go, see Issue #4237.

No trace data for deployed app

You deployed your application and instrumented it to collect trace data by using OpenTelemetry, yet trace data isn't collected.

For the root service, try updating your environment variables so that OpenTelemetry uses the traceidratio sampler with a sampling rate of 0.5:

export OTEL_TRACES_SAMPLER="traceidratio"
export OTEL_TRACES_SAMPLER_ARG="0.5"

For all other services, leave the OTEL_TRACES_SAMPLER environment variable unset to use the default sampler, which is parentbased_always_on. The default setting means the sampling decision for a span is inherited from its parent span, if such a span exists. If a parent span doesn't exist, then the span is sampled.

Unless your application always samples every span, it isn't possible, in general, to force a request to be traced end-to-end because each component in an end-to-end request makes its own sampling decision. However, you can influence the decision by adding to the trace header a sampled flag, with this flag set to true. This setting is a hint to child components to sample the request. For more information about trace headers, see Protocols for context propagation.

Missing span ID message in trace

Your trace contains a "Missing span ID" message.

In distributed tracing systems, incomplete traces are expected. A trace is incomplete when a sampled span contains a reference to another span that hasn't been received. The unresolved reference can occur for the following reasons:

  • The referenced span wasn't sampled.
  • The referenced span was sampled but hasn't yet been received by Cloud Trace or the span was received but not stored.

When you are viewing an incomplete trace, Cloud Trace displays the message "Missing span ID" on the trace details pane.

If you are consistently seeing the message "Missing span ID", then try the following:

  • For components that you manage, ensure that they respect and propagate the header's sampled flag flag, when this field is present. This setting is a hint to child components to sample the request. For more information about trace headers, see Protocols for context propagation.

    Google Cloud services typically respect this hint. However, they also limit the rate at which they write trace data.

  • If you are using Anthos Service Mesh, then ensure that you follow the guidance for propagating the trace context for those configurations. For Anthos Service Mesh guidance, see Trace context propagation.