Troubleshoot

This page contains troubleshooting information for Trace.

No data in the Trace interface

If you can't view any traces in your Google Cloud project when you expect trace data to be present, then try the following steps:

1. In the navigation panel of the Google Cloud console, select APIs & Services, click Enable APIs and Services, and then enable the Cloud Trace API:

   Go to Cloud Trace API settings

2. If API enabled is displayed, then the API is already enabled. If not, click the Enable button.

   Note: There might be org policies in place that have disabled the Cloud Trace API. These policies prevent you from enabling the API.

3. In the navigation panel of the Google Cloud console, select APIs & Services, and find the row labeled Cloud Trace API.

   Go to APIs & Services settings

   Try the following:

   • If the column labeled Requests doesn't list any numeric information, then no trace data is being sent to your Google Cloud project.

     To resolve this case, check your applications and proxies to ensure they are configured to send the traces to the correct project.

   • Select Cloud Trace API, select the Metrics tab, and then find the chart labeled Errors by API method:

     • If writes are failing, then grant the service account that is providing the authentication credentials the role of Cloud Trace Agent (roles/cloudtrace.agent). This role includes the permission cloudtrace.traces.patch, which lets applications write span data to a Google Cloud project.

       For more information, see Cloud Trace IAM roles.

     • If reads are failing, then ensure that your IAM role on the Google Cloud project includes the permissions in the role Cloud Trace User (roles/cloutrace.user). For a list of permissions for this role, see Cloud Trace IAM roles.

No trace data after updating Go app to use OpenTelemetry

Your application relies on the client library to capture traces, and after updating your application to use OpenTelemetry, you no longer see Cloud Trace data.

Because some Google client libraries for Go are integrated with OpenCensus, you must use an OpenCensus Bridge. For more information about the problem solved by the bridge, see OpenCensus Bridge.

For information about the update of the Google client library for Go, see Issue #4237.

No trace data for deployed app

You've deployed an app that sends data to your Google Cloud project by using the Cloud Trace API, yet trace data isn't collected.

Try the following:

• If you aren't seeing any data in the Trace Explorer page of the Google Cloud console, then follow the steps in the section titled No data in the Trace interface.

• If the app isn't deployed on Google Cloud or if it uses a service account to provide authentication credentials, then ensure that the service account has been granted the role of Cloud Trace Agent (roles/cloudtrace.agent).

  This role includes the permission cloudtrace.traces.patch, which lets applications write span data to a Google Cloud project.

• If the app relies on OpenTelemetry, then do the following:

  • For the root service, try updating your environment variables so that OpenTelemetry uses the traceidratio sampler with a sampling rate of 0.5:

    export OTEL_TRACES_SAMPLER="traceidratio"
    export OTEL_TRACES_SAMPLER_ARG="0.5"
    

  • For all other services, leave the OTEL_TRACES_SAMPLER environment variable unset to use the default sampler, which is parentbased_always_on. The default setting means the sampling decision for a span is inherited from its parent span, if such a span exists. If a parent span doesn't exist, then the span is sampled.

  Unless your application always samples every span, it isn't possible, in general, to force a request to be traced end-to-end because each component in an end-to-end request makes its own sampling decision. However, you can influence the decision by adding to the trace header a sampled flag, with this flag set to true. This setting is a hint to child components to sample the request. For more information about trace headers, see Protocols for context propagation.

Missing span ID message in trace

Your trace contains a "Missing span ID" message.

In distributed tracing systems, incomplete traces are expected. A trace is incomplete when a sampled span contains a reference to another span that hasn't been received. The unresolved reference can occur for the following reasons:

• The referenced span wasn't sampled.
• The referenced span was sampled but hasn't yet been received by Cloud Trace or the span was received but not stored.

When you are viewing an incomplete trace, Cloud Trace displays the message "Missing span ID" on the trace details pane.

If you are consistently seeing the message "Missing span ID", then try the following:

• For components that you manage, ensure that they respect and propagate the header's sampled flag flag, when this field is present. This setting is a hint to child components to sample the request. For more information about trace headers, see Protocols for context propagation.

  Google Cloud services typically respect this hint. However, they also limit the rate at which they write trace data.

• If you are using Anthos Service Mesh, then ensure that you follow the guidance for propagating the trace context for those configurations. For Anthos Service Mesh guidance, see Trace context propagation.