Troubleshoot

This page contains troubleshooting information for Trace.

This document describes how to troubleshoot issues when you are using the Trace Explorer page or the legacy Trace Explorer page:

Troubleshoot the Trace Explorer page
Troubleshoot the legacy Trace Explorer page

Known issues

There are two known issues with the Trace Explorer page:

Spans written to your Google Cloud project before January 11, 2025 might not be available. These spans are available in the legacy Trace Explorer page.
The event details pane for spans written to your Google Cloud project before January 11, 2025 might be displayed differently than for spans written after that date.

Troubleshoot the Trace Explorer page

This section describes how to troubleshoot issues when you are using the Trace Explorer page.

No data in the Trace Explorer page

You are using the Trace Explorer page and you can't view any traces in your Google Cloud project when you expect trace data to be present.

Try the following steps:

Verify that the Cloud Trace API is enabled and that data is being written to your project:
1. In the Google Cloud console, go to the Enabled APIs & Services page:
  
  Go to Enabled APIs & Services
2. If the Cloud Trace API is listed, then proceed to the next step. Otherwise, enable the API.
  
  To enable the API, click Enable APIs and Services, search for "Cloud Trace API", select the option, and then click Enable.
  
  Note: There might be org policies in place that have disabled the Cloud Trace API. These policies prevent you from enabling the API.
On the Enable APIs and Services page, find the row labeled Cloud Trace API.
If the column labeled Requests doesn't list any numeric information, then no trace data is being sent to your Google Cloud project.

To resolve this case, check your applications and proxies to ensure they are configured to send the traces to the correct project.
If the column labeled Error lists a non-zero value, then there are errors reading or writing trace data. To get more information about the source of the errors, select Cloud Trace API, select the Metrics tab, and then find the chart labeled Errors by API method:
- If writes are failing, then grant the service account that is providing the authentication credentials the Cloud Trace Agent role (roles/cloudtrace.agent). This role includes the permission cloudtrace.traces.patch, which lets applications write span data to a Google Cloud project.
  
  For more information, see Cloud Trace IAM roles.
- If reads are failing, then ensure that your IAM role on the Google Cloud project includes the permissions in the Cloud Trace User role (roles/cloutrace.user). For a list of permissions for this role, see Cloud Trace IAM roles.
Verify the Trace Explorer page is searching your current project for trace data:
1. In the Google Cloud console, go to the Trace explorer page:
  Go to Trace explorer
  
  You can also find this page by using the search bar.
2. In the toolbar, go to the Scope element, expand the Refine scope menu, select Current project, and then select Apply.
Try one of the following:
- Switch to the legacy Trace Explorer page. This page reads trace and span data from a different database than the Trace Explorer page.
  
  Go to Legacy Trace explorer
- Use the Cloud Trace API to send a span to your Google Cloud project. For more information, see Force creation of database for the Trace Explorer page.

Missing older data in the Trace Explorer page

You are using the Trace Explorer page and you can view recent data, but when you set the time-range selector to 30 days or to a larger value, the older data isn't shown.

The Trace Explorer page doesn't display data for time periods that are larger than the data retention period of Cloud Trace, which is 30 days.

If the time-range selector is 30 days or less, then the missing data indicates that the database queried by the Trace Explorer page queries was created more recently than your time-range setting. For example, if you set this value to 20 days and you can only see the most recent 10 days of data, then the database was created 10 days in the past. Also, this database only contains traces that were sent to your Google Cloud project after the database was created.

To view and analyze older trace data, switch to the legacy Trace Explorer page. This page reads trace and span data from a different database than the Trace Explorer page.

Go to Legacy Trace explorer

Missing spans in a trace

You open the Trace Explorer page and select a span to view. The Details flyout shows the trace, but some spans are missing.

Spans might be missing for the following reasons:

The Trace Explorer page isn't searching all Google Cloud projects that store span data for the trace.
Your IAM role on a Google Cloud project that stores span data for the trace doesn't contain the permissions necessary to view trace data.
There is an instrumentation issue. For example, only some spans in a trace were sent to your Google Cloud project.
They were written to your Google Cloud project before January 11, 2025.

To resolve these problems, do the following:

Determine whether the trace is complete in the legacy Trace Explorer page:
1. From the Details flyout for the trace, copy the trace ID into your clipboard.
2. Go to the legacy Trace Explorer page:
  
  Go to Legacy Trace explorer
3. Paste the trace ID into the Trace ID field.
  
  The Trace details page is updated and lists the projects that store spans for the trace.
If the trace is complete in the legacy Trace Explorer page, then return to the Trace Explorer page and set the Scope element to a trace scope that lists all of the projects that you identified in the previous step. These are the projects that store the spans for the selected trace.

If there isn't a trace scope that includes the projects that you identified in the previous step, then create or modify an existing trace scope. For more information, see Create and manage trace scopes.
If the trace isn't complete in the legacy Trace Explorer page, then either spans weren't recorded or you don't have the Cloud Trace User role (roles/cloudtrace.user) on the projects that store the span data.

You don't have the required permissions to view trace data

You are viewing the Trace Explorer page and see the following notification:

You don't have the required permissions to view trace data for one or more projects listed in the trace scope.

To resolve this message, in the toolbar, do the following:

Expand the Scope element and identify the selected trace scope.
In the Refine scope flyout, select Manage scopes.
Locate the trace scope you identified in the first step, and then expand the details to view the list of Google Cloud projects.
For each Google Cloud project in the trace scope, ensure that you have the role of Cloud Trace User (roles/cloudtrace.user). If you don't have that role on a project, then ask an administrator or project owner to grant you that role.

Force creation of database for the Trace Explorer page

When the only trace data in your project is from Google Cloud services which are preconfigured for Cloud Trace, the database that the the Trace Explorer page reads for your trace and span data might not be created. However, you can force the creation of this database by sending a trace to your Google Cloud project by using the Cloud Trace API.

For example, you could do the following:

Go to the patchTraces documentation page.
In the Try this method pane, do the following:
1. In the projectId field, enter your PROJECT_ID.
2. Copy the JSON, which defines a trace with a single span, into your clipboard and then paste it into the Request body field.
  
  Before you copy the JSON, do the following:
  - Replace PROJECT_ID with the ID of your project.
  - Replace END_TIME with the current time, and replace START_TIME with a value earlier than the end time. If you are using Linux, then run date -Isec to get the current time in the correct format. For example, you might set these fields as follows:
```
"startTime": "2024-05-31T15:10:35.398448Z",
"endTime": "2024-05-31T15:10:35.574999047Z",
```
  - Update the traceId and spanId fields each time you execute the command.
```
"traces": [
  {
    "projectId": "PROJECT_ID",
    "traceId": "33fc0d8c45bb4e5cebb29f047931270d",
    "spans": [
      {
        "spanId": "17941747227541407973",
        "name": "/",
        "startTime": "START_TIME",
        "endTime": "END_TIME",
      }
    ]
  }
]
```
3. Press Execute.
  
  When the command completes successfully, the response body is empty. If you go to Trace Explorer, then you can view your trace. There might be a slight delay before the trace is available in the Google Cloud console.

Troubleshoot the legacy Trace Explorer page

This section describes how to troubleshoot the issues when you are using the legacy Trace Explorer page.

No data in the legacy interface

You are using the legacy Trace Explorer page and you can't view any traces in your Google Cloud project when you expect trace data to be present.

To resolve this failure, try the following steps:

Enable the Cloud Trace API.
Enable the API

Note: There might be org policies in place that have disabled the Cloud Trace API. These policies prevent you from enabling the API.
In the Google Cloud console, go to the APIs & Services page:

Go to APIs & Services

After you find the row labeled Cloud Trace API, try the following:
- If the column labeled Requests doesn't list any numeric information, then no trace data is being sent to your Google Cloud project.
  
  To resolve this case, check your applications and proxies to ensure they are configured to send the traces to the correct project.
- Select Cloud Trace API, select the Metrics tab, and then find the chart labeled Errors by API method:
  - If writes are failing, then grant the service account that is providing the authentication credentials the role of Cloud Trace Agent (roles/cloudtrace.agent). This role includes the permission cloudtrace.traces.patch, which lets applications write span data to a Google Cloud project.
    
    For more information, see Cloud Trace IAM roles.
  - If reads are failing, then ensure that your IAM role on the Google Cloud project includes the permissions in the role Cloud Trace User (roles/cloutrace.user). For a list of permissions for this role, see Cloud Trace IAM roles.

No trace data for deployed app in the legacy interface

You've deployed an app that sends data to your Google Cloud project by using the Cloud Trace API, yet trace data isn't collected.

Try the following:

If you aren't seeing any data in the legacy Trace Explorer page of the Google Cloud console, then follow the steps in the section titled No data in the legacy Trace interface.
If the app isn't deployed on Google Cloud or if it uses a service account to provide authentication credentials, then ensure that the service account has been granted the Cloud Trace Agent role (roles/cloudtrace.agent).

This role includes the permission cloudtrace.traces.patch, which lets applications write span data to a Google Cloud project.
If the app relies on OpenTelemetry, then do the following:
- For the root service, try updating your environment variables so that OpenTelemetry uses the traceidratio sampler with a sampling rate of 0.5:
```
export OTEL_TRACES_SAMPLER="traceidratio"
export OTEL_TRACES_SAMPLER_ARG="0.5"
```
- For all other services, leave the OTEL_TRACES_SAMPLER environment variable unset to use the default sampler, which is parentbased_always_on. The default setting means the sampling decision for a span is inherited from its parent span, if such a span exists. If a parent span doesn't exist, then the span is sampled.
Unless your application always samples every span, it isn't possible, in general, to force a request to be traced end-to-end because each component in an end-to-end request makes its own sampling decision. However, you can influence the decision by adding to the trace header a sampled flag, with this flag set to true. This setting is a hint to child components to sample the request. For more information about trace headers, see Protocols for context propagation.

Missing span ID message in trace

Your trace contains a "Missing span ID" message.

In distributed tracing systems, incomplete traces are expected. A trace is incomplete when a sampled span contains a reference to another span that hasn't been received. The unresolved reference can occur for the following reasons:

The referenced span wasn't sampled.
The referenced span was sampled but hasn't yet been received by Cloud Trace or the span was received but not stored.

When you are viewing an incomplete trace, Cloud Trace displays the message "Missing span ID" on the trace details pane.

If you are consistently seeing the message "Missing span ID", then try the following:

For components that you manage, ensure that they respect and propagate the header's sampled flag flag, when this field is present. This setting is a hint to child components to sample the request. For more information about trace headers, see Protocols for context propagation.

Google Cloud services typically respect this hint. However, they also limit the rate at which they write trace data.
If you are using Cloud Service Mesh, then ensure that you follow the guidance for propagating the trace context for those configurations. For Cloud Service Mesh guidance, see Trace context propagation.

No trace data after updating Go app to use OpenTelemetry

Your application relies on the client library to capture traces, and after updating your application to use OpenTelemetry, you no longer see Cloud Trace data.

Because some Cloud Client Libraries for Go are integrated with OpenCensus, you must use an OpenCensus Bridge. For more information about the problem solved by the bridge, see OpenCensus Bridge.

For information about the update of the Cloud Client Libraries for Go, see Issue #4237.