Find and explore traces

To find and explore individual traces in detail, use the Trace Explorer page. This page displays traces for which your Google Cloud project stores at least one span. The data shown for each displayed trace and span includes summary information, such as the latency and the command type, and detailed information. The detailed information includes logs, events, and other information.

The Trace Explorer page displays at most 1000 traces.

Before you begin

To get the permissions that you need to view trace data by using the Google Cloud console, ask your administrator to grant you the Cloud Trace User (roles/cloudtrace.user) IAM role on your project. For more information about granting roles, see Manage access to projects, folders, and organizations.

You might also be able to get the required permissions through custom roles or other predefined roles.

For more information about roles, see Control access with Identity and Access Management.

Display recent traces

To display the most recent traces, do the following:

  1. In the Google Cloud console, go to the Trace explorer page:

    Go to Trace explorer

    You can also find this page by using the search bar.

    If this is your first time using Trace, then it can take a few minutes for traces to appear. The following screenshot shows an example of the Trace Explorer page:

    Cloud Trace recent traces pane.

    The scatter plot displays a dot for each request in the selected time interval:

    • The (x,y) coordinates for a request correspond to the time and latency of the request.

    • Error information is encoded in the color of a dot. Blue indicates success and red indicates failure. In the previous screenshot, most of the commands completed successfully.

    • The tooltip that activates when you hold the pointer over a dot, displays the date, time, URI, and latency:

      Showing a trace tooltip that displays latency information.

  2. Optional: To filter the traces and spans that are shown, do any of the following:

    • Filter by time. By default, the most recent hour of data is displayed. To view a custom time interval, do the following:

      1. Select a preset option that is longer than your custom time interval.
      2. Place your pointer on the Select a trace plot at one end of the time interval and drag your pointer horizontally to the other end of the interval.

      To restore the graph to the preset time interval, click Reset.

    • Filter by latency. By default, all traces in the selected time interval are displayed. To limit the display to those traces whose latency is within an interval, place your pointer at one end of the latency value, and drag your pointer vertically to the other end.

      To restore the graph to the preset time interval, click Reset.

    • Click Add trace filter, select a filter option, and then select or enter a filter value. If you add multiple filters, then only traces that satisfy all filters are displayed. For example, to show all traces that contain an HTTP GET command and whose latency is at least 2 seconds, add the filters Method: GET and MinLatency: 2000.

      The following filter option are always available:

      • RootSpan: Match the root span name.
      • SpanName: Match the span name.
      • Method: Display traces whose root spans are annotated with the /http/method label, where the value of the label matches the filter value.
      • Status: Display traces whose root spans are annotated with the /http/status_code label, where the value label matches the filter value. The filter value can contain specific status codes, such as 301, or ranges of status values, such as 3xx.
      • MinLatency: Display traces whose latency is at least as large as as the filter value. The filter value must be in milliseconds.
      • HasLabel: Display traces where the label specified by the filter value appears in at least one span.

      • Service: Display traces that contain at least one span whose service name matches the filter value.

        The service name for a span is extracted from the OpenTelemetry attribute service.name, when that attribute is set. If that attribute isn't set and if the service is running on App Engine, then the App Engine service name is displayed. Otherwise, the service isn't specified.

      • Version: (App Engine only) Display traces that contain at least one span whose application's version matches the filter value.

      You might see other filter options such as URL or HTTP labels. If you have a user-defined label that matches a predefined filter, then the user-defined label is prefixed with LABEL so that you can distinguish between the two. For example, if you create a Service label, then you see both Service and LABEL:Service in the filter menu.

Find a trace by ID

When you are troubleshooting an incident or failure, you might know the trace ID. To explore that trace, do the following:

  1. In the Google Cloud console, go to the Trace explorer page:

    Go to Trace explorer

    You can also find this page by using the search bar.

  2. Select a Google Cloud project from which to view the trace data.

    You must select the Google Cloud project that stores the trace data unless your project is part of an organization. When a project is part of an organization you can configure cross-project tracing, and that lets you view trace data from any project in the organization. For more information, see View traces across projects.

  3. Enter the ID in the Trace ID field.

    When you enter a valid ID, the Trace details pane is populated with information about the trace and its spans. You can use the options in that pane to explore to the trace.

Explore a trace

To explore a trace, click a dot in the scatter plot. When you click a dot in the scatter plot, the following changes occur to the Trace Explorer page:

  • The scatter plot is refreshed and the dot you selected is highlighted with a circle drawn around the dot. The dots that represent all other traces are dimmed.

  • The Trace details pane displays the following:

    • The trace identifier. This globally unique identifier for the trace is a 128-bit integer represented as a 32-byte hexadecimal string. For details, see Resource: Trace.

    • A summary line that lists the start time, the duration, and the number of spans.

    • A Logs & events menu. The menu selection controls how logs and events are shown. By default, when a log or event exists, a circle is added to the trace span. Overlapping circles indicate that the span has multiple logs or events. To display each log or event as a row in the table, expand Logs & events and select Show expanded.

    • A table. The first row of the table is for the trace. A row is added for each span in the trace.

      For spans, the table lists the span name and the service associated with the span. The service name is extracted from the OpenTelemetry attribute service.name, when that attribute is set. If that attribute isn't set and if the service is running on App Engine, then the App Engine service name is displayed. Otherwise, the service isn't specified.

      The Latency column provides a visual representation of the latency, status, and, when present, event annotations. A blue latency bar indicates a successful completion, a red latency bar indicates an error occurred. Each event annotation in the space is represented with a circle on the latency bar.

The following illustrates an example Trace details pane:

Examples of the Cloud Trace details pane.

Search spans within a trace

You can search the spans and attributes in a trace for keywords. For example, in the previous screenshot, some spans and an attribute key are highlighted. These fields are highlighted because they match the search term, which was net.sock.peer.port.

To search the spans in a trace, in the toolbar of the Trace details pane, enter the search term in the Find in Trace field, and then press Enter.

Trace then searches select fields of each span in the displayed trace. A match occurs when a searched field contains the search term, ignoring case. The following fields are searched:

  • Span name.
  • Service name.
  • Keys and values of attributes.

After the search is complete, the Trace details pane is refreshed and matching fields are highlighted.

For example, if you enter the term get, then a span with an attribute whose key is /http/method and whose value is GET is highlighted. Also, spans with names like CurrencyService/GetSupportedCurrencies are highlighted.

You can't search by using a regular expression, and you can't search logs, events, or metadata.

View span details

To view detailed information about the trace or about a specific span, in the Trace details pane, click the latency bar for the entry. When you select the latency bar, the Trace details pane is redrawn and displays a tabbed table that provides additional information about the entry.

If you select the latency bar for the first row in the table, the row with the name Trace ID, then the table contains two tabs: Summary and Logs. The Summary tab lists general information about the trace. For example, for HTTP commands, this tab displays the type of HTTP command, the service, and latency information for each span in the trace.

If you select the latency bar for other rows in the table, that is, a row for a span, then the table contains four tabs: Attributes, Logs & events, Stacktraces, Metadata & Links:

  • To find the labels attached to a span, view the Attributes tab. For information about labels, see Trace labels. The following screenshot illustrates this tab:

    Example of the Cloud Trace attributes table.

    To locate a specific label or a group of labels, add a filter. For example, if you add the filter Key: g.co, then the table lists all labels where the label key contains g.co.

  • To view information about related log entries and events, when those exist, view the Logs & events tab. For information about event annotations, see Annotating trace spans. The following screenshot illustrates this tab:

    Example of the Cloud Tracelogs and events tab.

    When log data is available, to see details about a log entry, click Show more:

    Example of an expanded log.

  • To find information about the number of stack traces that are available and, detailed information about a captured stacktrace, use the Stacktraces tab.

  • To find general information about the span and a table of links to other spans, view the Metadata & Links tab. This information includes the following:

    • Span ID

      The span ID is a 64-bit integer other than 0. For details, see TraceSpan.

    • Parent span ID

    • Project ID

    • Start time and end time

    • Table that lists links to other spans

      Each row in table named Links lists a link between the current span and another span. The Attributes field lists the key-value pairs for the linked-to span. The Trace field links to the trace for the linked-to span. When this field contains Current trace, the linked-to span is in the same trace as the current span. Otherwise, the field contains a trace ID. For information about links, see the Links API reference page.

    The following screenshot illustrates this tab:

    Example of the Cloud Trace metadata list and links table.

What's next