Find and explore traces

The Trace explorer page lets you find, and examine, individual traces in detail. You can view and inspect all spans for a trace, view summary information for a request, and view detailed information for each span in the trace from this page. To restrict the traces being investigated, you add filters. For example, you can add a filter to display only traces whose latency exceeds 1 second.

The Trace explorer page stores and displays at most 1000 traces.

Find a trace

To view recent traces, do the following:

1. In the navigation panel of the Google Cloud console, select Trace, and then select Trace explorer:

   Go to Trace explorer

   If this is your first time using Trace, then it can take a few minutes for traces to appear. The following screenshot shows an example of the Trace explorer page:

   

   The scatter plot displays a dot for each request in the selected time interval:

   • The (x,y) coordinates for a request correspond to the time and latency of the request.

   • Error information is encoded in the color of a dot. Blue indicates success and red indicates failure. In the previous screenshot, most of the commands completed successfully.

   • The tooltip that activates when you hold the pointer over a dot, displays the date, time, URI, and latency:

     

2. Optional. Set the time interval. By default, the most recent hour of data is displayed. To view a custom time interval, do the following:

   1. Select a preset option that is longer than your desired custom time interval.
   2. Place your pointer on the Select a trace plot at one end of the time interval and drag your pointer horizontally to the other end of the interval.

   To restore the graph to the preset time interval, click Reset.

3. Optional. To view a custom range of latency values, placing your pointer at one end of the desired range, and drag your pointer vertically to the other end.

   To restore the graph to the preset time interval, click Reset.

4. Optional. To restrict the scatter plot to display only specific traces, click Add trace filter , select a filter option, and then select a filter value.

   For example, to show all traces that contain an HTTP GET command and whose latency is at least 2 seconds, add the filters Method: GET and MinLatency: 2000.

   The following filter option are always available:

   • RootSpan: Match the root span name.
   • SpanName: Match the span name.
   • Method: Display traces whose root spans are annotated with the /http/method label, where the value of the label matches the filter value.
   • Status: Display traces whose root spans are annotated with the /http/status_code label, where the value label matches the filter value. The filter value can contain specific status codes, such as 301, or ranges of status values, such as 3xx.
   • MinLatency: Display traces whose latency is at least as large as as the filter value. The filter value must be in milliseconds.
   • HasLabel: Display traces where the label specified by the filter value appears in at least one span.

   • Service: Display traces that contain at least one span whose service name matches the filter value.

     The service name for a span is extracted from the OpenTelemetry attribute service.name, when that attribute is set. If that attribute isn't set and if the service is running on App Engine, then the App Engine service name is displayed. Otherwise, the service isn't specified.

   • Version: (App Engine only) Display traces that contain at least one span whose application's version matches the filter value.

   You might see other filter options such as URL or HTTP labels. If you have a user-defined label that matches a predefined filter, then the user-defined label is prefixed with LABEL so that you can distinguish between the two. For example, if you create a Service label, then you see both Service and LABEL:Service in the filter menu.

   You can manually enter the filter value. That is, you don't have to select on the available values.

   Note: To perform an exact match, prefix the filter value with a plus sign (+). If you omit the plus sign, then a match is determined by performing a prefix test with the filter value. For more details on filtering options, see Trace filters.

   If you add multiple filters, then only traces that satisfy all filters are displayed.

Explore a trace

To explore a trace, click a dot in the scatter plot. When you click a dot in the scatter plot, the following changes occur to the Trace explorer page:

• The scatter plot is refreshed and the dot you selected is highlighted with a circle drawn around the dot. The dots that represent all other traces are dimmed.

• The Trace details pane displays the following:

  • The trace identifier. This globally unique identifier for the trace is a 128-bit integer represented as a 32-byte hexadecimal string. For details, see Resource: Trace.

  • A summary line that lists the start time, the duration, and the number of spans.

  • A Logs & events menu. The menu selection controls how logs and events are shown. By default, when a log or event exists, a circle is added to the trace span. Overlapping circles indicate that the span has multiple logs or events. To display each log or event as a row in the table, expand Logs & events and select Show expanded.

  • A table. The first row of the table is for the trace. A row is added to the table for each span in the trace.

    For spans, the table lists the span name and the service associated with the span. The service name is extracted from the OpenTelemetry attribute service.name, when that attribute is set. If that attribute isn't set and if the service is running on App Engine, then the App Engine service name is displayed. Otherwise, the service isn't specified.

    The Latency column provides a visual representation of the latency, status, and, when present, event annotations. A blue latency bar indicates a successful completion, a red latency bar indicates an error occurred. Each event annotation in the space is represented with a circle on the latency bar.

The following illustrates an example Trace details pane:

View span details

Note: To expand the Trace details pane to fill the Trace explorer page, go to the toolbar of the Trace details pane, and then click Expand trace details.

To view detailed information about the trace or about a specific span, in the Trace details pane, click the latency bar for the entry. When you select the latency bar, the Trace details pane is redrawn and displays a tabbed table that provides additional information about the entry.

If you select the latency bar for the first row in the table, the row with the name Trace ID, then the table contains two tabs: Summary and Logs. The Summary tab lists general information about the trace. For example, for HTTP commands, this tab displays the type of HTTP command, the service, and latency information for each span in the trace.

If you select the latency bar for other rows in the table, that is, a row for a span, then the table contains four tabs: Attributes, Logs & events, Stacktraces, Metadata & Links:

• To find the labels attached to a span, view the Attributes tab. For information about labels, see Trace labels. The following screenshot illustrates this tab:

  

  To locate a specific label or a group of labels, add a filter. For example, if you add the filter Key: g.co, then the table lists all labels where the label key contains g.co.

• To view information about related log entries and events, when those exist, view the Logs & events tab. For information about event annotations, see Annotating trace spans. The following screenshot illustrates this tab:

  

  When log data is available, to see details about a log entry, click Show more:

  

• To find information about the number of stacktraces that are available and, detailed information about a captured stacktrace, use the Stacktraces tab.

• To find general information about the span and a table of links to other spans, view the Metadata & Links tab. This information includes the following:

  • Span ID

    The span ID is a 64-bit integer other than 0. For details, see TraceSpan.

  • Parent span ID

  • Project ID

  • Start time and end time

  • Table that lists links to other spans

    Each row in table named Links lists a link between the current span and another span. The Attributes field lists the key-value pairs for the linked-to span. The Trace field links to the trace for the linked-to span. When this field contains Current trace, the linked-to span is in the same trace as the current span. Otherwise, the field contains a trace ID. For information about links, see the Links API reference page.

  The following screenshot illustrates this tab:

  

What's next

• Add event annotations to trace spans
• Create and view reports
• View traces across projects