Viewing trace details

If you select a trace, then Cloud Trace shows the details for a trace in the Trace list page. This view shows a summary of details about the request, a graphical timeline that shows the root span for the request and subspans for any RPC calls, and a detailed view of latency data collected for the spans.

Viewing trace details

  1. In the Google Cloud Console, go to the Trace list page:

    Go to Trace List

  2. There are multiple ways to display detailed information about a trace:

    • Click its URI listed in table.
    • Click its dot in the graph.
    • In the Selected trace details view, enter the trace ID in hex format. If you enter a trace ID, you might see a Selected trace may not match current filters or Selected trace is outside the current time range message. These informational messages indicate that the displayed trace doesn't match your filters or the time range settings.

Selected trace details pane

If you choose a trace to examine, then details about the selected trace are shown:

  • The Selected trace details text box displays the identifier of that trace.

  • A pane displays a graphical representation of latency data for the request by using a waterfall graph. By default, the root span of the selected trace is highlighted.

  • A pane displays, in a series of tables, detailed information about the span highlighted in waterfall graph.

These fields are refreshed with new data each time you select a trace to examine. The following screenshot illustrates these panes:

Cloud Trace details pane.

Waterfall graph

Each row in the waterfall graph corresponds to a span in the trace:

  • If the symbol is displayed, then Cloud Trace detected a span whose start time is earlier than the start time of the span's parent. Cloud Trace automatically compensates for this inconsistency when displaying the span; however, the span data isn't modified.

    The timestamp inconsistency can occur when a service relies on multiple clock sources or different language libraries.

  • If the symbol is displayed, then that indicates that the span contains an HTTP error.

  • The name of the RPC call in the format service_name.call_name. For example, datastore_v3.RunQuery.

    For App Engine standard environment, the internal name reported here might not match the name from a language-specific service API.

  • The time it took to complete the round-trip RPC call.

Show events checkbox

If you check Show events, then the waterfall graph is redrawn and event annotations are included as rows in the graph. For more information on annotations, see Annotating spans.

Click-to-copy

You can copy the URL of the detail view for a specific trace into the clipboard by clicking Copy .

Tables with span details

These tables contain detailed information about the row currently highlighted in the waterfall graph. Each row in the waterfall graph corresponds to a trace span.

If you highlight a row, then the details for that span include its URI name and the relative start time and the name of the RPC call.

The data displayed in the tables varies depending on the element that is highlighted. However, the data listed in the following table is always displayed:

Property Description
Relative start time The first entry is the relative start time for the span.

For root spans, this value is always @0 ms.

For subspans, this is the time when the subspan started relative to the request start. That is, this value displays how long it takes from the start of the overall request, for this RPC call to begin.
Name The next entry is the name of the RPC call. Names are formatted as service_name.call_name. For example, datastore_v3.RunQuery.

For App Engine standard environment, the internal name reported here might not match the name from a language-specific service API

If a root span is highlighted and the request is an HTTP request, then a table titled Summary is shown. There is a row in the table for the root span, and one row for each RPC:

Summary Table
Column
Description
Name Name of the RPC call in format service_name.call_name. Displayed in the span label.
RPCs The number of times the procedure was called.
Total Duration The total time spent performing the RPCs.

The table titled Detail lists metadata about the span. The following properties are always displayed:

Details Table
Property
Description
Timestamp The time at which the application received the incoming request.
Log A link to a log entry if log data is available. If log data isn't available, this row is omitted.
Report A link to the most recent analysis report that includes this trace. If no reports include this trace, this row is omitted.
Trace ID To view this field, click the Details menu .

The globally unique identifier for the trace. This identifier is a 128-bit numeric form represented as a 32-byte hex string. For details, see Resource: Trace.

Span ID To view this field, click the Details menu .

The identifier for the span. This identifier is a 64-bit numeric integer other than 0. For details, see TraceSpan.

GKE Container

When this table is displayed, the span contains canonical labels for a GKE container. Each row in the table is for a specific label and the value is a link that you can use to view information about the resource in more detail. For detailed information about all labels, see Trace labels.

The labels and their meaning are listed in the following table:

Label Value
Project ID The Google Cloud project hosting the GKE cluster. Click the project name to go to the GKE dashboard in the Google Cloud Console.
Cluster Name Identifies the GKE cluster. To go to the Clusters configuration page, click this value.
Namespace Identifies the namespace. To go to the GKE Workloads page, click this value.
Pod Name Identifies the GKE pod. To go to the Pods details dashboard, click this value.
Container Name Identifies the GKE container. To go to the details page for the container, click this value.
Labels A table of labels and values in the request. Labels are application specific. For detailed information about all labels, see Trace labels.

A few common labels, and their meaning, are listed in the following table:

Label Value
http/host Hostname where the application is running.
http/response/size Number of bytes in the HTTP response body.
http/url Relative URL for the request.
gae/request_log_id Internal App Engine request ID.

Additional properties might be listed when your request type is HTTP or when your application is running on App Engine. The following table displays a partial list of some commonly observed properties:

Property Description
Traced time (HTTP only) The aggregate time it took for all RPC calls to be completed.
Untraced time (HTTP only) The time during which no RPC calls were made. That is, this measures the time spent locally in the application. Note that a longer than typical untraced time at the beginning of a request can often be caused by a new instance being created to handle the load.
HTTP method (HTTP only) The HTTP method for the request.
Service (App Engine only) The App Engine service that handled the request. For more information, go to App Engine.
Version (App Engine only) The version of the application that handled the request.

If you highlight an event, then the event details are displayed in the details pane. By default, events aren't displayed in the waterfall graph. To include them in the waterfall graph, click Show events.

Insights

Insights on the performance of the request are shown below the timeline, when insights are available.

For more information, see Insights.

Annotating spans

You can annotate a trace by using Cloud Trace API v1 or Cloud Trace API v2. This section describes the different options available to you.

To associate a trace with a Cloud Logging LogEntry object, you use annotations. For more information on the integration between Cloud Trace and Cloud Logging, see Integrating with Cloud Logging.

Annotating with labels

You can add annotations to spans by creating a labels object and attaching it to the TraceSpan object when you use Cloud Trace API v1 patchTraces.

These annotations are displayed as labels when you view a traces' details. For more information about the labels, see Trace labels.

Annotating with attributes

You can annotate spans by creating an attributes object and attaching it to the Span object when you use Cloud Trace API v2 batchWrite.

These annotations are displayed as labels when you view a traces' details. For more detail, go to Viewing trace details on this page.

Annotating with time events

You can add annotations and message event to spans by creating a TimeEvents object and attaching it to the Span object when you use Cloud Trace API v2 batchWrite.

The TimeEvents object is an array of TimeEvent objects, each one contains a message event and an annotation.

To view the TimeEvents for a trace, go to the traces' waterfall graph and click Show events. If you select a TimeEvent, then its details are displayed in the details pane.

What's next