Viewing trace details

Cloud Trace shows the details for a trace in the Trace details pane of the Trace list page. This view shows a summary of details about the request, a graphical timeline that shows the root span for the request and subspans for any RPC calls, and a detailed view of latency data collected for the spans.

Viewing trace details

  1. In the Google Cloud Console, go to the Trace list page:

    Go to Trace List

  2. There are two ways to display detailed information about a trace:

    • Click its URI listed in table.
    • Click its dot in the graph.

Timeline

The first time you choose a trace to examine, two new panes are displayed on the Trace list page:

  • One of the new panes displays a graphical representation of latency data for the request using a waterfall graph. By default, the root span of the selected trace is highlighted.

  • The other new pane displays detailed information about the row highlighted in waterfall graph.

These two panes are refreshed with new data each time you select a trace to examine:

Cloud Trace details pane.

Each row in the waterfall graph corresponds to a span in the trace:

  • If the symbol is displayed, then Cloud Trace detected a span whose start time is earlier than the start time of the span's parent. Cloud Trace automatically compensates for this inconsistency when displaying the span; however, the span data isn't modified.

    The timestamp inconsistency can occur when a service relies on multiple clock sources or different language libraries.

  • If the symbol is displayed, then that indicates that the span contains an HTTP error.

  • The name of the RPC call in the format service_name.call_name. For example, datastore_v3.RunQuery.

    For App Engine standard environment, the internal name reported here might not match the name from a language-specific service API.

  • The time it took to complete the round-trip RPC call.

If you click Show events, then the waterfall graph is redrawn and event annotations are included as rows in the graph. For more information on annotations, see Annotating spans.

Span-details pane

Adjacent to the waterfall graph is the span-details pane. This pane displays detailed information about the row highlighted in the waterfall graph. The data displayed varies depending on the element that is highlighted.

If you highlight a span, then the details include the URI name for the span and the relative start time and the name of the RPC call:

Property Description
Relative start time The first entry is the relative start time for the span.

For root spans, this value is always @0 ms.

For subspans, this is the time when the subspan started relative to the request start. That is, this value displays how long it takes from the start of the overall request, for this RPC call to begin.
Name The next entry is the name of the RPC call. Names are formatted as service_name.call_name. For example, datastore_v3.RunQuery.

For App Engine standard environment, the internal name reported here might not match the name from a language-specific service API

When a root span is highlighted, the next entry is the Summary table. There is a row in the table for the root span, and one row for each RPC called by a subspan:

Summary Table
Column
Description
Name Name of the RPC call in format service_name.call_name. Displayed in the span label.
RPCs The number of times the procedure was called.
Total Duration The total time spent performing the RPCs.

The final entry in the pane is the Detail table. This table lists metadata about the span. The following properties are always displayed:

Details Table
Property
Description
Timestamp The time at which the application received the incoming request.
Log A link to a log entry if log data is available. If log data isn't available, this row is omitted.
Report A link to the most recent analysis report that includes this trace. If no reports include this trace, this row is omitted.
Trace ID
(Only available for the preview Trace list page)
To view this field, click the Details menu .

The globally unique identifier for the trace. This identifier is a 128-bit numeric form represented as a 32-byte hex string. For details, see Resource: Trace.

Span ID
(Only available for the preview Trace list page)
To view this field, click the Details menu .

The identifier for the span. This identifier is a 64-bit numeric integer other than 0. For details, see TraceSpan.

Labels A table of labels and values in the request. Labels are application specific. A few common labels, and their meaning, are listed in the following table:
http/host Hostname where the application is running.
http/response/size Number of bytes in the HTTP response body.
http/url Relative URL for the request.
gae/request_log_id Internal App Engine request ID.

Additional properties might be listed when your request type is HTTP or when your application is running on App Engine. The following table displays a partial list of some commonly observed properties:

Property Description
Traced time (HTTP only) The aggregate time it took for all RPC calls to be completed.
Untraced time (HTTP only) The time during which no RPC calls were made. That is, this measures the time spent locally in the application. Note that a longer than typical untraced time at the beginning of a request can often be caused by a new instance being created to handle the load.
HTTP method (HTTP only) The HTTP method for the request.
Service (App Engine only) The App Engine service that handled the request. For more information, go to App Engine.
Version (App Engine only) The version of the application that handled the request..

If you highlight an event, then the event details are displayed in the details pane. By default, events aren't displayed in the waterfall graph. To include them in the waterfall graph, click Show events.

Insights

Insights on the performance of the request are shown below the timeline, when insights are available.

For more information, go to Insights.

Annotating spans

You can annotate a trace by using Cloud Trace API v1 or Cloud Trace API v2. This section describes the different options available to you.

To associate a trace with a Cloud Logging LogEntry object, you use annotations. For more information on the integration between Cloud Trace and Cloud Logging, see Integrating with Cloud Logging.

Annotating with labels

You can add annotations to spans by creating a labels object and attaching it to the TraceSpan object when you use Cloud Trace API v1 patchTraces.

These annotations are displayed as labels when you view a traces' details. For more information about the labels, see Trace labels.

Annotating with attributes

You can annotate spans by creating an attributes object and attaching it to the Span object when you use Cloud Trace API v2 batchWrite.

These annotations are displayed as labels when you view a traces' details. For more detail, go to Viewing trace details on this page.

Annotating with time events

You can add annotations and message event to spans by creating a TimeEvents object and attaching it to the Span object when you use Cloud Trace API v2 batchWrite.

The TimeEvents object is an array of TimeEvent objects, each one contains a message event and an annotation.

To view the TimeEvents for a trace, go to the traces' waterfall graph and click Show events. If you select a TimeEvent, then its details are displayed in the details pane.

What's next