This page provides a conceptual overview of exporting trace data using Cloud Trace. You might want to export trace data for the following reasons:
- To store trace data for a period longer than the default retention period of 30 days.
To let you use BigQuery tools to analyze your trace data. For example, using BigQuery, you can identify span counts and quantiles. For information on the query used to generate the following table, see HipsterShop query.
How exports work
Exporting involves creating a sink for a Google Cloud project. A sink defines a BigQuery dataset as the destination.
You can create a sink by using the Cloud Trace API or by using the
gcloud command-line tool.
Sink properties and terminology
Sinks are defined for a Google Cloud project and have the following properties:
Name: A name for the sink. For example, a name might be:
[PROJECT_NUMBER]is the sink's Google Cloud project number and
my-sinkis the sink identifier.
Parent: The resource in which you create the sink. The parent must be a Google Cloud project:
[PROJECT_ID]can either be a Google Cloud project identifier or number.
Destination: A single place to send trace spans. Trace supports exporting traces to BigQuery. The destination can be sink's Google Cloud project or any other Google Cloud project that is in the same organization.
For example, a valid destination is:
[DESTINATION_PROJECT_NUMBER]is the Google Cloud project number of the destination, and
[DATASET_ID]is the BigQuery dataset identifier.
Writer Identity: A service account name. The export destination's owner must give this service account permissions to write to the export destination. When exporting traces, Trace adopts this identity for authorization. For increased security, new sinks get a unique service account:
[PROJECT_NUMBER]is your Google Cloud project number, in Hex, and
[GENERATED_ID]is a randomly generated value.
For information on using the writer identity, see destination permissions.
How sinks work
Every time a trace span arrives in a project, Trace exports a copy of the span.
Traces that Trace received before the sink was created cannot be exported.
To create or modify a sink, you must have one of the following Identity and Access Management roles:
- Trace Admin
- Trace User
- Project Owner
- Project Editor
For more information, see Access control.
To export traces to a destination, the sink's writer service account must be permitted to write to the destination. For more information about writer identities, see Sink properties on this page.
Quotas and limits
Cloud Trace utilizes the BigQuery streaming API to send trace spans to the destination. Cloud Trace batches API calls. Cloud Trace doesn't implement a retry or throttling mechanism. Trace spans might not be exported successfully if the amount of data exceeds the destination quotas.
For details on BigQuery quotas and limits, see Quotas and limits.
Exporting traces doesn't incur Cloud Trace charges. However, you might incur BigQuery charges. See BigQuery pricing for more information.
Estimating your costs
BigQuery charges for data ingestion and storage. To estimate your monthly BigQuery costs, do the following:
Estimate the total number of trace spans that are ingested in a month.
The Trace Overview window displays the number of chargeable spans ingested in the current month and in the previous month:
The number of spans listed on the overview pane doesn't include those spans that are not chargeable. For more information, see Pricing.
Estimate the streaming requirements based on the number of trace spans ingested.
Each span is written to a table row. Each row in BigQuery requires at least 1024 bytes. Therefore, a lower bound on your BigQuery streaming requirements is to assign 1024 bytes to each span. For example, if your Google Cloud project ingested 200 spans, then those spans require at least 20,400 bytes for the streaming insert.
Use the Pricing calculator to estimate your BigQuery costs due to storage, streaming inserts, and queries.
Viewing and managing your BigQuery usage
You can use Metrics Explorer to view your BigQuery usage. You can also create an alerting policy that notifies you if your BigQuery usage exceeds predefined limits. The following table contains the settings to create an alerting policy. You can use the settings in the target pane table when creating a chart or when using Metrics Explorer.
To create an alerting policy that triggers when the ingested BigQuery metrics exceed a user-defined level, do the following:
Steps to create an alerting policy.
To create an alerting policy, do the following:
- In the Google Cloud Console, go to Monitoring or use the following button:
Go to Monitoring
- In the Monitoring navigation pane, select notificationsAlerting and then select Create Policy.
- Click Add Condition, and then complete the Target and Configuration sections of the dialog by using the settings in the following tables. Click Add. To monitor another resource, repeat this step.
- To advance to the notifications section, click Next.
- (Optional) To add notifications to your alerting policy, click
Notification channels. In the dialog, select one or more notification
channels from the menu and then click OK.
If a notification channel that you want to add isn't listed, then click Manage notification channels. You are taken to the Notification channels page in a new browser tab. From this page, you can update the configured notification channels. After you have completed your updates, return to the original tab, click Refresh autorenew, and then select the notification channels to add to the alerting policy.
- To advance to the documentation section, click Next.
- Click Name and enter a name for the alerting policy.
- (Optional) Click Documentation and add any information that you want included in a notification message.
- Click Save.
||Metrics specific to usage include
||project_id: Your Google Cloud project ID.
dataset_id: Your dataset ID.
||dataset_id: Your dataset ID.|
||You determine the acceptable value.|
To configure a sink, see Exporting traces.