Cloud Monitoring

Preview

This product or feature is covered by the Pre-GA Offerings Terms of the Google Cloud Platform Terms of Service. Pre-GA products and features may have limited support, and changes to pre-GA products and features may not be compatible with other pre-GA versions. For more information, see the launch stage descriptions.

BigQuery Data Transfer Service exports monitoring data to Cloud Monitoring.

You can use monitoring metrics for the following purposes:

• Evaluate the usage and performance a data transfer configuration.
• Troubleshoot problems.
• Monitor transfer run statuses.

To create custom dashboards, set up alerts, and query metrics with Monitoring, you can use the Google Cloud Console or the Monitoring API.

Viewing transfer data in Metrics Explorer

1. In the Cloud Console, go to the Monitoring page.

   Go to Monitoring.

   The first time that you access any Monitoring functionality for a Google Cloud project, the project is associated with a Workspace. If you've never used monitoring, then a Workspace is automatically created. Otherwise, a dialog pops up. From the dialog, select between creating a Workspace and adding your project to an existing Workspace.

2. In the navigation pane, click Metrics Explorer.

3. Select your project.

4. In the Find resource type and metric box, enter the following:

   • For Resource type, enter BigQuery DTS Config.

   • For Metric, select one of the metrics listed in Monitoring metrics for transfer configurations, for example, Completed run count.

     

5. Optional: Select aligner, reducer, and other parameters.

6. The metrics are displayed in the Metrics explorer window.

   

Defining Cloud Monitoring alerts

You can define Monitoring alerts for BigQuery Data Transfer Service metrics:

1. In the Cloud Console, go to the Monitoring page.

   Go to Monitoring.

   The first time that you access any Monitoring functionality for a Google Cloud project, the project is associated with a Workspace. If you've never used Monitoring, then a Workspace is automatically created. Otherwise, a dialog pops up. From the dialog, select between creating a Workspace and adding your project to an existing Workspace.

2. In the navigation pane, select Alerting > Create policy.

   For more information about alerting policies and concepts behind them, see Types of alerting policies.

3. Click Add Condition and select a condition type.

4. Select metrics and filters. For metrics, the resource type is BigQuery DTS Config.

5. Click Save Condition.

6. Enter policy name, and then click Save Policy.

For more information about alerting policies and concepts, see Introduction to alerting.

Defining Cloud Monitoring custom dashboards

You can create custom dashboards over BigQuery Data Transfer Service metrics:

1. Go to Monitoring in Google Cloud Console.

   The first time you access any Monitoring functionality for a Google Cloud project, the project is associated with a Workspace. If you've never used Monitoring, then a Workspace is automatically created. Otherwise, a dialog is displayed and you are asked to select between creating a Workspace and adding your project to an existing Workspace.

2. In the navigation pane, Select Dashboards > Create Dashboard.

3. Click on Add Chart.

4. Give the chart a title.

5. Select metrics and filters. For metrics, the resource type is BigQuery DTS Config.

6. Click Save.

For more information about how to manage your dashboard by using the Cloud Console, see Managing dashboards through the Cloud Console.

Metric reporting frequency and retention

Metrics for BigQuery Data Transfer Service runs are exported to Monitoring in batches, at 1-minute intervals. Monitoring data is retained for 6 weeks.

The dashboard provides data analysis in default intervals of 1h (1 hour), 6H (6 hours), 1D (1 day), 1W (1 week), and 6W (6 weeks). You can manually request analysis in any interval between 1M (1 minute) to 6W (6 weeks).

Monitoring metrics for transfer configurations

The following metrics for BigQuery Data Transfer Service configs are exported to Monitoring:

Metric	Description
Run latency distribution	Distribution of the execution time (in seconds) of each transfer run, per transfer configuration.
Active run count	Number of transfer runs that are running or pending, per transfer configuration.
Completed run count	Number of completed transfer runs in a time period, per transfer configuration.

Filtering dimensions for metrics

Metrics are aggregated for each BigQuery Data Transfer Service configuration. You can filter aggregated metrics by the following dimensions:

Property	Description
TRANSFER_STATE	Represents the current transfer state of the transfer run. This dimension can have one of the following values: unspecified pending running succeeded failed cancelled
ERROR_CODE	Represents the final error code of the transfer run. This dimension can have one of the following values: OK CANCELLED UNKNOWN INVALID_ARGUMENT DEADLINE_EXCEEDED NOT_FOUND ALREADY_EXISTS PERMISSION_DENIED UNAUTHENTICATED RESOURCE_EXHAUSTED FAILED_PRECONDITION ABORTED OUT_OF_RANGE UNIMPLEMENTED INTERNAL UNAVAILABLE DATA_LOSS
RUN_CAUSE	Represents how a transfer run was triggered. This dimension can have one of the following values: USER_REQUESTED AUTO_SCHEDULE

What's next

• Learn more about Monitoring.
• Read an overview of Cloud Audit Logs and Cloud Logging.