Cloud Logging

Each BigQuery Data Transfer Service run is logged using Cloud Logging. Logging is automatically enabled for all data transfers.

Permissions and roles

The Logs Viewer role (roles/logging.viewer) gives you read-only access to all features of Logging. For more information on the IAM permissions and roles that apply to Cloud Logging data, see the Cloud Logging access control guide.

Viewing logs

To view logs, go to the Logs Explorer.

Go to the Logs Explorer

BigQuery Data Transfer Service logs are indexed first by the transfer configuration and then by the individual transfer run.

Viewing transfer run logs

To show only the log entries from a given transfer run_id, in the Query builder, add the following filter:

resource.type="bigquery_dts_config"
labels.run_id="transfer_run_id"

View run logs.

Viewing transfer configuration logs

To show log entries from a given transfer config_id, in the Query builder, add the following filter:

resource.type="bigquery_dts_config"
labels.run_id="transfer_config_id"

View config logs.

Viewing all logs

To see all BigQuery Data Transfer Service logs, do one of the following:

  • Under Log fields, for Resource type, select BigQuery DTS Config.

    View all logs.

  • In the Query builder, add the resource.type="bigquery_dts_config" filter:

    resource.type="bigquery_dts_config"
    

For more information about how to use the Log Explorer, see Using the Log Explorer.

Log format

BigQuery Data Transfer Service logs messages in the following format:

{
  "insertId": "0000000000",
  "jsonPayload": {
    "message": "DTS transfer run message."
  },
  "resource": {
    "type": "bigquery_dts_config",
    "labels": {
      "project_id": "my_project_id",
      "config_id": "transfer_config_id",
      "location": "us"
    }
  },
  "timestamp": "2020-11-25T04:45:48.545732221Z",
  "severity": "INFO",
  "labels": {
    "run_id": "transfer_run_id"
  },
  "logName": "projects/your_project_id/logs/bigquerydatatransfer.googleapis.com%2Ftransfer_config",
  "receiveTimestamp": "2020-11-25T04:45:48.960214929Z"
}

What is logged

BigQuery Data Transfer Service log entries contain information that is useful for monitoring and debugging your transfer runs. Log entries contain the following types of information:

  • timestamp
  • severity: can be INFO, WARNING or ERROR
  • message_text: holds a string that explains the current status of the transfer run

What's next