View Cloud Logging for Dataform

This documents shows you how to view Cloud Logging logs for Dataform workflow invocations.

Each Dataform workflow invocation is logged using Cloud Logging. Logging is automatically enabled for Dataform workflow invocations, which can incur Cloud Logging billing charges. For more information, see Dataform pricing.

Permissions and roles

The Logs Viewer role (roles/logging.viewer) gives you read-only access to all features of Logging. For more information on the IAM permissions and roles that apply to Cloud Logging data, see the Cloud Logging access control guide.

View logs

To view logs, go to the Logs Explorer.

Go to the Logs Explorer

For information about filtering Cloud Logging logs in Logs Explorer, see View logs by using the Logs Explorer.

Dataform logs are indexed by Dataform repository.

Log format

Dataform sends workflow invocation log messages in the following format:

{
  "insertId": "14ip1tza5",
  "jsonPayload": {
    "terminalState": "SUCCEEDED",
    "workflowInvocationId": "1678383230-ac4ed48c-eb70-4555-b2b0-cda54d13edfa",
    "@type": "type.googleapis.com/google.cloud.dataform.logging.v1.WorkflowInvocationCompletionLogEntry"
  },
  "resource": {
    "type": "dataform.googleapis.com/Repository",
    "labels": {
      "repository_id": "repository-name-example",
      "resource_container": "PROJECT_ID",
      "location": "us-central1"
    }
  },
  "timestamp": "2023-03-09T17:34:06.965981805Z",
  "severity": "INFO",
  "logName": "projects/project-name-example/logs/dataform.googleapis.com%2Fworkflow_invocation_completion",
  "receiveTimestamp": "2023-03-09T17:34:07.362371980Z"
}

What is logged

Dataform log entries contain information that is useful for monitoring and debugging your workflow invocations.

Log entries contain the following types of information:

  • receiveTimestamp
  • release_config_id
  • repository_id
  • resource_container
  • workflow_invocation_id
  • workflow_config_id
  • severity: can be INFO, WARNING, or ERROR
  • terminalState: can be SUCCEEDED, CANCELED or FAILED
  • timestamp
  • @type

Configure metrics and alerts with Cloud Monitoring

You can use Cloud Monitoring to observe trends in your logs and to notify you when conditions you describe occur.

To provide Cloud Monitoring with data from your logs, Cloud Logging offers you the following:

  • Log-based metrics, which you can use as follows:
    • To create alerting policies that notify you of changes over time.
    • To create charts that display changes over time.
  • Log-based alerts, which notify you anytime a specific event appears in a log.

For more information about log-based metrics and log-based alerts, see Monitor your logs.

To learn how to configure log-based alerts for Dataform workflow invocations, see Configure alerts for failed workflow invocations.

What's next