Each Dataform workflow invocation is logged using Cloud Logging. Logging is automatically enabled for Dataform workflow invocations, which can incur Cloud Logging billing charges. For more information, see Dataform pricing.
Permissions and roles
The Logs Viewer role (roles/logging.viewer
) gives you read-only access to all
features of Logging. For more information on the IAM permissions
and roles that apply to Cloud Logging data, see the Cloud Logging
access control guide.
View logs
To view logs, go to the Logs Explorer.
For information about filtering Cloud Logging logs in Logs Explorer, see View logs by using the Logs Explorer.
Dataform logs are indexed by Dataform repository.
Log format
Dataform sends workflow invocation log messages in the following format:
{
"insertId": "14ip1tza5",
"jsonPayload": {
"terminalState": "SUCCEEDED",
"workflowInvocationId": "1678383230-ac4ed48c-eb70-4555-b2b0-cda54d13edfa",
"@type": "type.googleapis.com/google.cloud.dataform.logging.v1.WorkflowInvocationCompletionLogEntry"
},
"resource": {
"type": "dataform.googleapis.com/Repository",
"labels": {
"repository_id": "repository-name-example",
"resource_container": "PROJECT_ID",
"location": "us-central1"
}
},
"timestamp": "2023-03-09T17:34:06.965981805Z",
"severity": "INFO",
"logName": "projects/project-name-example/logs/dataform.googleapis.com%2Fworkflow_invocation_completion",
"receiveTimestamp": "2023-03-09T17:34:07.362371980Z"
}
What is logged
Dataform log entries contain information that is useful for monitoring and debugging your workflow invocations.
Log entries contain the following types of information:
receiveTimestamp
release_config_id
repository_id
resource_container
workflow_invocation_id
workflow_config_id
severity
: can beINFO
,WARNING
, orERROR
terminalState
: can beSUCCEEDED
,CANCELED
orFAILED
timestamp
@type
Configure metrics and alerts with Cloud Monitoring
You can use Cloud Monitoring to observe trends in your logs and to notify you when conditions you describe occur.
To provide Cloud Monitoring with data from your logs, Cloud Logging offers you the following:
- Log-based metrics, which you can use as follows:
- To create alerting policies that notify you of changes over time.
- To create charts that display changes over time.
- Log-based alerts, which notify you anytime a specific event appears in a log.
For more information about log-based metrics and log-based alerts, see Monitor your logs.
To learn how to configure log-based alerts for Dataform workflow invocations, see Configure alerts for failed workflow invocations.
What's next
- Read an overview of Cloud Audit Logs and Cloud Monitoring.
- Learn more about Cloud Logging.
- Learn more about Cloud Logging pricing.
- Learn more about Dataform pricing.
- Learn more about Cloud Monitoring log-based metrics and log-based alerts.
- Learn how to configure log-based alerts for Dataform workflow invocations.