This page describes how to set up alerts for data pipelines in Cloud Data Fusion.
You can create alerts through the following services:
Alerts through Cloud Data Fusion: you set up these alerts when you design a batch pipeline in Cloud Data Fusion. After you deploy the pipeline, you can't add or edit an alert.
Log-based alerts through Cloud Monitoring: you set up these alerts in the Google Cloud console before and after you deploy the pipeline.
Before you begin
You need a Cloud Data Fusion instance. For more information, see Creating a Cloud Data Fusion instance.
Add an alert in Cloud Data Fusion
Open your instance.
In the Google Cloud console, go to the Cloud Data Fusion page.
To open the instance in the Cloud Data Fusion Studio, click Instances, and then click View instance.
Go to the Studio page where you're designing your pipeline.
Click Configure > Pipeline alerts > Add.
Select and configure the alert. For more information about configuring alerts, see Batch pipeline alerts.
After you configure the alert, click Validate to check for errors, and then click Confirm.
Create an alert in Monitoring
Enable Cloud Logging in your Cloud Data Fusion instance.
-
In the Google Cloud console, go to the Logs Explorer page.
This page lets you store, query, update, and delete data.
Enter the following query:
resource.type="cloud_dataproc_cluster" logName="projects/PROJECT_ID/logs/datafusion-pipeline-logs" resource.labels.project_id="PROJECT_ID" severity=ERROR jsonPayload.message=~"Pipeline '.*' failed."
Replace
PROJECT_ID
with your project's ID.Click Create alert and configure the alert.
An alert is only generated when the pipeline runs on a Dataproc cluster and fails. For Dataproc cluster creation failures, the alert is not generated.
What's next
- Refer to the CDAP OS Batch Pipeline Alerts reference pages.