This page explains how to enable Cloud Logging for your Cloud Data Fusion instance and how to view pipeline logs.
Before you begin
Create a Cloud Data Fusion instance. It takes up to 30 minutes for the instance creation process to complete.
Enabling Cloud Logging
To use Cloud Logging with your Cloud Data Fusion pipeline, enable Cloud Logging when you create your Cloud Data Fusion instance.
- In the console open the Instances page.
- Click Create instance.
- Click Show advanced options.
Under Logging and monitoring, click Enable Stackdriver logging service.
After you create your instance, you cannot enable Cloud Logging in the console. Instead, run this gcloud CLI command:
gcloud beta data-fusion instances update INSTANCE_NAME \ --project=PROJECT_ID \ --location=LOCATION \ --enable_stackdriver_logging
Every Cloud Data Fusion pipeline run is assigned a unique
you deploy and run your pipeline, find its
RunID. Then, in
Logging, use the
RunID to view your pipeline
Getting your pipeline RunID
- Go to the Pipeline Studio in the Cloud Data Fusion UI.
- Click Summary.
- In the Summary page, click the Table tab.
RunIdlink to copy.
Go to the Cloud Logging > Logs Explorer page in the console:
In the filter menus, select Cloud Dataproc Cluster >
Use the filter menus to filter your logs. You can filter by various log
or by components such as
Click Download logs.
For more information, see the Cloud Logging documentation for downloading log entries.
- Learn how to view audit logs.
- Learn how to view pipeline logs from the Cloud Data Fusion Pipeline Studio.