This page explains how to enable Stackdriver Logging for your Cloud Data Fusion instance and how to view pipeline logs.
Before you begin
Create a Cloud Data Fusion instance. It takes up to 30 minutes for the instance creation process to complete.
Enabling Stackdriver Logging
To use Stackdriver Logging with your Cloud Data Fusion pipeline, enable Stackdriver Logging when you create your Cloud Data Fusion instance. After you create your instance, you will not be able to enable Stackdriver Logging.
- In the Cloud Console open the Instances page.
- Click Create instance.
- Click Show advanced options.
Under Logging and monitoring, click Enable Stackdriver logging service.
Viewing logs in Stackdriver
Every Cloud Data Fusion pipeline run is assigned a unique
you deploy and run your pipeline, find its
RunID. Then, in
Stackdriver, use the
RunID to view your pipeline
Getting your pipeline RunID
- Go to the Pipeline Studio in the Cloud Data Fusion UI.
- Click Summary.
- In the Summary page, click the Table tab.
RunIdlink to copy.
Go to the Stackdriver Logging > Logs Viewer page in the Cloud Console:
Using the Audited resources drop-down menu arrow_drop_down, select Cloud Dataproc Cluster > cdap-<your-pipeline-name>-<your-runID>.
Use the filter drop-down menus arrow_drop_down
to filter your logs. You can filter by various log levels or by components
Click Download logs to download Stackdriver logs.