When you run a pipeline in Cloud Data Fusion, you can get logs for monitoring and debugging directly from the Pipeline Studio page after the pipeline run begins.
For more logging options, see Viewing audit logs and Viewing pipeline logs in Cloud Logging.
Before you begin
You need a Cloud Data Fusion pipeline. If you don't have one, see the Quickstart.
View a pipeline log
To view a pipeline log in Cloud Data Fusion:
Open the Pipeline Studio page where you edit and run your pipeline.
Click Logs to get a list of log messages.
Monitor a running pipeline
To monitor the progress of a pipeline, click Logs while the pipeline is running.
Filter the logs
To filter the pipeline log messages, open the Level drop-down list and select the desired type of message:
- Error
- Warn
- Info
- Debug
- Trace
Use advanced logs
When you view pipeline logs in Cloud Data Fusion, you get basic messages about your pipeline at various stages of its run. To get additional messages about the health of the Dataproc cluster and execution engine, click View Advanced Logs.
You can get advanced logs at any time during a pipeline preview or run.
To return to viewing basic logs, click Hide Advanced Logs.
View the latest log messages
Starting in Cloud Data Fusion 6.1.4, when you click Logs, you go directly to the latest log messages. In all versions, to view the latest messages, click Scroll to Latest Logs.
Download logs
You can download logs from Cloud Logging. If Cloud Logging is not enabled, after a pipeline run completes, you can download a log file directly from the Pipeline Studio page. The file includes advanced log messages and trace messages, which are useful for debugging.
Open the Pipeline List page and click on the desired pipeline.
The pipeline opens in the Pipeline Studio page.
Click Logs.
Click Download All.
Optional: To view a plain-text log in your browser, open the Download All drop-down list and select View Raw Logs.
What's next
- Learn how to view audit logs.
- Learn how to view pipeline logs in Cloud Logging.