Viewing logs from the Pipeline Studio

When you run a pipeline in Cloud Data Fusion, you can get logs for monitoring and debugging directly from the Pipeline Studio page after the pipeline run begins.

For more logging options, see Viewing audit logs and Viewing pipeline logs in Cloud Logging.

Before you begin

You need a Cloud Data Fusion pipeline. If you don't have one, see the Quickstart.

Viewing a pipeline log

To view a pipeline log in Cloud Data Fusion:

  1. Open the Pipeline Studio page where you edit and run your pipeline.

  2. Click Logs to get a list of log messages.

Monitoring a running pipeline

To monitor the progress of a pipeline, click Logs while the pipeline is running.

Filtering the logs

To filter the pipeline log messages, open the Level drop-down list and select the desired type of message:

  • Error
  • Warn
  • Info
  • Debug
  • Trace

Using advanced logs

When you view pipeline logs in Cloud Data Fusion, you get basic messages about your pipeline at various stages of its run. To get additional messages about the health of the Dataproc cluster and execution engine, click View Advanced Logs.

You can get advanced logs at any time during a pipeline preview or run.

To return to viewing basic logs, click Hide Advanced Logs.

Viewing the latest log messages

Starting in Cloud Data Fusion 6.1.4, when you click Logs, you go directly to the latest log messages. In all versions, to view the latest messages, click Scroll to Latest Logs.

Downloading logs

You can download logs from Cloud Logging. If Cloud Logging is not enabled, after a pipeline run completes, you can download a log file directly from the Pipeline Studio page. The file includes advanced log messages and trace messages, which are useful for debugging.

  1. Open the Pipeline List page and click on the desired pipeline.

    The pipeline opens in the Pipeline Studio page.

  2. Click Logs.

  3. Click Download All.

  4. (Optional) To view a plain-text log in your browser, open the Download All drop-down list and select View Raw Logs.

What's next