Viewing pipeline logs in Cloud Logging

This page explains how to enable Cloud Logging for your Cloud Data Fusion instance and how to view pipeline logs.

Before you begin

Create a Cloud Data Fusion instance. It takes up to 30 minutes for the instance creation process to complete.

Enabling Cloud Logging

To use Cloud Logging with your Cloud Data Fusion pipeline, enable Cloud Logging when you create your Cloud Data Fusion instance. After you create your instance, you can only enable (or disable) Cloud Logging through an update operation using the gcloud SDK or the Cloud Data Fusion API.

  1. In the Cloud Console open the Instances page.
  2. Click Create instance.
  3. Click Show advanced options.
  4. Under Logging and monitoring, click Enable Stackdriver logging service.

    Enable Cloud Logging.

Viewing logs

Every Cloud Data Fusion pipeline run is assigned a unique RunID. After you deploy and run your pipeline, find its RunID. Then, in Logging, use the RunID to view your pipeline logs.

Getting your pipeline RunID

  1. Go to the Pipeline Studio in the Cloud Data Fusion UI.
  2. Click Summary.
  3. In the Summary page, click the Table tab.
  4. Click the RunId link to copy.

    Copy RunID.

Viewing logs

  1. Go to the Cloud Logging > Logs Viewer page in the Cloud Console:

    Go to the Logs Viewer page

  2. Using the Audited resources drop-down menu , select Cloud Dataproc Cluster > cdap-PIPELINE_NAME-YOUR_RUNID.

    Go to Cloud Logging.

Filtering logs

Use the filter drop-down menus to filter your logs. You can filter by various log levels or by components such as datafusion-pipeline-logs.

Downloading logs

Click Download logs.

For more information, see the Cloud Logging documentation for downloading log entries.

What's next