View pipeline logs in Cloud Logging

This page explains how to enable Cloud Logging for your Cloud Data Fusion instance and how to view pipeline logs.

Before you begin

Create a Cloud Data Fusion instance. It takes up to 30 minutes for the instance creation process to complete.

Note: When using Cloud Data Fusion, you use both the Google Cloud console and the separate Cloud Data Fusion UI. In the Google Cloud console, you can create a Google Cloud project, create and delete Cloud Data Fusion instances, and view instance details. In the Cloud Data Fusion UI, you can use the various pages such as Pipeline Studio or Wrangler to utilize Cloud Data Fusion functionality.

Enable Cloud Logging

To use Cloud Logging with your Cloud Data Fusion pipeline, enable Cloud Logging when you create your Cloud Data Fusion instance.

1. In the Google Cloud console open the Instances page.
2. Click Create instance.
3. Click Show advanced options.

4. Under Logging and monitoring, click Enable Stackdriver logging service.

   

After you create your instance, you cannot enable Cloud Logging in the Google Cloud console. Instead, run this gcloud CLI command:

gcloud beta data-fusion instances update INSTANCE_NAME \
  --project=PROJECT_ID \
  --location=LOCATION \
  --enable_stackdriver_logging

View logs

Every Cloud Data Fusion pipeline run is assigned a unique RunID. After you deploy and run your pipeline, find its RunID. Then, in Logging, use the RunID to view your pipeline logs.

Get the pipeline's RunID

1. Go to your instance:

   1. In the Google Cloud console, go to the Cloud Data Fusion page.

   2. To open the instance in the Cloud Data Fusion web interface, click Instances, and then click View instance.

      Go to Instances

2. Click Summary > Table.

3. Click the RunId link to copy.

   

View the logs in Logs Explorer

1. Go to the Cloud Logging > Logs Explorer page in the Google Cloud console:

   Go Logs Explorer

2. In the filter menus, select Cloud Dataproc Cluster > cdap-PIPELINE_NAME-YOUR_RUNID.

   

Optional: Filter the logs

Use the filter menus to filter your logs. You can filter by various log severity levels or by components such as datafusion-pipeline-logs.

Optional: Download the logs

Click Download logs.

For more information, see the Cloud Logging documentation for downloading log entries.

What's next

• Learn how to view audit logs.
• Learn how to view pipeline logs from the Cloud Data Fusion Pipeline Studio.