Stay organized with collections
Save and categorize content based on your preferences.
This document describes how to access and view pipeline logs and service logs
for Cloud Data Fusion.
Starting with Cloud Data Fusion version 6.11, pipeline logs and
service logs are available in Cloud Logging.
About log types
Cloud Data Fusion generates several types of logs to help monitor and
troubleshoot data integration processes:
Pipeline logs
These logs provide detailed information about individual
pipeline runs, including preview runs. They cover all stages of a pipeline
run, such as pre-run setup, job execution, and post-run activities.
Service logs
These logs provide detailed information about Cloud Data Fusion
system services, such as AppFabric and AppFabric processor.
These logs provide detailed information about individual pipeline runs.
These logs are particularly useful for troubleshooting pipeline failures, as
they capture information even before the Dataproc cluster
finishes provisioning, which can help identify cluster issues or delays. For
more information, see
View advanced pipeline logs in Cloud Logging.
This displays the list of pipeline logs. You can use filters to refine
the results.
Filter pipeline logs
You can filter pipeline logs by run ID, instance ID, pipeline ID, location,
namespace, or custom labels.
Every Cloud Data Fusion pipeline run is assigned a unique RunID.
After you deploy and run your pipeline, you can find the RunID of your
pipeline and view the corresponding pipeline logs.
To filter pipeline logs by RunID, follow these steps:
Cloud Data Fusion 6.11.0 offers configurable
logging, with Cloud Logging enabled by default. While disabling
Cloud Logging is possible, it's strongly recommended to keep it enabled to
ensure you have access to critical pipeline and instance logs.
To disable Cloud Logging, run the following command:
[[["Easy to understand","easyToUnderstand","thumb-up"],["Solved my problem","solvedMyProblem","thumb-up"],["Other","otherUp","thumb-up"]],[["Hard to understand","hardToUnderstand","thumb-down"],["Incorrect information or sample code","incorrectInformationOrSampleCode","thumb-down"],["Missing the information/samples I need","missingTheInformationSamplesINeed","thumb-down"],["Other","otherDown","thumb-down"]],["Last updated 2025-09-04 UTC."],[[["\u003cp\u003eCloud Data Fusion version 6.11 and later versions utilize Cloud Logging for pipeline and service logs, offering detailed insights into data integration processes.\u003c/p\u003e\n"],["\u003cp\u003eThere are different types of logs available, including pipeline logs, service logs, and audit logs, each providing specific information about pipeline runs, system services, and administrative actions.\u003c/p\u003e\n"],["\u003cp\u003ePipeline and service logs can be viewed using the Cloud Data Fusion Logging dashboard or directly within the Logs Explorer in the Google Cloud console.\u003c/p\u003e\n"],["\u003cp\u003ePipeline logs can be filtered by various parameters like run ID, instance ID, and location to help narrow down the data to the exact process needing to be analyzed.\u003c/p\u003e\n"],["\u003cp\u003eService logs can be viewed by entering specific queries in the Logs Explorer based on the service, such as Appfabric or Dataset executor.\u003c/p\u003e\n"]]],[],null,["# View Cloud Data Fusion logs\n\nThis document describes how to access and view pipeline logs and service logs\nfor Cloud Data Fusion.\n\nStarting with Cloud Data Fusion version 6.11, pipeline logs and\nservice logs are available in Cloud Logging.\n\nAbout log types\n---------------\n\nCloud Data Fusion generates several types of logs to help monitor and\ntroubleshoot data integration processes:\n\n\u003cbr /\u003e\n\nPipeline logs These logs provide detailed information about individual pipeline runs, including preview runs. They cover all stages of a pipeline run, such as pre-run setup, job execution, and post-run activities.\n\n\u003cbr /\u003e\n\n\u003cbr /\u003e\n\nService logs These logs provide detailed information about Cloud Data Fusion system services, such as AppFabric and AppFabric processor.\n\n\u003cbr /\u003e\n\n\u003cbr /\u003e\n\nAudit logs Cloud Data Fusion generates audit logs, such as [Admin Activity audit logs](/logging/docs/audit#admin-activity). For more information, see [Cloud Data Fusion audit logging](/data-fusion/docs/how-to/audit-logging).\n\n\u003cbr /\u003e\n\n\u003cbr /\u003e\n\nAdvanced pipeline logs These logs provide detailed information about individual pipeline runs. These logs are particularly useful for troubleshooting pipeline failures, as they capture information even before the Dataproc cluster finishes provisioning, which can help identify cluster issues or delays. For more information, see [View advanced pipeline logs in Cloud Logging](/data-fusion/docs/how-to/viewing-stackdriver-logs).\n\n\u003cbr /\u003e\n\nPricing\n-------\n\nCloud Logging and Cloud Monitoring usage incurs charges. For more\ninformation, see [Google Cloud Observability pricing](/stackdriver/pricing).\n\nOptional: Import the Cloud Data Fusion Logging dashboard\n--------------------------------------------------------\n\nTo view pipeline logs and service logs using the\nCloud Data Fusion Logging dashboard, import the dashboard:\n\n1. In the Google Cloud console, go to the Cloud Monitoring **Dashboards** page.\n\n [Go to Dashboards](https://console.cloud.google.com/monitoring/dashboards)\n2. Click **View dashboard templates**.\n\n3. Search for **Cloud Data Fusion Logging** and select the dashboard.\n\n4. Click playlist_add\n **Add Cloud Data Fusion Logging dashboard to your list**.\n\nView pipeline logs\n------------------\n\nYou can view pipeline logs using the Cloud Data Fusion Logging\ndashboard or directly in the Logs Explorer.\n\n### View pipeline logs using dashboard\n\n1. If you haven't done already,\n [import the **Cloud Data Fusion Logging** dashboard](#import-dashboard).\n\n2. In the **My dashboards** section, click **Cloud Data Fusion Logging**.\n\n3. In the **Pipeline logs** section, view the list of pipeline logs. You can\n filter the logs by severity, field names, and values.\n\n To refine your search using queries, use Logs Explorer.\n\n### View pipeline logs in Logs Explorer\n\n1. In the Google Cloud console, go to the **Logs Explorer** page.\n\n [Go to Logs Explorer](https://console.cloud.google.com/logs/query)\n2. Enter the following query:\n\n resource.type=\"datafusion.googleapis.com/PipelineV2\"\n\n This displays the list of pipeline logs. You can use filters to refine\n the results.\n\n### Filter pipeline logs\n\nYou can filter pipeline logs by run ID, instance ID, pipeline ID, location,\nnamespace, or custom labels.\n\nEvery Cloud Data Fusion pipeline run is assigned a unique `RunID`.\nAfter you deploy and run your pipeline, you can find the `RunID` of your\npipeline and view the corresponding pipeline logs.\n\nTo filter pipeline logs by `RunID`, follow these steps:\n\n1. [Get the pipeline's RunID](/data-fusion/docs/how-to/viewing-stackdriver-logs#get_the_pipelines_runid).\n\n2. In the Google Cloud console, go to the **Logs Explorer** page.\n\n3. Enter the following query:\n\n resource.type=\"datafusion.googleapis.com/PipelineV2\" resource.labels.run_id=\u003cvar translate=\"no\"\u003eRUN_ID\u003c/var\u003e\n\nView service logs\n-----------------\n\nYou can view service logs using the Cloud Data Fusion Logging\ndashboard or in the Logs Explorer.\n\n### View service logs using dashboard\n\n1. If you haven't done already,\n [import the **Cloud Data Fusion Logging** dashboard](#import-dashboard).\n\n2. In the **My dashboards** section, click **Cloud Data Fusion Logging**.\n\n3. In the **Service logs** section, view the list of service logs. You can\n filter the logs by severity, field names, and values.\n\n To refine your search using queries, use Logs Explorer.\n\n### View service logs in Logs Explorer\n\n1. In the Google Cloud console, go to the **Logs Explorer** page.\n\n [Go to Logs Explorer](https://console.cloud.google.com/logs/query)\n2. Find the service logs by entering the specific query for that service.\n\n### Configurable logging in Cloud Data Fusion\n\nCloud Data Fusion 6.11.0 offers configurable\nlogging, with Cloud Logging enabled by default. While disabling\nCloud Logging is possible, it's strongly recommended to keep it enabled to\nensure you have access to critical pipeline and instance logs.\n\nTo disable Cloud Logging, run the following command: \n\n echo '{ \"loggingConfig\": {\"instance_cloud_logging_disabled\": \"true\"}}' | curl -X POST \\\n -H \"Authorization: Bearer $(gcloud auth print-access-token)\" \\\n -H \"Content-Type: application/json\" \\\n --data @- \\\n \"https://datafusion.googleapis.com/v1/projects/\u003cvar translate=\"no\"\u003ePROJECT_ID\u003c/var\u003e/locations/\u003cvar translate=\"no\"\u003eLOCATION\u003c/var\u003e/instances?instanceId=$\u003cvar translate=\"no\"\u003eINSTANCE_ID\u003c/var\u003e?updateMask=logging_config\"\n\nReplace the following:\n\n- \u003cvar translate=\"no\"\u003ePROJECT_ID\u003c/var\u003e: the Google Cloud project ID\n- \u003cvar translate=\"no\"\u003eLOCATION\u003c/var\u003e: the [location](/data-fusion/pricing#supported_regions) of your instance\n- \u003cvar translate=\"no\"\u003eINSTANCE_ID\u003c/var\u003e: the ID of your Cloud Data Fusion instance\n\nWhat's next\n-----------\n\n- Learn more about [Cloud Data Fusion audit logging](/data-fusion/docs/how-to/audit-logging).\n- Learn how to [view advanced pipeline logs](/data-fusion/docs/how-to/viewing-stackdriver-logs)."]]
