Mit Sammlungen den Überblick behalten
Sie können Inhalte basierend auf Ihren Einstellungen speichern und kategorisieren.
In diesem Dokument wird beschrieben, wie Sie auf Pipeline- und Dienstlogs für Cloud Data Fusion zugreifen und diese ansehen.
Ab Cloud Data Fusion-Version 6.11 sind Pipeline-Logs und Dienstlogs in Cloud Logging verfügbar.
Logtypen
Cloud Data Fusion generiert verschiedene Arten von Logs, mit denen Sie Datenintegrationsprozesse überwachen und Fehler beheben können:
Pipeline-Logs
Diese Logs enthalten detaillierte Informationen zu einzelnen Pipeline-Ausführungen, einschließlich Vorschauausführungen. Sie decken alle Phasen einer Pipelineausführung ab, z. B. die Einrichtung vor der Ausführung, die Jobausführung und die Aktivitäten nach der Ausführung.
Dienstprotokolle
Diese Logs enthalten detaillierte Informationen zu Cloud Data Fusion-Systemdiensten wie AppFabric und AppFabric-Prozessor.
Diese Logs enthalten detaillierte Informationen zu einzelnen Pipeline-Ausführungen.
Diese Logs sind besonders nützlich für die Fehlerbehebung bei Pipelinefehlern, da sie Informationen erfassen, noch bevor die Bereitstellung des Dataproc-Clusters abgeschlossen ist. So können Clusterprobleme oder Verzögerungen erkannt werden. Weitere Informationen finden Sie unter Erweiterte Pipeline-Logs in Cloud Logging ansehen.
Preise
Für die Nutzung von Cloud Logging und Cloud Monitoring fallen Gebühren an. Weitere Informationen finden Sie unter Google Cloud Observability-Preise.
Optional: Cloud Data Fusion-Logging-Dashboard importieren
Wenn Sie Pipeline-Logs und Dienstlogs über das Cloud Data Fusion-Logging-Dashboard aufrufen möchten, importieren Sie das Dashboard:
Rufen Sie in der Google Cloud Console die Cloud Monitoring-Seite Dashboards auf.
Dadurch wird die Liste der Pipeline-Logs angezeigt. Mit Filtern können Sie die Ergebnisse eingrenzen.
Pipeline-Logs filtern
Sie können Pipeline-Logs nach Ausführungs-ID, Instanz-ID, Pipeline-ID, Standort, Namespace oder benutzerdefinierten Labels filtern.
Jeder Cloud Data Fusion-Pipelineausführung wird eine eindeutige RunID zugewiesen.
Nachdem Sie Ihre Pipeline bereitgestellt und ausgeführt haben, können Sie das RunID Ihrer Pipeline aufrufen und die entsprechenden Pipeline-Logs ansehen.
Cloud Data Fusion 6.11.0 bietet konfigurierbares Logging, wobei Cloud Logging standardmäßig aktiviert ist. Cloud Logging kann zwar deaktiviert werden, es wird jedoch dringend empfohlen, es aktiviert zu lassen, damit Sie auf wichtige Pipeline- und Instanzlogs zugreifen können.
Führen Sie den folgenden Befehl aus, um Cloud Logging zu deaktivieren:
[[["Leicht verständlich","easyToUnderstand","thumb-up"],["Mein Problem wurde gelöst","solvedMyProblem","thumb-up"],["Sonstiges","otherUp","thumb-up"]],[["Schwer verständlich","hardToUnderstand","thumb-down"],["Informationen oder Beispielcode falsch","incorrectInformationOrSampleCode","thumb-down"],["Benötigte Informationen/Beispiele nicht gefunden","missingTheInformationSamplesINeed","thumb-down"],["Problem mit der Übersetzung","translationIssue","thumb-down"],["Sonstiges","otherDown","thumb-down"]],["Zuletzt aktualisiert: 2025-09-04 (UTC)."],[[["\u003cp\u003eCloud Data Fusion version 6.11 and later versions utilize Cloud Logging for pipeline and service logs, offering detailed insights into data integration processes.\u003c/p\u003e\n"],["\u003cp\u003eThere are different types of logs available, including pipeline logs, service logs, and audit logs, each providing specific information about pipeline runs, system services, and administrative actions.\u003c/p\u003e\n"],["\u003cp\u003ePipeline and service logs can be viewed using the Cloud Data Fusion Logging dashboard or directly within the Logs Explorer in the Google Cloud console.\u003c/p\u003e\n"],["\u003cp\u003ePipeline logs can be filtered by various parameters like run ID, instance ID, and location to help narrow down the data to the exact process needing to be analyzed.\u003c/p\u003e\n"],["\u003cp\u003eService logs can be viewed by entering specific queries in the Logs Explorer based on the service, such as Appfabric or Dataset executor.\u003c/p\u003e\n"]]],[],null,["# View Cloud Data Fusion logs\n\nThis document describes how to access and view pipeline logs and service logs\nfor Cloud Data Fusion.\n\nStarting with Cloud Data Fusion version 6.11, pipeline logs and\nservice logs are available in Cloud Logging.\n\nAbout log types\n---------------\n\nCloud Data Fusion generates several types of logs to help monitor and\ntroubleshoot data integration processes:\n\n\u003cbr /\u003e\n\nPipeline logs These logs provide detailed information about individual pipeline runs, including preview runs. They cover all stages of a pipeline run, such as pre-run setup, job execution, and post-run activities.\n\n\u003cbr /\u003e\n\n\u003cbr /\u003e\n\nService logs These logs provide detailed information about Cloud Data Fusion system services, such as AppFabric and AppFabric processor.\n\n\u003cbr /\u003e\n\n\u003cbr /\u003e\n\nAudit logs Cloud Data Fusion generates audit logs, such as [Admin Activity audit logs](/logging/docs/audit#admin-activity). For more information, see [Cloud Data Fusion audit logging](/data-fusion/docs/how-to/audit-logging).\n\n\u003cbr /\u003e\n\n\u003cbr /\u003e\n\nAdvanced pipeline logs These logs provide detailed information about individual pipeline runs. These logs are particularly useful for troubleshooting pipeline failures, as they capture information even before the Dataproc cluster finishes provisioning, which can help identify cluster issues or delays. For more information, see [View advanced pipeline logs in Cloud Logging](/data-fusion/docs/how-to/viewing-stackdriver-logs).\n\n\u003cbr /\u003e\n\nPricing\n-------\n\nCloud Logging and Cloud Monitoring usage incurs charges. For more\ninformation, see [Google Cloud Observability pricing](/stackdriver/pricing).\n\nOptional: Import the Cloud Data Fusion Logging dashboard\n--------------------------------------------------------\n\nTo view pipeline logs and service logs using the\nCloud Data Fusion Logging dashboard, import the dashboard:\n\n1. In the Google Cloud console, go to the Cloud Monitoring **Dashboards** page.\n\n [Go to Dashboards](https://console.cloud.google.com/monitoring/dashboards)\n2. Click **View dashboard templates**.\n\n3. Search for **Cloud Data Fusion Logging** and select the dashboard.\n\n4. Click playlist_add\n **Add Cloud Data Fusion Logging dashboard to your list**.\n\nView pipeline logs\n------------------\n\nYou can view pipeline logs using the Cloud Data Fusion Logging\ndashboard or directly in the Logs Explorer.\n\n### View pipeline logs using dashboard\n\n1. If you haven't done already,\n [import the **Cloud Data Fusion Logging** dashboard](#import-dashboard).\n\n2. In the **My dashboards** section, click **Cloud Data Fusion Logging**.\n\n3. In the **Pipeline logs** section, view the list of pipeline logs. You can\n filter the logs by severity, field names, and values.\n\n To refine your search using queries, use Logs Explorer.\n\n### View pipeline logs in Logs Explorer\n\n1. In the Google Cloud console, go to the **Logs Explorer** page.\n\n [Go to Logs Explorer](https://console.cloud.google.com/logs/query)\n2. Enter the following query:\n\n resource.type=\"datafusion.googleapis.com/PipelineV2\"\n\n This displays the list of pipeline logs. You can use filters to refine\n the results.\n\n### Filter pipeline logs\n\nYou can filter pipeline logs by run ID, instance ID, pipeline ID, location,\nnamespace, or custom labels.\n\nEvery Cloud Data Fusion pipeline run is assigned a unique `RunID`.\nAfter you deploy and run your pipeline, you can find the `RunID` of your\npipeline and view the corresponding pipeline logs.\n\nTo filter pipeline logs by `RunID`, follow these steps:\n\n1. [Get the pipeline's RunID](/data-fusion/docs/how-to/viewing-stackdriver-logs#get_the_pipelines_runid).\n\n2. In the Google Cloud console, go to the **Logs Explorer** page.\n\n3. Enter the following query:\n\n resource.type=\"datafusion.googleapis.com/PipelineV2\" resource.labels.run_id=\u003cvar translate=\"no\"\u003eRUN_ID\u003c/var\u003e\n\nView service logs\n-----------------\n\nYou can view service logs using the Cloud Data Fusion Logging\ndashboard or in the Logs Explorer.\n\n### View service logs using dashboard\n\n1. If you haven't done already,\n [import the **Cloud Data Fusion Logging** dashboard](#import-dashboard).\n\n2. In the **My dashboards** section, click **Cloud Data Fusion Logging**.\n\n3. In the **Service logs** section, view the list of service logs. You can\n filter the logs by severity, field names, and values.\n\n To refine your search using queries, use Logs Explorer.\n\n### View service logs in Logs Explorer\n\n1. In the Google Cloud console, go to the **Logs Explorer** page.\n\n [Go to Logs Explorer](https://console.cloud.google.com/logs/query)\n2. Find the service logs by entering the specific query for that service.\n\n### Configurable logging in Cloud Data Fusion\n\nCloud Data Fusion 6.11.0 offers configurable\nlogging, with Cloud Logging enabled by default. While disabling\nCloud Logging is possible, it's strongly recommended to keep it enabled to\nensure you have access to critical pipeline and instance logs.\n\nTo disable Cloud Logging, run the following command: \n\n echo '{ \"loggingConfig\": {\"instance_cloud_logging_disabled\": \"true\"}}' | curl -X POST \\\n -H \"Authorization: Bearer $(gcloud auth print-access-token)\" \\\n -H \"Content-Type: application/json\" \\\n --data @- \\\n \"https://datafusion.googleapis.com/v1/projects/\u003cvar translate=\"no\"\u003ePROJECT_ID\u003c/var\u003e/locations/\u003cvar translate=\"no\"\u003eLOCATION\u003c/var\u003e/instances?instanceId=$\u003cvar translate=\"no\"\u003eINSTANCE_ID\u003c/var\u003e?updateMask=logging_config\"\n\nReplace the following:\n\n- \u003cvar translate=\"no\"\u003ePROJECT_ID\u003c/var\u003e: the Google Cloud project ID\n- \u003cvar translate=\"no\"\u003eLOCATION\u003c/var\u003e: the [location](/data-fusion/pricing#supported_regions) of your instance\n- \u003cvar translate=\"no\"\u003eINSTANCE_ID\u003c/var\u003e: the ID of your Cloud Data Fusion instance\n\nWhat's next\n-----------\n\n- Learn more about [Cloud Data Fusion audit logging](/data-fusion/docs/how-to/audit-logging).\n- Learn how to [view advanced pipeline logs](/data-fusion/docs/how-to/viewing-stackdriver-logs)."]]
