Mantieni tutto organizzato con le raccolte
Salva e classifica i contenuti in base alle tue preferenze.
Questo documento descrive come accedere e visualizzare i log delle pipeline e i log di servizio
per Cloud Data Fusion.
A partire dalla versione 6.11 di Cloud Data Fusion, i log delle pipeline e
i log di servizio sono disponibili in Cloud Logging.
Informazioni sui tipi di log
Cloud Data Fusion genera diversi tipi di log per monitorare e risolvere i problemi relativi ai processi di integrazione dei dati:
Log della pipeline
Questi log forniscono informazioni dettagliate sulle singole esecuzioni della pipeline, incluse le esecuzioni di anteprima. Coprono tutte le fasi di esecuzione di una pipeline, ad esempio la configurazione pre-esecuzione, l'esecuzione del job e le attività post-esecuzione.
Log di servizio
Questi log forniscono informazioni dettagliate sui servizi di sistema di Cloud Data Fusion, come AppFabric e il processore AppFabric.
Questi log forniscono informazioni dettagliate sulle singole esecuzioni della pipeline.
Questi log sono particolarmente utili per la risoluzione dei problemi relativi agli errori della pipeline, in quanto acquisiscono informazioni anche prima che il provisioning del cluster Dataproc venga completato, il che può aiutare a identificare problemi o ritardi del cluster. Per
maggiori informazioni, consulta
Visualizzare i log avanzati della pipeline in Cloud Logging.
Prezzi
L'utilizzo di Cloud Logging e Cloud Monitoring comporta addebiti. Per ulteriori informazioni, consulta la pagina Prezzi di Google Cloud Observability.
(Facoltativo) Importa la dashboard di Cloud Data Fusion Logging
Per visualizzare i log della pipeline e i log del servizio utilizzando la
dashboard Cloud Data Fusion Logging, importa la dashboard:
Nella console Google Cloud , vai alla pagina Dashboard di Cloud Monitoring.
Viene visualizzato l'elenco dei log della pipeline. Puoi utilizzare i filtri per perfezionare
i risultati.
Filtra i log della pipeline
Puoi filtrare i log della pipeline in base a ID esecuzione, ID istanza, ID pipeline, località,
spazio dei nomi o etichette personalizzate.
A ogni esecuzione della pipeline Cloud Data Fusion viene assegnato un RunID univoco.
Dopo aver eseguito il deployment e l'esecuzione della pipeline, puoi trovare l'RunID della pipeline e visualizzare i log corrispondenti.
Per filtrare i log della pipeline in base a RunID, segui questi passaggi:
Cloud Data Fusion 6.11.0 offre la registrazione configurabile, con Cloud Logging abilitato per impostazione predefinita. Sebbene sia possibile disattivare
Cloud Logging, è vivamente consigliato di mantenerlo attivo per
assicurarti di avere accesso ai log critici della pipeline e dell'istanza.
Per disattivare Cloud Logging, esegui questo comando:
[[["Facile da capire","easyToUnderstand","thumb-up"],["Il problema è stato risolto","solvedMyProblem","thumb-up"],["Altra","otherUp","thumb-up"]],[["Difficile da capire","hardToUnderstand","thumb-down"],["Informazioni o codice di esempio errati","incorrectInformationOrSampleCode","thumb-down"],["Mancano le informazioni o gli esempi di cui ho bisogno","missingTheInformationSamplesINeed","thumb-down"],["Problema di traduzione","translationIssue","thumb-down"],["Altra","otherDown","thumb-down"]],["Ultimo aggiornamento 2025-09-04 UTC."],[[["\u003cp\u003eCloud Data Fusion version 6.11 and later versions utilize Cloud Logging for pipeline and service logs, offering detailed insights into data integration processes.\u003c/p\u003e\n"],["\u003cp\u003eThere are different types of logs available, including pipeline logs, service logs, and audit logs, each providing specific information about pipeline runs, system services, and administrative actions.\u003c/p\u003e\n"],["\u003cp\u003ePipeline and service logs can be viewed using the Cloud Data Fusion Logging dashboard or directly within the Logs Explorer in the Google Cloud console.\u003c/p\u003e\n"],["\u003cp\u003ePipeline logs can be filtered by various parameters like run ID, instance ID, and location to help narrow down the data to the exact process needing to be analyzed.\u003c/p\u003e\n"],["\u003cp\u003eService logs can be viewed by entering specific queries in the Logs Explorer based on the service, such as Appfabric or Dataset executor.\u003c/p\u003e\n"]]],[],null,["# View Cloud Data Fusion logs\n\nThis document describes how to access and view pipeline logs and service logs\nfor Cloud Data Fusion.\n\nStarting with Cloud Data Fusion version 6.11, pipeline logs and\nservice logs are available in Cloud Logging.\n\nAbout log types\n---------------\n\nCloud Data Fusion generates several types of logs to help monitor and\ntroubleshoot data integration processes:\n\n\u003cbr /\u003e\n\nPipeline logs These logs provide detailed information about individual pipeline runs, including preview runs. They cover all stages of a pipeline run, such as pre-run setup, job execution, and post-run activities.\n\n\u003cbr /\u003e\n\n\u003cbr /\u003e\n\nService logs These logs provide detailed information about Cloud Data Fusion system services, such as AppFabric and AppFabric processor.\n\n\u003cbr /\u003e\n\n\u003cbr /\u003e\n\nAudit logs Cloud Data Fusion generates audit logs, such as [Admin Activity audit logs](/logging/docs/audit#admin-activity). For more information, see [Cloud Data Fusion audit logging](/data-fusion/docs/how-to/audit-logging).\n\n\u003cbr /\u003e\n\n\u003cbr /\u003e\n\nAdvanced pipeline logs These logs provide detailed information about individual pipeline runs. These logs are particularly useful for troubleshooting pipeline failures, as they capture information even before the Dataproc cluster finishes provisioning, which can help identify cluster issues or delays. For more information, see [View advanced pipeline logs in Cloud Logging](/data-fusion/docs/how-to/viewing-stackdriver-logs).\n\n\u003cbr /\u003e\n\nPricing\n-------\n\nCloud Logging and Cloud Monitoring usage incurs charges. For more\ninformation, see [Google Cloud Observability pricing](/stackdriver/pricing).\n\nOptional: Import the Cloud Data Fusion Logging dashboard\n--------------------------------------------------------\n\nTo view pipeline logs and service logs using the\nCloud Data Fusion Logging dashboard, import the dashboard:\n\n1. In the Google Cloud console, go to the Cloud Monitoring **Dashboards** page.\n\n [Go to Dashboards](https://console.cloud.google.com/monitoring/dashboards)\n2. Click **View dashboard templates**.\n\n3. Search for **Cloud Data Fusion Logging** and select the dashboard.\n\n4. Click playlist_add\n **Add Cloud Data Fusion Logging dashboard to your list**.\n\nView pipeline logs\n------------------\n\nYou can view pipeline logs using the Cloud Data Fusion Logging\ndashboard or directly in the Logs Explorer.\n\n### View pipeline logs using dashboard\n\n1. If you haven't done already,\n [import the **Cloud Data Fusion Logging** dashboard](#import-dashboard).\n\n2. In the **My dashboards** section, click **Cloud Data Fusion Logging**.\n\n3. In the **Pipeline logs** section, view the list of pipeline logs. You can\n filter the logs by severity, field names, and values.\n\n To refine your search using queries, use Logs Explorer.\n\n### View pipeline logs in Logs Explorer\n\n1. In the Google Cloud console, go to the **Logs Explorer** page.\n\n [Go to Logs Explorer](https://console.cloud.google.com/logs/query)\n2. Enter the following query:\n\n resource.type=\"datafusion.googleapis.com/PipelineV2\"\n\n This displays the list of pipeline logs. You can use filters to refine\n the results.\n\n### Filter pipeline logs\n\nYou can filter pipeline logs by run ID, instance ID, pipeline ID, location,\nnamespace, or custom labels.\n\nEvery Cloud Data Fusion pipeline run is assigned a unique `RunID`.\nAfter you deploy and run your pipeline, you can find the `RunID` of your\npipeline and view the corresponding pipeline logs.\n\nTo filter pipeline logs by `RunID`, follow these steps:\n\n1. [Get the pipeline's RunID](/data-fusion/docs/how-to/viewing-stackdriver-logs#get_the_pipelines_runid).\n\n2. In the Google Cloud console, go to the **Logs Explorer** page.\n\n3. Enter the following query:\n\n resource.type=\"datafusion.googleapis.com/PipelineV2\" resource.labels.run_id=\u003cvar translate=\"no\"\u003eRUN_ID\u003c/var\u003e\n\nView service logs\n-----------------\n\nYou can view service logs using the Cloud Data Fusion Logging\ndashboard or in the Logs Explorer.\n\n### View service logs using dashboard\n\n1. If you haven't done already,\n [import the **Cloud Data Fusion Logging** dashboard](#import-dashboard).\n\n2. In the **My dashboards** section, click **Cloud Data Fusion Logging**.\n\n3. In the **Service logs** section, view the list of service logs. You can\n filter the logs by severity, field names, and values.\n\n To refine your search using queries, use Logs Explorer.\n\n### View service logs in Logs Explorer\n\n1. In the Google Cloud console, go to the **Logs Explorer** page.\n\n [Go to Logs Explorer](https://console.cloud.google.com/logs/query)\n2. Find the service logs by entering the specific query for that service.\n\n### Configurable logging in Cloud Data Fusion\n\nCloud Data Fusion 6.11.0 offers configurable\nlogging, with Cloud Logging enabled by default. While disabling\nCloud Logging is possible, it's strongly recommended to keep it enabled to\nensure you have access to critical pipeline and instance logs.\n\nTo disable Cloud Logging, run the following command: \n\n echo '{ \"loggingConfig\": {\"instance_cloud_logging_disabled\": \"true\"}}' | curl -X POST \\\n -H \"Authorization: Bearer $(gcloud auth print-access-token)\" \\\n -H \"Content-Type: application/json\" \\\n --data @- \\\n \"https://datafusion.googleapis.com/v1/projects/\u003cvar translate=\"no\"\u003ePROJECT_ID\u003c/var\u003e/locations/\u003cvar translate=\"no\"\u003eLOCATION\u003c/var\u003e/instances?instanceId=$\u003cvar translate=\"no\"\u003eINSTANCE_ID\u003c/var\u003e?updateMask=logging_config\"\n\nReplace the following:\n\n- \u003cvar translate=\"no\"\u003ePROJECT_ID\u003c/var\u003e: the Google Cloud project ID\n- \u003cvar translate=\"no\"\u003eLOCATION\u003c/var\u003e: the [location](/data-fusion/pricing#supported_regions) of your instance\n- \u003cvar translate=\"no\"\u003eINSTANCE_ID\u003c/var\u003e: the ID of your Cloud Data Fusion instance\n\nWhat's next\n-----------\n\n- Learn more about [Cloud Data Fusion audit logging](/data-fusion/docs/how-to/audit-logging).\n- Learn how to [view advanced pipeline logs](/data-fusion/docs/how-to/viewing-stackdriver-logs)."]]
