Mantenha tudo organizado com as coleções
Salve e categorize o conteúdo com base nas suas preferências.
Neste documento, descrevemos como acessar e visualizar registros de pipeline e de serviço
do Cloud Data Fusion.
A partir da versão 6.11 do Cloud Data Fusion, os registros de pipeline e
de serviço estão disponíveis no Cloud Logging.
Sobre os tipos de registro
O Cloud Data Fusion gera vários tipos de registros para ajudar a monitorar e
resolver problemas nos processos de integração de dados:
Registros do pipeline
Esses registros fornecem informações detalhadas sobre execuções de pipeline individuais, incluindo execuções de prévia. Elas abrangem todas as etapas de uma execução de pipeline, como configuração pré-execução, execução de jobs e atividades pós-execução.
Registros de serviço
Esses registros fornecem informações detalhadas sobre os serviços do sistema do Cloud Data Fusion, como o AppFabric e o processador do AppFabric.
Esses registros fornecem informações detalhadas sobre execuções de pipeline individuais.
Esses registros são particularmente úteis para resolver falhas de pipeline, já que capturam informações mesmo antes de o cluster do Dataproc terminar o provisionamento, o que pode ajudar a identificar problemas ou atrasos no cluster. Para
mais informações, consulte
Ver registros avançados de pipeline no Cloud Logging.
Isso mostra a lista de registros de pipeline. Você pode usar filtros para refinar os resultados.
Filtrar registros de pipeline
É possível filtrar os registros do pipeline por ID da execução, ID da instância, ID do pipeline, local, namespace ou rótulos personalizados.
Cada execução do pipeline do Cloud Data Fusion recebe um RunID exclusivo.
Depois de implantar e executar o pipeline, encontre o RunID dele e confira os registros correspondentes.
Para filtrar registros de pipeline por RunID, siga estas etapas:
Geração de registros configurável no Cloud Data Fusion
O Cloud Data Fusion 6.11.0 oferece registros configuráveis, com o Cloud Logging ativado por padrão. Embora seja possível desativar o Cloud Logging, é altamente recomendável mantê-lo ativado para garantir o acesso a registros essenciais de pipeline e de instância.
Para desativar o Cloud Logging, execute o seguinte comando:
[[["Fácil de entender","easyToUnderstand","thumb-up"],["Meu problema foi resolvido","solvedMyProblem","thumb-up"],["Outro","otherUp","thumb-up"]],[["Difícil de entender","hardToUnderstand","thumb-down"],["Informações incorretas ou exemplo de código","incorrectInformationOrSampleCode","thumb-down"],["Não contém as informações/amostras de que eu preciso","missingTheInformationSamplesINeed","thumb-down"],["Problema na tradução","translationIssue","thumb-down"],["Outro","otherDown","thumb-down"]],["Última atualização 2025-09-04 UTC."],[[["\u003cp\u003eCloud Data Fusion version 6.11 and later versions utilize Cloud Logging for pipeline and service logs, offering detailed insights into data integration processes.\u003c/p\u003e\n"],["\u003cp\u003eThere are different types of logs available, including pipeline logs, service logs, and audit logs, each providing specific information about pipeline runs, system services, and administrative actions.\u003c/p\u003e\n"],["\u003cp\u003ePipeline and service logs can be viewed using the Cloud Data Fusion Logging dashboard or directly within the Logs Explorer in the Google Cloud console.\u003c/p\u003e\n"],["\u003cp\u003ePipeline logs can be filtered by various parameters like run ID, instance ID, and location to help narrow down the data to the exact process needing to be analyzed.\u003c/p\u003e\n"],["\u003cp\u003eService logs can be viewed by entering specific queries in the Logs Explorer based on the service, such as Appfabric or Dataset executor.\u003c/p\u003e\n"]]],[],null,["# View Cloud Data Fusion logs\n\nThis document describes how to access and view pipeline logs and service logs\nfor Cloud Data Fusion.\n\nStarting with Cloud Data Fusion version 6.11, pipeline logs and\nservice logs are available in Cloud Logging.\n\nAbout log types\n---------------\n\nCloud Data Fusion generates several types of logs to help monitor and\ntroubleshoot data integration processes:\n\n\u003cbr /\u003e\n\nPipeline logs These logs provide detailed information about individual pipeline runs, including preview runs. They cover all stages of a pipeline run, such as pre-run setup, job execution, and post-run activities.\n\n\u003cbr /\u003e\n\n\u003cbr /\u003e\n\nService logs These logs provide detailed information about Cloud Data Fusion system services, such as AppFabric and AppFabric processor.\n\n\u003cbr /\u003e\n\n\u003cbr /\u003e\n\nAudit logs Cloud Data Fusion generates audit logs, such as [Admin Activity audit logs](/logging/docs/audit#admin-activity). For more information, see [Cloud Data Fusion audit logging](/data-fusion/docs/how-to/audit-logging).\n\n\u003cbr /\u003e\n\n\u003cbr /\u003e\n\nAdvanced pipeline logs These logs provide detailed information about individual pipeline runs. These logs are particularly useful for troubleshooting pipeline failures, as they capture information even before the Dataproc cluster finishes provisioning, which can help identify cluster issues or delays. For more information, see [View advanced pipeline logs in Cloud Logging](/data-fusion/docs/how-to/viewing-stackdriver-logs).\n\n\u003cbr /\u003e\n\nPricing\n-------\n\nCloud Logging and Cloud Monitoring usage incurs charges. For more\ninformation, see [Google Cloud Observability pricing](/stackdriver/pricing).\n\nOptional: Import the Cloud Data Fusion Logging dashboard\n--------------------------------------------------------\n\nTo view pipeline logs and service logs using the\nCloud Data Fusion Logging dashboard, import the dashboard:\n\n1. In the Google Cloud console, go to the Cloud Monitoring **Dashboards** page.\n\n [Go to Dashboards](https://console.cloud.google.com/monitoring/dashboards)\n2. Click **View dashboard templates**.\n\n3. Search for **Cloud Data Fusion Logging** and select the dashboard.\n\n4. Click playlist_add\n **Add Cloud Data Fusion Logging dashboard to your list**.\n\nView pipeline logs\n------------------\n\nYou can view pipeline logs using the Cloud Data Fusion Logging\ndashboard or directly in the Logs Explorer.\n\n### View pipeline logs using dashboard\n\n1. If you haven't done already,\n [import the **Cloud Data Fusion Logging** dashboard](#import-dashboard).\n\n2. In the **My dashboards** section, click **Cloud Data Fusion Logging**.\n\n3. In the **Pipeline logs** section, view the list of pipeline logs. You can\n filter the logs by severity, field names, and values.\n\n To refine your search using queries, use Logs Explorer.\n\n### View pipeline logs in Logs Explorer\n\n1. In the Google Cloud console, go to the **Logs Explorer** page.\n\n [Go to Logs Explorer](https://console.cloud.google.com/logs/query)\n2. Enter the following query:\n\n resource.type=\"datafusion.googleapis.com/PipelineV2\"\n\n This displays the list of pipeline logs. You can use filters to refine\n the results.\n\n### Filter pipeline logs\n\nYou can filter pipeline logs by run ID, instance ID, pipeline ID, location,\nnamespace, or custom labels.\n\nEvery Cloud Data Fusion pipeline run is assigned a unique `RunID`.\nAfter you deploy and run your pipeline, you can find the `RunID` of your\npipeline and view the corresponding pipeline logs.\n\nTo filter pipeline logs by `RunID`, follow these steps:\n\n1. [Get the pipeline's RunID](/data-fusion/docs/how-to/viewing-stackdriver-logs#get_the_pipelines_runid).\n\n2. In the Google Cloud console, go to the **Logs Explorer** page.\n\n3. Enter the following query:\n\n resource.type=\"datafusion.googleapis.com/PipelineV2\" resource.labels.run_id=\u003cvar translate=\"no\"\u003eRUN_ID\u003c/var\u003e\n\nView service logs\n-----------------\n\nYou can view service logs using the Cloud Data Fusion Logging\ndashboard or in the Logs Explorer.\n\n### View service logs using dashboard\n\n1. If you haven't done already,\n [import the **Cloud Data Fusion Logging** dashboard](#import-dashboard).\n\n2. In the **My dashboards** section, click **Cloud Data Fusion Logging**.\n\n3. In the **Service logs** section, view the list of service logs. You can\n filter the logs by severity, field names, and values.\n\n To refine your search using queries, use Logs Explorer.\n\n### View service logs in Logs Explorer\n\n1. In the Google Cloud console, go to the **Logs Explorer** page.\n\n [Go to Logs Explorer](https://console.cloud.google.com/logs/query)\n2. Find the service logs by entering the specific query for that service.\n\n### Configurable logging in Cloud Data Fusion\n\nCloud Data Fusion 6.11.0 offers configurable\nlogging, with Cloud Logging enabled by default. While disabling\nCloud Logging is possible, it's strongly recommended to keep it enabled to\nensure you have access to critical pipeline and instance logs.\n\nTo disable Cloud Logging, run the following command: \n\n echo '{ \"loggingConfig\": {\"instance_cloud_logging_disabled\": \"true\"}}' | curl -X POST \\\n -H \"Authorization: Bearer $(gcloud auth print-access-token)\" \\\n -H \"Content-Type: application/json\" \\\n --data @- \\\n \"https://datafusion.googleapis.com/v1/projects/\u003cvar translate=\"no\"\u003ePROJECT_ID\u003c/var\u003e/locations/\u003cvar translate=\"no\"\u003eLOCATION\u003c/var\u003e/instances?instanceId=$\u003cvar translate=\"no\"\u003eINSTANCE_ID\u003c/var\u003e?updateMask=logging_config\"\n\nReplace the following:\n\n- \u003cvar translate=\"no\"\u003ePROJECT_ID\u003c/var\u003e: the Google Cloud project ID\n- \u003cvar translate=\"no\"\u003eLOCATION\u003c/var\u003e: the [location](/data-fusion/pricing#supported_regions) of your instance\n- \u003cvar translate=\"no\"\u003eINSTANCE_ID\u003c/var\u003e: the ID of your Cloud Data Fusion instance\n\nWhat's next\n-----------\n\n- Learn more about [Cloud Data Fusion audit logging](/data-fusion/docs/how-to/audit-logging).\n- Learn how to [view advanced pipeline logs](/data-fusion/docs/how-to/viewing-stackdriver-logs)."]]
