Sign in to your Google Cloud account. If you're new to
Google Cloud,
create an account to evaluate how our products perform in
real-world scenarios. New customers also get $300 in free credits to
run, test, and deploy workloads.
In the Google Cloud console, on the project selector page,
select or create a Google Cloud project.
Di bagian Labels, klik Add label dan buat label berikut. Setelah memasukkan setiap label, klik Selesai, lalu klik
Tambahkan label lagi untuk membuat label berikutnya.
Nama label
Jenis label
Nama kolom
Project
STRING
resource.labels.project_id
Message
STRING
jsonPayload.message
LoggerName
STRING
labels.loggerName
ClusterName
STRING
resource.labels.cluster_name
SparkPhase
STRING
labels.".workflowSparkId"
Region
STRING
resource.labels.region
Pipeline
STRING
labels.".applicationId"
RunId
STRING
labels.".runId"
Namespace
STRING
labels.".namespaceId"
LogLevel
STRING
labels.levelName
Klik Create metric.
Metrik yang baru dibuat akan muncul di tabel metrik yang ditentukan pengguna.
Jika metrik tidak langsung terlihat, muat ulang halaman.
Dasbor berisi diagram berikut:
Semua pipeline
Pipeline yang selesai
Pipeline yang gagal
Semua operasi pipeline
Operasi pipeline selesai
Operasi pipeline gagal
Cluster Dataproc untuk menjalankan
Setelah metrik dibuat, mungkin perlu waktu hingga 24 jam untuk mulai
menampilkan data deret waktu.
Di editor teks, buka file JSON yang Anda download.
Salin konten file JSON yang didownload dan tempelkan ke editor JSON, menggantikan konten yang ada di editor JSON secara default.
Klik Terapkan perubahan.
Tindakan ini akan memuat ulang dasbor. Pipeline Cloud Data Fusion yang berjalan setelah metrik dibuat akan muncul di dasbor. Jika tidak ada pipeline yang dijalankan setelah metrik dibuat, dasbor akan kosong.
Simpan otomatis diaktifkan secara default. Jika simpan otomatis dinonaktifkan, klik Simpan
untuk menyimpan dasbor.
Klik Close editor.
Dasbor baru Anda akan muncul dalam daftar dasbor di halaman
Ringkasan pemantauan.
Pembersihan
Agar akun Google Cloud Anda tidak dikenai biaya untuk
resource yang digunakan pada halaman ini, ikuti langkah-langkah berikut.
[[["Mudah dipahami","easyToUnderstand","thumb-up"],["Memecahkan masalah saya","solvedMyProblem","thumb-up"],["Lainnya","otherUp","thumb-up"]],[["Sulit dipahami","hardToUnderstand","thumb-down"],["Informasi atau kode contoh salah","incorrectInformationOrSampleCode","thumb-down"],["Informasi/contoh yang saya butuhkan tidak ada","missingTheInformationSamplesINeed","thumb-down"],["Masalah terjemahan","translationIssue","thumb-down"],["Lainnya","otherDown","thumb-down"]],["Terakhir diperbarui pada 2025-09-04 UTC."],[[["\u003cp\u003eThis guide explains how to create a pipeline monitoring dashboard in Cloud Monitoring for Cloud Data Fusion pipelines.\u003c/p\u003e\n"],["\u003cp\u003eBefore creating the dashboard, you must enable the necessary APIs (Cloud Data Fusion, BigQuery, Cloud Storage, and Dataproc) and have the Monitoring Editor IAM role.\u003c/p\u003e\n"],["\u003cp\u003eCreating a Cloud Data Fusion instance with Stackdriver logging enabled is required to use Cloud Logging with pipelines, which is needed for the dashboard.\u003c/p\u003e\n"],["\u003cp\u003eThe process involves creating a log-based metric named \u003ccode\u003epipeline_logs\u003c/code\u003e with specific filters and labels to capture pipeline data and running the pipelines after the metric is created.\u003c/p\u003e\n"],["\u003cp\u003eA JSON file containing the dashboard configuration must be downloaded, and imported into Cloud Monitoring to display the metrics for the created pipeline logs.\u003c/p\u003e\n"]]],[],null,["# Create a pipeline monitoring dashboard using Cloud Monitoring\n=============================================================\n\nLearn how to use Cloud Monitoring to create a dashboard to monitor pipelines.\n\n*** ** * ** ***\n\nTo follow step-by-step guidance for this task directly in the\nGoogle Cloud console, click **Guide me**:\n\n[Guide me](https://console.cloud.google.com/freetrial?redirectPath=/?walkthrough_id=data-fusion--pipeline-monitoring-dashboard)\n\n*** ** * ** ***\n\nBefore you begin\n----------------\n\n- Sign in to your Google Cloud account. If you're new to Google Cloud, [create an account](https://console.cloud.google.com/freetrial) to evaluate how our products perform in real-world scenarios. New customers also get $300 in free credits to run, test, and deploy workloads.\n- In the Google Cloud console, on the project selector page,\n select or create a Google Cloud project.\n\n | **Note**: If you don't plan to keep the resources that you create in this procedure, create a project instead of selecting an existing project. After you finish these steps, you can delete the project, removing all resources associated with the project.\n\n [Go to project selector](https://console.cloud.google.com/projectselector2/home/dashboard)\n-\n [Verify that billing is enabled for your Google Cloud project](/billing/docs/how-to/verify-billing-enabled#confirm_billing_is_enabled_on_a_project).\n\n-\n\n\n Enable the Cloud Data Fusion, BigQuery, Cloud Storage, and Dataproc APIs.\n\n\n [Enable the APIs](https://console.cloud.google.com/flows/enableapi?apiid=datafusion.googleapis.com,bigquery.googleapis.com,storage.googleapis.com,dataproc.googleapis.com)\n\n- In the Google Cloud console, on the project selector page,\n select or create a Google Cloud project.\n\n | **Note**: If you don't plan to keep the resources that you create in this procedure, create a project instead of selecting an existing project. After you finish these steps, you can delete the project, removing all resources associated with the project.\n\n [Go to project selector](https://console.cloud.google.com/projectselector2/home/dashboard)\n-\n [Verify that billing is enabled for your Google Cloud project](/billing/docs/how-to/verify-billing-enabled#confirm_billing_is_enabled_on_a_project).\n\n-\n\n\n Enable the Cloud Data Fusion, BigQuery, Cloud Storage, and Dataproc APIs.\n\n\n [Enable the APIs](https://console.cloud.google.com/flows/enableapi?apiid=datafusion.googleapis.com,bigquery.googleapis.com,storage.googleapis.com,dataproc.googleapis.com)\n\n1.\n\n\n Enable the Cloud Data Fusion, BigQuery, Cloud Storage, and Dataproc APIs.\n\n\n [Enable the APIs](https://console.cloud.google.com/flows/enableapi?apiid=datafusion.googleapis.com,bigquery.googleapis.com,storage.googleapis.com,dataproc.googleapis.com)\n2. To create custom dashboards, you must be granted the\n [Monitoring Editor](/iam/docs/understanding-roles#monitoring.editor)\n (`roles/monitoring.editor`) IAM role on the service account.\n\n For more information about granting roles, see\n [Manage access](/iam/docs/granting-changing-revoking-access).\n\nCreate a Cloud Data Fusion instance with Cloud Logging enabled\n--------------------------------------------------------------\n\nTo use Cloud Logging with your Cloud Data Fusion pipeline, create a\nCloud Data Fusion instance with Cloud Logging enabled:\n\n1. Go to the Cloud Data Fusion **Instances** page and\n click **Create instance**.\n\n\n [Create an instance](https://console.cloud.google.com/data-fusion/instance-create)\n\n \u003cbr /\u003e\n\n \u003cbr /\u003e\n\n2. In the **Instance name** field, enter a name for your new instance.\n\n3. From the **Region** drop-down, select the Google Cloud region in which you\n want to create the instance.\n\n4. From the **Version** drop-down, select a Cloud Data Fusion version.\n\n5. Select an **Edition**.\n\n6. Expand **Advanced options**.\n\n7. In the **Logging and monitoring** section, select\n **Enable Stackdriver logging service**.\n\n8. Click **Create**.\n\nAfter you create an instance, you can't enable Cloud Logging in the Google Cloud console.\n\n\u003cbr /\u003e\n\nCreate a log-based metric\n-------------------------\n\n1. Go to the Cloud Logging **Log-based metrics** page:\n\n\n [Go to Log-based metrics](https://console.cloud.google.com/logs/metrics)\n\n \u003cbr /\u003e\n\n \u003cbr /\u003e\n\n2. Click **Create metric**.\n\n3. On the **Create a metric** page, do the following:\n\n 1. For **Metric type** , select **Counter**.\n 2. In the **Log-based metric name** field, enter `pipeline_logs`.\n 3. In the **Units** field, enter `1`.\n 4. In the **Build filter** field, enter the following:\n\n resource.type=\"cloud_dataproc_cluster\"\n log_name=~\"projects/.*/logs/datafusion-pipeline-logs\"\n\n 5. In the **Labels** section, click **Add label** and create the following\n labels. After entering each label, click **Done** , and click\n **Add label** again to create the next label.\n\n 6. Click **Create metric**.\n\n The newly created metric appears in the user-defined metrics table.\n If the metric isn't immediately visible, refresh the page.\n\n The dashboard contains the following charts:\n - All pipelines\n - Completed pipelines\n - Failed pipelines\n - All pipeline runs\n - Completed pipeline runs\n - Failed pipeline runs\n - Dataproc clusters for runs\n\n After a metric is created, it might take up to 24 hours to start\n displaying the time series data.\n\nInstall the dashboard\n---------------------\n\n1. [Download the JSON file](/static/data-fusion/docs/tutorials/sample_datasets/datafusion-pipelines-overview.json) to your local machine.\n\n2. Go to the Cloud Monitoring **Dashboards** page:\n\n\n [Go to Monitoring dashboards](https://console.cloud.google.com/monitoring/dashboards)\n\n \u003cbr /\u003e\n\n \u003cbr /\u003e\n\n3. Click **Create dashboard**.\n\n4. Click settings **Dashboard settings**\n **\\\u003e JSON \\\u003e JSON editor**.\n\n5. In a text editor, open the JSON file that you downloaded.\n\n6. Copy the content of the downloaded JSON file and paste it into the JSON\n editor, replacing the content that the JSON editor contains by default.\n\n7. Click **Apply changes**.\n\n This refreshes the dashboard. The Cloud Data Fusion pipelines run\n after the metric was created, appear in the dashboard. If no pipelines were\n run after the metric was created, the dashboard will be empty.\n8. Autosave is enabled by default. If autosave is disabled, click **Save**\n to save the dashboard.\n\n9. Click **Close editor**.\n\n Your new dashboard appears in the list of dashboards on the\n **Monitoring overview** page.\n\nClean up\n--------\n\n\nTo avoid incurring charges to your Google Cloud account for\nthe resources used on this page, follow these steps.\n\n### Delete the Cloud Data Fusion instance\n\nFollow these instructions to\n[delete your Cloud Data Fusion instance](/data-fusion/docs/how-to/delete-instance).\n\n### Delete the project\n\n\nThe easiest way to eliminate billing is to delete the project that you\ncreated for the tutorial.\n\nTo delete the project:\n\n| **Caution** : Deleting a project has the following effects:\n|\n| - **Everything in the project is deleted.** If you used an existing project for the tasks in this document, when you delete it, you also delete any other work you've done in the project.\n| - **Custom project IDs are lost.** When you created this project, you might have created a custom project ID that you want to use in the future. To preserve the URLs that use the project ID, such as an `appspot.com` URL, delete selected resources inside the project instead of deleting the whole project.\n|\n|\n| If you plan to explore multiple architectures, tutorials, or quickstarts, reusing projects\n| can help you avoid exceeding project quota limits.\n1. In the Google Cloud console, go to the **Manage resources** page.\n\n [Go to Manage resources](https://console.cloud.google.com/iam-admin/projects)\n2. In the project list, select the project that you want to delete, and then click **Delete**.\n3. In the dialog, type the project ID, and then click **Shut down** to delete the project.\n\n\u003cbr /\u003e\n\n\u003cbr /\u003e\n\nWhat's next\n-----------\n\n- Learn more about [Cloud Monitoring](/monitoring/docs/monitoring-overview)."]]
