Dataflow project monitoring dashboard

The Dataflow web-based monitoring interface includes a dashboard that monitors your Dataflow jobs at the project level. The charts show data for all of the jobs in one project.

Go to dashboard

The dashboard can help you with the following tasks:

  • Detect and identify the source of quota errors.
  • Detect anomalous horizontal autoscaling in a job.
  • Identify slow or stuck streaming jobs.

The dashboard uses Cloud Monitoring to access Dataflow job metrics. To customize the information displayed in the charts, use Metrics Explorer.

Features

The dashboard includes the following features:

  • Choose which jobs appear in the dashboard by using regular expressions.
  • Access the job details page from individual charts.
  • Customize the dashboard widgets and charts.

Required roles

To get the permission that you need to see the graph data, ask your administrator to grant you the Monitoring Viewer (roles/monitoring.viewer) IAM role. For more information about granting roles, see Manage access to projects, folders, and organizations.

This predefined role contains the monitoring.timeSeries.list permission, which is required to see the graph data.

You might also be able to get this permission with custom roles or other predefined roles.

Metrics

By default, the following charts appear in your dashboard. For more information about the metrics displayed, see Job metrics.

Chart Description Support
Running jobs A time-series chart that shows the number of jobs running in the project. Batch and streaming jobs
Workers per job A time-series chart that shows the number of workers being used per job. Use this chart to understand autoscaling behavior across the project. You can see if jobs have unexpected or unusual scaling behavior.

Use this chart with the quota and CPU charts to identify jobs whose scaling is limited by quota errors.
Batch and streaming jobs
Quota exceeded errors A time-series chart that shows the history of quota exceeded errors in the project, scoped to Compute Engine CPU quotas. Compute Engine has both per-region total CPU quotas and, for some machine families, per-region, per-type quotas. Any of these quotas might prevent a job from starting or scaling up.

Use this chart with the quota and CPU charts to identify the source of quota errors.
Batch and streaming jobs
CPUs per job A time-series chart that shows the number of CPUs being used by the workers of each job. This chart also shows the machine type and location for each job. Machine types in the same family have different numbers of CPUs. The total number of CPUs affects Compute Engine quotas.

Use this chart to identify the source of quota errors.
Batch and streaming jobs
System latency A time-series chart that shows the maximum number of seconds that an item of data has been processing or awaiting processing for each job.

Use this chart to identify streaming jobs that have an unusual delay between when data appears in a source and is written to all sinks.
Streaming jobs
Data freshness A time-series chart that shows the maximum data freshness for any stage in each job.

Use this chart to find streaming jobs that might be slow or stuck.
Streaming jobs
Max backlog bytes A time-series chart that shows the maximum backlog bytes for any stage in each job.

Use this chart to identify anomalies that indicate a processing bottleneck.
Streaming jobs

Access the dashboard

To access the dashboard, follow these steps:

  1. Sign in to the Google Cloud console.
  2. Select your Google Cloud project.
  3. Open the navigation menu.
  4. In Analytics, click Dataflow.
  5. In the Dataflow navigation menu, click Monitoring.

    Go to dashboard

Customize the dashboard

You can customize the dashboard contents and the information displayed in the charts. When you edit the dashboard, a new, customized dashboard is created.

The dashboard uses Cloud Monitoring to access Dataflow job metrics. Use the Cloud Monitoring tools to customize the charts.

  1. Open the dashboard and click Customize Dashboard.
  2. Modify your dashboard.
  3. Click Save, and then click View customized dashboard.

After you create a customized dashboard, to return to the default dashboard, in the Dashboard menu, select Predefined.

Troubleshooting

This section provides instructions for troubleshooting common issues

No data is available

When you open your dashboard, one or more charts shows the following message:

No data is available for the selected time frame.

This message appears when the time period covered in the charts doesn't have any data. To resolve this issue, change or expand the time range.

To change the displayed time range, on the chart, click Explore data, and then use the time-range selector.

Unable to restore deleted widgets

When you remove a widget from the dashboard, you create a customized dashboard. After you create a customized dashboard, to return to the default dashboard, in the Dashboard menu, select Predefined.

Unable to view charts

To see the graph data, you need the monitoring.timeSeries.list permission. For more information, see Required roles.

What's next