The Dataflow web-based monitoring interface includes a dashboard that monitors your Dataflow jobs at the project level. The charts show data for all of the jobs in one project.
The dashboard can help you with the following tasks:
- Detect and identify the source of quota errors.
- Detect anomalous horizontal autoscaling in a job.
- Identify slow or stuck streaming jobs.
The dashboard uses Cloud Monitoring to access Dataflow job metrics. To customize the information displayed in the charts, use Metrics Explorer.
Features
The dashboard includes the following features:
- Choose which jobs appear in the dashboard by using regular expressions.
- Access the job details page from individual charts.
- Customize the dashboard widgets and charts.
Required roles
To get the permission that you need to see the graph data,
ask your administrator to grant you the
Monitoring Viewer (roles/monitoring.viewer) IAM role.
For more information about granting roles, see Manage access.
This predefined role contains the
monitoring.timeSeries.list permission, which is
required to see the graph data.
You might also be able to get this permission with custom roles or other predefined roles.
Metrics
By default, the following charts appear in your dashboard. For more information about the metrics displayed, see Job metrics.
| Chart | Description | Support |
|---|---|---|
| Running jobs | A time-series chart that shows the number of jobs running in the project. | Batch and streaming jobs |
| Workers per job | A time-series chart that shows the number of workers being used per
job. Use this chart to understand autoscaling behavior across the project.
You can see if jobs have unexpected or unusual scaling behavior. Use this chart with the quota and CPU charts to identify jobs whose scaling is limited by quota errors. |
Batch and streaming jobs |
| Quota exceeded errors | A time-series chart that shows the history of quota exceeded errors in
the project, scoped to Compute Engine CPU quotas. Compute Engine has
both per-region total CPU quotas and, for some machine families,
per-region, per-type quotas. Any of these quotas might prevent a job from
starting or scaling up. Use this chart with the quota and CPU charts to identify the source of quota errors. |
Batch and streaming jobs |
| CPUs per job | A time-series chart that shows the number of CPUs being used by the
workers of each job. This chart also shows the machine type and location
for each job. Machine types in the same family have different numbers of
CPUs. The total number of CPUs affects Compute Engine quotas. Use this chart to identify the source of quota errors. |
Batch and streaming jobs |
| System latency | A time-series chart that shows the maximum number
of seconds that an item of data has been processing or awaiting processing
for each job. Use this chart to identify streaming jobs that have an unusual delay between when data appears in a source and is written to all sinks. |
Streaming jobs |
| Data freshness | A time-series chart that shows the maximum data
freshness for any stage in each job. Use this chart to find streaming jobs that might be slow or stuck. |
Streaming jobs |
| Max backlog bytes | A time-series chart that shows the maximum
backlog bytes for any stage in each job. Use this chart to identify anomalies that indicate a processing bottleneck. |
Streaming jobs |
Access the dashboard
To access the dashboard, follow these steps:
- Sign in to the Google Cloud console.
- Select your Google Cloud project.
- Open the navigation menu.
- In Analytics, click Dataflow.
In the Dataflow navigation menu, click Monitoring.
Customize the dashboard
You can customize the dashboard contents and the information displayed in the charts. When you edit the dashboard, a new, customized dashboard is created.
The dashboard uses Cloud Monitoring to access Dataflow job metrics. Use the Cloud Monitoring tools to customize the charts.
- Open the dashboard and click Customize Dashboard.
- Modify your dashboard.
- To filter the jobs that display on the dashboard, see Add temporary filters to a custom dashboard and Add permanent filters to a custom dashboard.
- To edit or remove widgets, see Manage dashboard widgets.
- To edit the contents of the charts, see Select metrics for charts on dashboards.
- To add charts to the dashboard, see Add charts and tables to a custom dashboard.
- Click Save, and then click View customized dashboard.
After you create a customized dashboard, to return to the default dashboard, in the Dashboard menu, select Predefined.
Troubleshooting
This section provides instructions for troubleshooting common issues
No data is available
When you open your dashboard, one or more charts shows the following message:
No data is available for the selected time frame.
This message appears when the time period covered in the charts doesn't have any data. To resolve this issue, change or expand the time range.
To change the displayed time range, on the chart, click Explore data, and then use the time-range selector.
Unable to restore deleted widgets
When you remove a widget from the dashboard, you create a customized dashboard. After you create a customized dashboard, to return to the default dashboard, in the Dashboard menu, select Predefined.
Unable to view charts
To see the graph data, you need the monitoring.timeSeries.list
permission. For more information, see Required roles.
What's next
- Learn more about individual job metrics.
- Explore metrics with Cloud Monitoring.
- Troubleshoot slow or stuck jobs.