When you use BigQuery Engine for Apache Flink to create deployments and run jobs, you can use the web-based monitoring interface to view details about and interact with your deployments and jobs.
Access the BigQuery Engine for Apache Flink monitoring interface by using the Google Cloud console. The monitoring interface can show you:
- A list of all BigQuery Engine for Apache Flink deployments
- A list of all running BigQuery Engine for Apache Flink jobs
- Details about the status of your job
- A graphical representation of each job
- Errors or warnings that occur during a job
- Additional diagnostics for deployments and jobs
You can view charts within the BigQuery Engine for Apache Flink monitoring interface. These charts display metrics over the duration of your deployments and jobs.
Access the BigQuery Engine for Apache Flink monitoring interface
To access the monitoring interface, follow these steps:
- Sign in to the Google Cloud console.
- Select your Google Cloud project.
- Open the navigation menu.
- In Analytics, click BigQuery Engine for Apache Flink.
- To see details about deployments, click Deployments. To see details about jobs, click Jobs.
A list of deployments or jobs appears along with their statuses. For more information about a deployment or job, click the name. If you don't see any deployments or jobs, you need to create a deployment or run a new job.
To learn how to create a deployment, see Create and manage deployments. To learn how to run a job, see Create and manage jobs.
A deployment can have the following statuses:
- —: the monitoring interface has not yet received a status from the BigQuery Engine for Apache Flink service
- Initializing: the deployment is being created
- Created: the deployment is created
- Terminating: the deployment is being deleted
- Terminated: the deployment has been deleted
A job can have the following statuses:
- —: the monitoring interface has not yet received a status from the BigQuery Engine for Apache Flink service
- Initializing: the job is created, but the system needs some time to prepare before launching
- Created: the job is created but not yet running
- Running: the job is running
- Cancelling: the job is being cancelled
- Cancelled: the job is cancelled
- Updating: the job is being updated
- Updated: the job is updated
- Finished: the job finished successfully
- Failed: the job failed to complete
- Restarting: the job is starting again. This status might appear if the job encountered an error and is restarting from a checkpoint.
- Suspended: the job is suspended
Access monitoring views
You can access monitoring views for deployments and jobs.
Deployments
To access charts for monitoring your deployments, in the BigQuery Engine for Apache Flink monitoring interface, click the deployment name. The Deployment details page displays, and it contains the following information:
- Deployment info panel: descriptive information about your deployment
- Logs: logs generated by the BigQuery Engine for Apache Flink service at the deployment level. For more information, see Logging.
- Metrics: charts with information about your deployment
Within the Deployment details page, you can switch your deployment view with the Details, Logs, and Metrics tabs.
Jobs
To access charts for monitoring your jobs, in the BigQuery Engine for Apache Flink monitoring interface, click the job name. The Job details page displays, and it contains the following information:
- Job graph: a graphical representation of your job
- Job info panel: descriptive information about your job
- Logs: job logs generated by the BigQuery Engine for Apache Flink service. For more information, see Logging.
- Metrics: charts with information about the running of your job
Within the Job details page, you can switch your job view with the Graph and the Metrics tabs.
Job graphs
When you select a specific job, the monitoring interface provides a graphical representation of your job: the job graph. The Graph tab in the console also provides a job summary, a job log, and information about each step in the job. For more details about job graphs, see BigQuery Engine for Apache Flink job graphs.
Metrics
You can access monitoring charts for deployments and jobs.
Deployment metrics
In the Metrics tab of the BigQuery Engine for Apache Flink deployment monitoring interface, you can view charts. The following metrics charts are available:
- CPU utilization
- Memory utilization
Job metrics
In the Metrics tab of the BigQuery Engine for Apache Flink job monitoring interface, you can view charts. The following metrics charts are available:
- Number of records in per second
- Number of records out per second
- Input watermark
- Output watermark
- Autoscaling charts
What's next
See details about the available job metrics
Use Cloud Monitoring to view additional BigQuery Engine for Apache Flink metrics, including custom metrics