Monitoring environments with Cloud Monitoring

You can use Cloud Monitoring and Cloud Logging with Cloud Composer.

Monitoring provides visibility into the performance, uptime, and overall health of cloud-powered applications. Cloud Monitoring collects and ingests metrics, events, and metadata from Cloud Composer to generate insights via dashboards and charts. You can use Monitoring to understand the performance and health of your Cloud Composer environments and Airflow metrics.

Logging captures the logs that the scheduler and worker containers produce. The logs contain useful system-level and Airflow dependency information to help with debugging. For information about viewing logs, see Viewing Airflow logs.

Note: Cloud Composer Monitoring metrics are enabled automatically. See Pricing: Monitoring details for how to control your costs.

Before you begin

• The following permissions are required to access the logs and metrics for your Cloud Composer environment:

  • Read-only access to logs and metrics: logging.viewer and monitoring.viewer
  • Read-only access to logs, including private logs: logging.privateLogViewer
  • Read/write access to metrics: monitoring.editor

  For more information, see Cloud Composer Access Control.

• To avoid duplicate logging, Cloud Logging for Google Kubernetes Engine is disabled.

• Cloud Logging produces an entry for each status and event that occurs in your Google Cloud project. You can use exclusion filters to reduce the volume of logs, including the logs that Cloud Logging produces for Cloud Composer. Note that excluding logs from jobs.py can cause health check failures and CrashLoopBackOff errors. You must include -jobs.py in exclusion filters to prevent it from being excluded.

• Monitoring cannot plot the count values for workflows and tasks that execute more than once per minute, and currently, does not plot metrics for failed tasks.

Metrics and resource types

You can examine Airflow metrics in Monitoring for workflows (DAGs) and the Celery Executor.

Environment metrics

You might use environment metrics to check the resource usage and health of your Cloud Composer environments.

Environment health

To check the health of your environment, you can use the following health status metric: composer.googleapis.com/environment/healthy

Cloud Composer runs a liveness DAG named airflow_monitoring every 5 minutes and reports environment health as follows:

• When the DAG run finishes successfully, the health status is True.
• If the DAG run fails, the health status is False.
• If the DAG run does not finish, Cloud Composer polls the DAG's state every 5 minutes and reports False if the one-hour timeout occurs.

The liveness DAG is stored in the dags/ folder and visible in the Airflow web UI. The frequency and contents of the liveness DAG are immutable and should not be modified, as changes will not persist.

Database health

To check the health of your database, you can use the following health status metric: composer.googleapis.com/environment/database_health

The Cloud Composer Airflow monitoring pod pings the database every minute and reports health status as True if a SQL connection can be established or False if not.

Database metrics

The following environment metrics are available for the Airflow metadata database used by Cloud Composer environments.

You can use these metrics to monitor the performance and resource usage of your environment's Cloud SQL instance. For example, you might want to upgrade the Cloud SQL machine type of your environment as your environment approaches resource limits. Or you might want to optimize costs related to Airflow metadata database usage by doing a database cleanup, in order to keep storage under a certain threshold.

Database Metric	API
Database CPU usage	composer.googleapis.com/environment/database/cpu/usage_time
Database CPU cores	composer.googleapis.com//environment/database/cpu/reserved_cores
Database CPU utilization	composer.googleapis.com/environment/database/cpu/utilization
Database Memory usage	composer.googleapis.com/environment/database/memory/bytes_used
Database Memory quota	composer.googleapis.com/environment/database/memory/quota
Database Memory utilization	composer.googleapis.com/environment/database/memory/utilization
Database Disk usage	composer.googleapis.com/environment/database/disk/bytes_used
Database Disk quota	composer.googleapis.com/environment/database/database/disk/quota
Database Disk utilization	composer.googleapis.com/environment/database/disk/utilization

Workflow metrics

To help you monitor the efficiency of your workflow runs and identify straggler tasks that cause long latency, the following workflow metrics are available:

Workflow Metric	API
Number of workflow runs	composer.googleapis.com/workflow/run_count
Duration of each workflow run	composer.googleapis.com/workflow/run_duration
Number of task runs	composer.googleapis.com/workflow/task/run_count
Duration of each task	composer.googleapis.com/workflow/task/run_duration

Cloud Monitoring shows only the metrics for completed workflow and task runs (success or failure). No Data displays when there is no workflow activity and for in-progress workflow and task runs.

Celery Executor metrics

The following Celery Executor metrics are available. These metrics can help you determine if there are sufficient worker resources in your environment.

Celery Executor Metric	API
Number of tasks in the queue	composer.googleapis.com/environment/task_queue_length
Number of online Celery workers	composer.googleapis.com/environment/num_celery_workers

The Cloud Monitoring documentation also includes the following information about Cloud Composer metrics and resources:

• For the list of usage metrics that Cloud Composer reports to Cloud Monitoring, see Metrics List.
• For details on the cloud_composer_environment resource type, see Monitored Resource Types in the Cloud Monitoring documentation.

Using Monitoring on Cloud Composer environments

You can access Monitoring from the Monitoring console or using the Monitoring API.

Console

1. After creating a Cloud Composer environment, go to the Monitoring console to view environment monitoring data.
   You can also access the Monitoring console from the Google Cloud Console menu.

   

2. When you first access Monitoring, you are asked to create a Workspace and select a project.

   

After setting up the Workspace, the Monitoring console appears.



3. Select Resources > Metrics Explorer and choose Cloud Composer:
   1. Click in the Find resource type and metric input box to display the resource drop-down list.
   2. Select the Cloud Composer Environment or Cloud Composer Workflow resource. Alternatively, enter cloud_composer_environment or cloud_composer_workflow in the box.

   

4. Click again in the input box and then select a metric from the drop-down list. Hovering over the metric name displays information about the metric.

   

5. Cloud Composer environment information is contained in the workflow_name label: workflow_name=environment.workflow. To view workflow metrics for a specific environment, add a filter:
   1. Create a filter for workflow_name.
   2. Filter the prefix by using the regular expression =~ "your-environment-name.*" with the name of the environment you want to view workflow metrics for. For information about using regular expression in filtering labels, see Filtering.

   

6. Click Save Chart.

   You can also group by metric labels, perform aggregations, and select chart viewing options. See the Monitoring documentation.

API

You can use the Monitoring timeSeries.list API to capture and list metrics defined by a filter expression. Use the Try this API template on the API page to send an API request and display the response.

Expand the Try this API page to see the equivalent cURL and JavaScript for your templated request and the JSON response returned by Monitoring.

Building a custom Monitoring dashboard

You can build a custom Monitoring dashboard that display charts of selected metrics for your Cloud Composer environment.

1. In the Google Cloud Console, select Monitoring, or use the following button:

   Go to Monitoring

2. Select Dashboards > Create Dashboard.

3. In the Untitled Dashboard, click Add Chart and create the chart:

   1. In the Add Chart window, select Cloud Composer Environment as the resource type.
   2. Select one or more metrics and chart properties.
   3. Confirm or type a new chart title and click Save.
   4. Add additional charts to your dashboard, as needed, and Save.

   The following example shows the metric Task Duration. This metric plots the duration of active tasks in your workflows, which is useful for fine-tuning performance.

   

4. To view the dashboard, click the title in the Monitoring Dashboards menu.

   

5. From the dashboard display page, you can view, update, and delete charts.

   

Using Monitoring alerts

You can create alerting policies to monitor the values of metrics and to notify you when those metrics violate a condition.

To create an alerting policy that monitors one or more Cloud Composer Environment or Cloud Composer Workflow resources, follow these steps:

1. In the Google Cloud Console, go to the Monitoring page.

   Go to Monitoring

   If you have never used Cloud Monitoring, then on your first access of Monitoring in the Google Cloud Console, a Workspace is automatically created and your project is associated with that Workspace. Otherwise, if your project isn't associated with a Workspace, then a dialog appears and you can either create a Workspace or add your project to an existing Workspace. We recommend that you create a Workspace. After you make your selection, click Add.

2. In the Monitoring navigation pane, select Alerting, and then select Create policy.
3. Click Add condition:
   
   1. The settings in the Target pane specify the resource and metric to be monitored. In the Find resource type and metric field, select the resource Cloud Composer Environment or Cloud Composer Workflow. Next, select a metric from the metrics list.
   2. The settings in the Configuration pane of the alerting policy determine when the alert is triggered. Most fields in this pane are populated with default values. For more information about the fields in the pane, see Configuration in the Alerting policies documentation.
   3. Click Add.
4. To advance to the notifications section, click Next.
5. Optional: To add notifications to your alerting policy, click Notification channels. In the dialog, select one or more notification channels from the menu, and then click OK.

   If a notification channel that you want to add isn't listed, then click Manage notification channels. You are taken to the Notification channels page in a new browser tab. From this page, you can update the configured notification channels. After you have completed your updates, return to the original tab, click Refresh, and then select the notification channels to add to the alerting policy.

6. To advance to the documentation section, click Next.
7. Click Name and enter a name for the alerting policy.
8. Optional: Click Documentation, and then add any information that you want included in a notification message.
9. Click Save.

For more information, see Alerting policies.

Viewing alerts

When an alert is triggered by a metric threshold condition, Monitoring creates an incident (and a corresponding event).

You can review incidents from the Monitoring Alerting > Incidents page.

If you defined a notification mechanism in the alert policy, such as an email or SMS notification, Monitoring also sends a notification of the incident.

What's next

• Learn how to set up monitoring and alerting for Cloud Composer based on a customized metric.
• Learn about Monitoring Workspaces.
• Learn more about creating Monitoring charts.