Monitoring environments with Cloud Monitoring

You can use Cloud Monitoring and Cloud Logging with Cloud Composer.

Monitoring provides visibility into the performance, uptime, and overall health of cloud-powered applications. Cloud Monitoring collects and ingests metrics, events, and metadata from Cloud Composer to generate insights via dashboards and charts. You can use Monitoring to understand the performance and health of your Cloud Composer environments and Airflow metrics.

Logging captures the logs that the scheduler and worker containers produce. The logs contain useful system-level and Airflow dependency information to help with debugging. For information about viewing logs, see Viewing Airflow logs.

Before you begin

  • The following permissions are required to access the logs and metrics for your Cloud Composer environment:

    • Read-only access to logs and metrics: logging.viewer and monitoring.viewer
    • Read-only access to logs, including private logs: logging.privateLogViewer
    • Read/write access to metrics: monitoring.editor

    For more information, see Access control.

  • To avoid duplicate logging, Cloud Logging for Google Kubernetes Engine is disabled.

  • Cloud Logging produces an entry for each status and event that occurs in your Google Cloud project. You can use exclusion filters to reduce the volume of logs, including the logs that Cloud Logging produces for Cloud Composer. Note that excluding logs from can cause health check failures and CrashLoopBackOff errors. You must include in exclusion filters to prevent it from being excluded.

  • Monitoring cannot plot the count values for workflows and tasks that execute more than once per minute, and currently, does not plot metrics for failed tasks.

Metrics and resource types

You can examine Airflow metrics in Monitoring for workflows (DAGs) and the Celery Executor.

Environment metrics

You might use environment metrics to check the resource usage and health of your Cloud Composer environments.

Environment health

To check the health of your environment, you can use the following health status metric:

Cloud Composer runs a liveness DAG named airflow_monitoring every 5 minutes and reports environment health as follows:

  • When the DAG run finishes successfully, the health status is True.
  • If the DAG run fails, the health status is False.
  • If the DAG run does not finish, Cloud Composer polls the DAG's state every 5 minutes and reports False after 20 minutes have passed.

The liveness DAG is stored in the dags/ folder and visible in the Airflow web UI. The frequency and contents of the liveness DAG are immutable and should not be modified, as changes will not persist.

Database health

To check the health of your database, you can use the following health status metric:

The Cloud Composer Airflow monitoring pod pings the database every minute and reports health status as True if a SQL connection can be established or False if not.

Database metrics

The following environment metrics are available for the Airflow metadata database used by Cloud Composer environments.

You can use these metrics to monitor the performance and resource usage of your environment's Cloud SQL instance. For example, you might want to upgrade the Cloud SQL machine type of your environment as your environment approaches resource limits. Or you might want to optimize costs related to Airflow metadata database usage by doing a database cleanup, in order to keep storage under a certain threshold.

Database Metric API
Database CPU usage
Database CPU cores
Database CPU utilization
Database Memory usage
Database Memory quota
Database Memory utilization
Database Disk usage
Database Disk quota
Database Disk utilization

Web server metrics

The following environment metrics are available for the Airflow web server used by Cloud Composer environments.

You can use these metrics to check the performance and resource usage of your environment's Airflow web server instance. For example, you might want to upgrade the web server machine type if it constantly approaches resource limits.

Web server metric API
Web server CPU usage
Web server CPU quota
Web server memory usage
Web server memory quota

Workflow metrics

To help you monitor the efficiency of your workflow runs and identify straggler tasks that cause long latency, the following workflow metrics are available:

Workflow Metric API
Number of workflow runs
Duration of each workflow run
Number of task runs
Duration of each task

Cloud Monitoring shows only the metrics for completed workflow and task runs (success or failure). No Data displays when there is no workflow activity and for in-progress workflow and task runs.

Celery Executor metrics

The following Celery Executor metrics are available. These metrics can help you determine if there are sufficient worker resources in your environment.

Celery Executor Metric API
Number of tasks in the queue
Number of online Celery workers

The Cloud Monitoring documentation also includes the following information about Cloud Composer metrics and resources:

  • For the list of usage metrics that Cloud Composer reports to Cloud Monitoring, see Metrics List.
  • For details on the cloud_composer_environment resource type, see Monitored Resource Types in the Cloud Monitoring documentation.

Using Monitoring on Cloud Composer environments

You can access Monitoring from the Monitoring console or using the Monitoring API.


  1. After creating a Cloud Composer environment, go to the Monitoring console to view environment monitoring data.
  2. Select Resources > Metrics Explorer and choose Cloud Composer:
    1. Click in the Find resource type and metric input box to display the resource drop-down list.
    2. Select the Cloud Composer Environment or Cloud Composer Workflow resource. Alternatively, enter cloud_composer_environment or cloud_composer_workflow in the box.
  3. Click again in the input box and then select a metric from the drop-down list. Hovering over the metric name displays information about the metric.
  4. Cloud Composer environment information is contained in the workflow_name label: workflow_name=environment.workflow. To view workflow metrics for a specific environment, add a filter:
    1. Create a filter for workflow_name.
    2. Filter the prefix by using the regular expression =~ "your-environment-name.*" with the name of the environment you want to view workflow metrics for. For information about using regular expression in filtering labels, see Filtering.
  5. Click Save Chart.

    You can also group by metric labels, perform aggregations, and select chart viewing options. See the Monitoring documentation.


You can use the Monitoring timeSeries.list API to capture and list metrics defined by a filter expression. Use the Try this API template on the API page to send an API request and display the response.

Building a custom Monitoring dashboard

You can build a custom Monitoring dashboard that display charts of selected metrics for your Cloud Composer environment.

  1. In the Google Cloud Console, select Monitoring, or use the following button:

    Go to Monitoring

  2. Select Dashboards > Create Dashboard.

  3. In the Untitled Dashboard, click Add Chart and create the chart:

    1. In the Add Chart window, select Cloud Composer Environment as the resource type.
    2. Select one or more metrics and chart properties.
    3. Confirm or type a new chart title and click Save.
    4. Add additional charts to your dashboard, as needed, and Save.

    The following example shows the metric Task Duration. This metric plots the duration of active tasks in your workflows, which is useful for fine-tuning performance.

  4. To view the dashboard, click the title in the Monitoring Dashboards menu.

  5. From the dashboard display page, you can view, update, and delete charts.

Using Monitoring alerts

You can create alerting policies to monitor the values of metrics and to notify you when those metrics violate a condition.

To create an alerting policy that monitors one or more Cloud Composer Environment or Cloud Composer Workflow resources, follow these steps:

  1. In the Google Cloud Console, go to the Monitoring page.

    Go to Monitoring

  2. In the Monitoring navigation pane, select Alerting, and then select Create policy.
  3. If the button Return to Legacy UI is displayed and if you want to follow these instructions, then click it. You can create an alerting policy by using the Preview interface; however, these instructions are for the Legacy UI.
  4. Click Add condition:
    1. The settings in the Target pane specify the resource and metric to be monitored. In the Find resource type and metric field, select the resource Cloud Composer Environment or Cloud Composer Workflow. Next, select a metric from the metrics list.
    2. The settings in the Configuration pane of the alerting policy determine when the alert is triggered. Most fields in this pane are populated with default values. For more information about the fields in the pane, see Configuration in the Alerting policies documentation.
    3. Click Add.
  5. To advance to the notifications section, click Next.
  6. Optional: To add notifications to your alerting policy, click Notification channels. In the dialog, select one or more notification channels from the menu, and then click OK.

    If a notification channel that you want to add isn't listed, then click Manage notification channels. You are taken to the Notification channels page in a new browser tab. From this page, you can update the configured notification channels. After you have completed your updates, return to the original tab, click Refresh, and then select the notification channels to add to the alerting policy.

  7. To advance to the documentation section, click Next.
  8. Click Name and enter a name for the alerting policy.
  9. Optional: Click Documentation, and then add any information that you want included in a notification message.
  10. Click Save.
For more information, see Alerting policies.

Viewing alerts

When an alert is triggered by a metric threshold condition, Monitoring creates an incident (and a corresponding event).

You can review incidents from the Monitoring Alerting > Incidents page.

If you defined a notification mechanism in the alert policy, such as an email or SMS notification, Monitoring also sends a notification of the incident.

What's next