Monitoring provides visibility into the performance, uptime, and overall health of cloud-powered applications. Cloud Monitoring collects and ingests metrics, events, and metadata from Cloud Composer to generate insights via dashboards and charts. You can use Monitoring to understand the performance and health of your Cloud Composer environments and Airflow metrics.
Logging captures the logs that the scheduler and worker containers produce. The logs contain useful system-level and Airflow dependency information to help with debugging. For information about viewing logs, see Viewing Airflow logs.
Before you begin
The following permissions are required to access the logs and metrics for your Cloud Composer environment:
- Read-only access to logs and metrics:
- Read-only access to logs, including private logs:
- Read/write access to metrics:
For more information, see Cloud Composer Access Control.
- Read-only access to logs and metrics:
To avoid duplicate logging, Cloud Logging for Google Kubernetes Engine is disabled.
Cloud Logging produces an entry for each status and event that occurs in your Google Cloud project. You can use exclusion filters to reduce the volume of logs, including the logs that Cloud Logging produces for Cloud Composer. Note that excluding logs from
jobs.pycan cause health check failures and
CrashLoopBackOfferrors. You must include
-jobs.pyin exclusion filters to prevent it from being excluded.
Monitoring cannot plot the count values for workflows and tasks that execute more than once per minute, and currently, does not plot metrics for failed tasks.
Metrics and resource types
You can examine Airflow metrics in Monitoring for workflows (DAGs) and the Celery Executor.
You might use environment metrics to check the resource usage and health of your Cloud Composer environments.
To check the health of your environment, you can use the following health status metric: composer.googleapis.com/environment/healthy
Cloud Composer runs a liveness DAG named
airflow_monitoring every 5
minutes and reports environment health as follows:
- When the DAG run finishes successfully, the health status is
- If the DAG run fails, the health status is
- If the DAG run does not finish, Cloud Composer polls the DAG's
state every 5 minutes and reports
Falseif the one-hour timeout occurs.
The liveness DAG is stored in the
dags/ folder and visible in
the Airflow web UI. The frequency and contents of the liveness DAG are immutable
and should not be modified, as changes will not persist.
To check the health of your database, you can use the following health status metric: composer.googleapis.com/environment/database_health
The Cloud Composer Airflow monitoring pod pings the database every minute
and reports health status as
True if a SQL connection can be established or
The following environment metrics are available for the Airflow metadata database used by Cloud Composer environments.
You can use these metrics to monitor the performance and resource usage of your environment's Cloud SQL instance. For example, you might want to upgrade the Cloud SQL machine type of your environment as your environment approaches resource limits. Or you might want to optimize costs related to Airflow metadata database usage by doing a database cleanup, in order to keep storage under a certain threshold.
|Database CPU usage||composer.googleapis.com/environment/database/cpu/usage_time|
|Database CPU cores||composer.googleapis.com//environment/database/cpu/reserved_cores|
|Database CPU utilization||composer.googleapis.com/environment/database/cpu/utilization|
|Database Memory usage||composer.googleapis.com/environment/database/memory/bytes_used|
|Database Memory quota||composer.googleapis.com/environment/database/memory/quota|
|Database Memory utilization||composer.googleapis.com/environment/database/memory/utilization|
|Database Disk usage||composer.googleapis.com/environment/database/disk/bytes_used|
|Database Disk quota||composer.googleapis.com/environment/database/database/disk/quota|
|Database Disk utilization||composer.googleapis.com/environment/database/disk/utilization|
To help you monitor the efficiency of your workflow runs and identify straggler tasks that cause long latency, the following workflow metrics are available:
|Number of workflow runs||composer.googleapis.com/workflow/run_count|
|Duration of each workflow run||composer.googleapis.com/workflow/run_duration|
|Number of task runs||composer.googleapis.com/workflow/task/run_count|
|Duration of each task||composer.googleapis.com/workflow/task/run_duration|
Cloud Monitoring shows only the metrics for completed workflow and task runs (success or failure). No Data displays when there is no workflow activity and for in-progress workflow and task runs.
Celery Executor metrics
The following Celery Executor metrics are available. These metrics can help you determine if there are sufficient worker resources in your environment.
|Celery Executor Metric||API|
|Number of tasks in the queue||composer.googleapis.com/environment/task_queue_length|
|Number of online Celery workers||composer.googleapis.com/environment/num_celery_workers|
The Cloud Monitoring documentation also includes the following information about Cloud Composer metrics and resources:
- For the list of usage metrics that Cloud Composer reports to Cloud Monitoring, see Metrics List.
- For details on the
cloud_composer_environmentresource type, see Monitored Resource Types in the Cloud Monitoring documentation.
Using Monitoring on Cloud Composer environments
- After creating a Cloud Composer environment, go to the Monitoring console to view environment monitoring data.
- When you first access Monitoring, you are asked to create a Workspace and select a project.
- Select Resources > Metrics Explorer and choose Cloud Composer:
- Click in the Find resource type and metric input box to display the resource drop-down list.
- Select the Cloud Composer Environment or Cloud Composer Workflow resource. Alternatively, enter cloud_composer_environment or cloud_composer_workflow in the box.
- Click again in the input box and then select a metric from the drop-down list. Hovering over the metric name displays information about the metric.
- Cloud Composer environment information is contained
in the workflow_name label: workflow_name=
environment.workflow. To view workflow metrics for a specific environment, add a filter:
- Create a filter for
- Filter the prefix by using the regular expression
=~ "your-environment-name.*"with the name of the environment you want to view workflow metrics for. For information about using regular expression in filtering labels, see Filtering.
- Create a filter for
- Click Save Chart.
You can also group by metric labels, perform aggregations, and select chart viewing options. See the Monitoring documentation.
After setting up the Workspace, the Monitoring console appears.
You can use the Monitoring
API to capture and list metrics defined by a
Use the Try this API template on the API page to send
an API request and display the response.
Building a custom Monitoring dashboard
You can build a custom Monitoring dashboard that display charts of selected metrics for your Cloud Composer environment.
In the Google Cloud Console, select Monitoring, or use the following button:
Select Dashboards > Create Dashboard.
In the Untitled Dashboard, click Add Chart and create the chart:
- In the Add Chart window, select Cloud Composer Environment as the resource type.
- Select one or more metrics and chart properties.
- Confirm or type a new chart title and click Save.
- Add additional charts to your dashboard, as needed, and Save.
The following example shows the metric
Task Duration. This metric plots the duration of active tasks in your workflows, which is useful for fine-tuning performance.
To view the dashboard, click the title in the Monitoring Dashboards menu.
From the dashboard display page, you can view, update, and delete charts.
Using Monitoring alerts
You can create alerting policies to monitor the values of metrics and to notify you when those metrics violate a condition.
To create an alerting policy that monitors one or more Cloud Composer Environment or Cloud Composer Workflow resources, follow these steps:
- In the Google Cloud Console, go to the Monitoring page.
If you have never used Cloud Monitoring, then on your first access of Monitoring in the Google Cloud Console, a Workspace is automatically created and your project is associated with that Workspace. Otherwise, if your project isn't associated with a Workspace, then a dialog appears and you can either create a Workspace or add your project to an existing Workspace. We recommend that you create a Workspace. After you make your selection, click Add.
- In the Monitoring navigation pane, select notificationsAlerting, and then select Create policy.
- Click Add condition:
- The settings in the Target pane specify the resource and metric to be monitored. In the Find resource type and metric field, select the resource Cloud Composer Environment or Cloud Composer Workflow. Next, select a metric from the metrics list.
- The settings in the Configuration pane of the alerting policy determine when the alert is triggered. Most fields in this pane are populated with default values. For more information about the fields in the pane, see Configuration in the Alerting policies documentation.
- Click Add.
- To advance to the notifications section, click Next.
- Optional: To add notifications to your alerting policy, click
Notification channels. In the dialog, select one or more notification
channels from the menu, and then click OK.
If a notification channel that you want to add isn't listed, then click Manage notification channels. You are taken to the Notification channels page in a new browser tab. From this page, you can update the configured notification channels. After you have completed your updates, return to the original tab, click autorenewRefresh, and then select the notification channels to add to the alerting policy.
- To advance to the documentation section, click Next.
- Click Name and enter a name for the alerting policy.
- Optional: Click Documentation, and then add any information that you want included in a notification message.
- Click Save.
When an alert is triggered by a metric threshold condition, Monitoring creates an incident (and a corresponding event).
You can review incidents from the Monitoring Alerting > Incidents page.
If you defined a notification mechanism in the alert policy, such as an email or SMS notification, Monitoring also sends a notification of the incident.
- Learn how to set up monitoring and alerting for Cloud Composer based on a customized metric.
- Learn about Monitoring Workspaces.
- Learn more about creating Monitoring charts.