Monitoring Cloud Composer Environments in Stackdriver

This page provides information about and an example of using Stackdriver to monitor Cloud Composer environments.

Stackdriver is a free monitoring and logging service that enables you to view logs and metrics about your environment. For more information, see the Stackdriver documentation.

Viewing metrics

Go to Stackdriver in the Google Cloud Platform Console to view the Stackdriver monitoring dashboards or to define Stackdriver alerts. You can also use the Stackdriver monitoring API to query and view metrics.

Metrics and resource types

The Stackdriver documentation includes the following information about Cloud Composer metrics and resources:

For the usage metrics that Cloud Composer reports to Stackdriver, see Metrics List.
For the details for the cloud_composer_environment resource type, see Monitored Resource Types in the Stackdriver documentation.

Health checking your Cloud Composer environment

Because of Cloud Composer's microservice architecture, the end-to-end liveness monitoring of your Airflow setup involves multiple metrics.

The example in this section shows one approach to monitoring your entire Airflow setup using Stackdriver's logs based metrics. The approach involves:

Deploying a simple DAG that runs every few minutes in the Cloud Composer environment.
Leveraging the logs that Stackdriver produces for Cloud Composer by using the logs-based metrics to create a customized metric.
Setting up monitoring and alerting based on the customized metric.

Step 1. Add the DAG to your environment.

Upload the following DAG to the Cloud Storage bucket for the environment:

import airflow
from airflow import DAG
from airflow.operators.bash_operator import BashOperator
from datetime import timedelta
default_args = {
    'retries': 1,
    'retry_delay': timedelta(minutes=5),
    'start_date': airflow.utils.dates.days_ago(0)
}
dag = DAG(
    'liveness',
    default_args=default_args,
    description='liveness monitoring dag',
    schedule_interval=timedelta(minutes=1))
t1 = BashOperator(
    task_id='echo', bash_command='echo test', dag=dag, depends_on_past=False)

Wait until Airflow schedules the DAG. You can check the status in the Airflow web interface.

Airflow web interface

Step 2. Create the logs-based counter metric.

Go to the Logs Viewer.

Go to the Logs Viewer
Make sure the correct project is selected at the top of the page, or use the drop-down menu at the top of the page to select a project.
In the resource field, select Cloud Composer Environment and then select the location and environment name.
In the log type field, select airflow-worker.
In the filter bar, create a filter to show only logs that the liveness DAG emits by entering following label: label:workflow:liveness.
At the top of the page, click Create Metric. The Metric Editor displays on the right side of the page, and the viewer panel showing your logs displays on the left side.
In the Metric Editor panel, set the following fields:
1. Name: Choose a name that is unique among logs-based metrics in your project.
2. Description: Describe the metric.
3. Type: Counter.
Click Create Metric. The Logs-based metrics page displays.

Step 3: Create the alerting policy.

In the Logs-based metrics page, check the metric you want to create a policy for, and in the menu at the right side of the metric's listing, select Create alert from metric. A pre-populated Add Metric Threshold Condition window displays in Stackdriver Logging.
In the Add Metric Threshold Condition window, set the Threshold value and make any other adjustments, as needed.
Click Save Condition, which displays the Create new alerting policy panel with your completed condition. You can add additional conditions, as needed.
Fill in the Notifications, Documentation, and Name sections of the alerting policy.
Click Save Policy.