Health checking your Cloud Composer environment

---

Because of Cloud Composer's microservice architecture, the end-to-end liveness monitoring of your Airflow setup involves multiple metrics. This tutorial shows one approach to using Google Cloud's operations suite's logs based metrics to set up monitoring and alerting based on a customized metric.

Objectives

• Deploy a simple DAG that runs every few minutes in the Cloud Composer environment
• Leverage the logs that Google Cloud's operations suite produces for Cloud Composer by using the logs-based metrics to create a customized metric
• Set up monitoring and alerting based on the customized metric

Costs

In this document, you use the following billable components of Google Cloud:

• Cloud Composer

To generate a cost estimate based on your projected usage, use the pricing calculator. New Google Cloud users might be eligible for a free trial.

Before you begin

1. Sign in to your Google Cloud account. If you're new to Google Cloud, create an account to evaluate how our products perform in real-world scenarios. New customers also get $300 in free credits to run, test, and deploy workloads.

2. In the Google Cloud console, on the project selector page, select or create a Google Cloud project.

   Note: If you don't plan to keep the resources that you create in this procedure, create a project instead of selecting an existing project. After you finish these steps, you can delete the project, removing all resources associated with the project.

   Go to project selector

3. Make sure that billing is enabled for your Google Cloud project. Learn how to check if billing is enabled on a project.

4. In the Google Cloud console, on the project selector page, select or create a Google Cloud project.

   Note: If you don't plan to keep the resources that you create in this procedure, create a project instead of selecting an existing project. After you finish these steps, you can delete the project, removing all resources associated with the project.

   Go to project selector

5. Make sure that billing is enabled for your Google Cloud project. Learn how to check if billing is enabled on a project.

6. Create a Cloud Composer environment.
7. Ensure appropriate permissions. The following permissions are required to access the logs and metrics for your Cloud Composer environment:
   • Read-only logging and monitoring: logging.viewer and monitoring.viewer
   • Read-only logging including private logs: logging.privateLogViewer
   • Read-write monitoring: monitoring.editor

   For more information, see Access control.

Adding the DAG to your environment

1. Upload the following DAG to the Cloud Storage bucket for the environment:

   import airflow
   from airflow import DAG
   from airflow.operators.bash_operator import BashOperator
   from datetime import timedelta
   
   default_args = {
       'retries': 1,
       'retry_delay': timedelta(minutes=5),
       'start_date': airflow.utils.dates.days_ago(0)
   }
   
   dag = DAG(
       'liveness',
       default_args=default_args,
       description='liveness monitoring dag',
       schedule_interval=timedelta(minutes=1))
   
   t1 = BashOperator(
       task_id='echo', bash_command='echo test', dag=dag, depends_on_past=False)
   

2. Wait until Airflow schedules the DAG. You can check the status in the Airflow web interface.

   

Creating the logs-based counter metric in Google Cloud's operations suite

1. Go to the Logs Explorer.

   Go to the Logs Explorer

2. Make sure the correct project is selected at the top of the page, or use the drop-down menu at the top of the page to select a project.

3. In the resource field, select Cloud Composer Environment and then select the location and environment name.

4. In the log type field, select airflow-worker.

5. In the filter bar, create a filter to show only logs that the liveness DAG emits by entering the following label: label:workflow:liveness.

6. At the top of the page, click Create Metric. The Metric Editor displays on the right side of the page, and the viewer panel showing your logs displays on the left side.

   

7. In the Metric Editor panel, set the following fields:

   1. Name: Choose a name that is unique among logs-based metrics in your project, for example liveness-logs-cnt.
   2. Description: Describe the metric.
   3. Type: Counter.

8. Click Create Metric. The Logs-based metrics page displays.

Creating the alerting policy

1. In the Logs-based metrics page, check the metric you want to create a policy for, and in the menu at the right side of the metric's listing, select Create alert from metric. The Create New Alerting Policy page displays.

   

2. On the Create New Alerting Policy page, in the Conditions section, click Edit. A pre-populated Create METRIC window displays.

3. (Optional) Enter a name for the condition.

4. In the Target section, for Aggregator, choose count.

5. In the Configuration section, for Threshold, set the value, as needed.

   

6. Click Save. The Create new alerting policy panel displays with your completed condition.

7. Fill in the Notifications, Documentation, and Name this policy sections of the alerting policy.

8. Click Save.

Clean up

To avoid incurring charges to your Google Cloud account for the resources used in this tutorial, either delete the project that contains the resources, or keep the project and delete the individual resources.

Caution: Deleting a project has the following effects:

• Everything in the project is deleted. If you used an existing project for this tutorial, when you delete it, you also delete any other work you've done in the project.
• Custom project IDs are lost. When you created this project, you might have created a custom project ID that you want to use in the future. To preserve the URLs that use the project ID, such as an appspot.com URL, delete selected resources inside the project instead of deleting the whole project.

If you plan to explore multiple tutorials and quickstarts, reusing projects can help you avoid exceeding project quota limits.

1. In the Google Cloud console, go to the Manage resources page.

   Go to Manage resources

2. In the project list, select the project that you want to delete, and then click Delete.
3. In the dialog, type the project ID, and then click Shut down to delete the project.

What's next

To learn about using a custom dashboard to view metrics for your Cloud Composer environment, see Monitoring environments and workflows in Google Cloud's operations suite.