Health checking your Cloud Composer environment

Because of Cloud Composer's microservice architecture, the end-to-end liveness monitoring of your Airflow setup involves multiple metrics. This tutorial shows one approach to using Stackdriver's logs based metrics to set up monitoring and alerting based on a customized metric.

Objectives

  • Deploy a simple DAG that runs every few minutes in the Cloud Composer environment
  • Leverage the logs that Stackdriver produces for Cloud Composer by using the logs-based metrics to create a customized metric
  • Set up monitoring and alerting based on the customized metric

Costs

Cloud Composer is a paid product, and you will incur usage costs in this tutorial. For more information, refer to Pricing. You can also use the pricing calculator to generate a cost estimate based on your projected usage. New GCP users might be eligible for a free trial.

Before you begin

  1. Google アカウントにログインします。

    Google アカウントをまだお持ちでない場合は、新しいアカウントを登録します。

  2. GCP プロジェクトを選択または作成します。

    プロジェクト セレクタのページに移動

  3. Google Cloud Platform プロジェクトに対して課金が有効になっていることを確認します。 詳しくは、課金を有効にする方法をご覧ください。

  4. Create a Cloud Composer environment.
  5. Ensure appropriate permissions. The following permissions are required to access the logs and metrics for your Cloud Composer environment:
    • Read-only logging and monitoring: logging.viewer and monitoring.viewer
    • Read-only logging including private logs: logging.privateLogViewer
    • Read-write monitoring: monitoring.editor

    For more information, see Cloud Composer Access Control.

Adding the DAG to your environment

  1. Upload the following DAG to the Cloud Storage bucket for the environment:

    import airflow
    from airflow import DAG
    from airflow.operators.bash_operator import BashOperator
    from datetime import timedelta
    
    default_args = {
        'retries': 1,
        'retry_delay': timedelta(minutes=5),
        'start_date': airflow.utils.dates.days_ago(0)
    }
    
    dag = DAG(
        'liveness',
        default_args=default_args,
        description='liveness monitoring dag',
        schedule_interval=timedelta(minutes=1))
    
    t1 = BashOperator(
        task_id='echo', bash_command='echo test', dag=dag, depends_on_past=False)
    
  2. Wait until Airflow schedules the DAG. You can check the status in the Airflow web interface.

    Airflow web interface

Creating the logs-based counter metric in Stackdriver

  1. Go to the Logs Viewer.

    Go to the Logs Viewer

  2. Make sure the correct project is selected at the top of the page, or use the drop-down menu at the top of the page to select a project.

  3. In the resource field, select Cloud Composer Environment and then select the location and environment name.

  4. In the log type field, select airflow-worker.

  5. In the filter bar, create a filter to show only logs that the liveness DAG emits by entering the following label: label:workflow:liveness.

  6. At the top of the page, click Create Metric. The Metric Editor displays on the right side of the page, and the viewer panel showing your logs displays on the left side.

    Create metric

  7. In the Metric Editor panel, set the following fields:

    1. Name: Choose a name that is unique among logs-based metrics in your project, for example liveness-logs-cnt.
    2. Description: Describe the metric.
    3. Type: Counter.
  8. Click Create Metric. The Logs-based metrics page displays.

Creating the alerting policy

  1. In the Logs-based metrics page, check the metric you want to create a policy for, and in the menu at the right side of the metric's listing, select Create alert from metric. The Create New Alerting Policy page displays.

    Create alert

  2. On the Create New Alerting Policy page, in the Conditions section, click Edit. A pre-populated Create METRIC window displays.

  3. (Optional) Enter a name for the condition.

  4. In the Target section, for Aggregator, choose count.

  5. In the Configuration section, for Threshold, set the value, as needed.

    Alert policy

  6. Click Save. The Create new alerting policy panel displays with your completed condition.

  7. Fill in the Notifications, Documentation, and Name this policy sections of the alerting policy.

  8. Click Save.

Cleaning up

To avoid incurring charges to your Google Cloud Platform account for the resources used in this tutorial:

  1. GCP Console で [プロジェクト] ページに移動します。

    プロジェクト ページに移動

  2. プロジェクト リストで、削除するプロジェクトを選択し、[削除] をクリックします。
  3. ダイアログでプロジェクト ID を入力し、[シャットダウン] をクリックしてプロジェクトを削除します。

What's next

To learn about using a custom dashboard to view metrics for your Cloud Composer environment, see Monitoring environments and workflows in Stackdriver.

このページは役立ちましたか?評価をお願いいたします。

フィードバックを送信...