This page describes how to access and view the Apache Airflow logs for Cloud Composer.
Cloud Composer includes the following Airflow logs:
- Airflow logs: These logs are associated with single DAG tasks. You can view
the task logs in the Cloud Storage
logsfolder associated with the Cloud Composer environment. You can also view the logs in the Airflow web interface.
- Streaming logs: These logs are a superset of the logs in Airflow. To
access streaming logs, use the Cloud Logging Logs Viewer in
the Google Cloud Console,
or use Cloud Monitoring.
You can also use logs-based metrics in Monitoring to set up alerting policies, for example to health check your Cloud Composer environment. Logging and Monitoring quotas apply. To learn about Cloud Logging and Cloud Monitoring for your Cloud Composer environment, see Monitoring environments using Cloud Monitoring.
Logs in Cloud Storage
When you create an environment, Cloud Composer creates a
Cloud Storage bucket and associates the bucket with your environment.
Cloud Composer stores the logs for single DAG tasks in
logs folder in
the bucket. For the bucket name, see Determining the storage bucket name.
Log folder directory structure
logs folder includes folders for each workflow that has run
on the environment. Each workflow folder includes a folder for its DAGs and sub-DAGs.
Each folder contains the log files for each task. The task filename indicates
when the task started.
The following example shows the logs directory structure for an environment.
us-central1-my-environment-60839224-bucket └───dags | │ | | dag_1 | | dag_2 | | ... | └───logs │ └───dag_1 | │ | └───task_1 | | │ datefile_1 | | │ datefile_2 | | │ ... | | | └───task_2 | │ datefile_1 | │ datefile_2 | │ ... | └───dag_2 │ ...
To prevent data loss, the logs saved in Cloud Storage remain in storage after you delete your environment. You must manually delete the logs from Cloud Storage.
Before you begin
The following permission is required to access Airflow logs in
the Cloud Storage bucket for the Cloud Composer
For more information, see Cloud Composer Access control.
Viewing task logs in Cloud Storage
To view the log files for DAG tasks:
To view log files, enter the following command, replacing the VARIABLES with appropriate values:
gsutil ls -r gs://BUCKET/logs
(Optional) To copy a single log or a subfolder, enter the following command, replacing the VARIABLES with appropriate values:
gsutil cp -r gs://BUCKET/logs/PATH_TO_LOG_FILE LOCAL_FILE_OR_DIRECTORY
Viewing streaming logs in the Cloud Console
Cloud Composer produces the following logs:
- airflow: The uncategorized logs that Airflow pods generate.
- airflow-database-init-job: The logs Airflow database initialization job generates.
- airflow-scheduler: The logs the Airflow scheduler generates.
- dag-processor-manager: The logs of the DAG processor manager (the part of the scheduler that processes DAG files).
- airflow-webserver: The logs the Airflow web interface generates.
- airflow-worker: The logs generated as part of workflow and DAG execution.
- cloudaudit.googleapis.com/activity: The logs Admin Activity generates.
- composer-agent: The logs generated as part of create and update environment operations.
- gcs-syncd: The logs generated by the file syncing processes.
- build-log-worker-scheduler: The logs from the local build of the Airflow worker image (during upgrades and Python package installation).
- build-log-webserver: The logs from the build of the Airflow webserver image (during upgrades and python package installation).
- airflow-monitoring: The logs that Airflow monitoring generates.
To view the streaming log files:
Go to the Google Cloud's operations suite Logs Viewer in the Cloud Console.
GO TO THE LOGS VIEWER PAGE
Select the logs you want to see.
You can filter by properties such as log file and level, predefined label, task name, workflow, and execution date. For more information about selecting and filtering logs, see Viewing Logs.To learn about exporting logs, see Exporting with the Logs Viewer.