Cloud Composer 3 | Cloud Composer 2 | Cloud Composer 1
In Cloud Composer 2, you can specify a custom Cloud Storage bucket as an environment's bucket when you create an environment. Your environment uses this bucket in the same way as the default environment's bucket, and the default environment's bucket is not created.
With a custom bucket, you can:
- Use a bucket configuration that is compliant with your requirements.
- Use a continuously existing bucket for periodic development or testing purposes.
- Reuse an existing bucket from a previous environment and keep CI/CD and data integrations.
- Delete and re-create an environment without the need to transfer data between buckets.
- Pre-populate the custom bucket with required data, such as your DAG files, and then create an environment.
Before you begin
- This feature is supported starting from Cloud Composer version 2.5.0. Earlier versions of Cloud Composer 2 do not support this feature.
- The custom bucket must be located in the same region as the environment. Multi-region and dual-region buckets are not supported.
- The custom bucket and the environment must be in the same project.
- The custom bucket must have the Standard storage class.
- The custom bucket must have no retention policies or retention policy locks applied to it.
- It is not possible to use the same custom bucket with several environments at the same time. Cloud Composer generates an error if the specified bucket is already used by another environment. It is possible to delete an environment and then use the same bucket for another environment.
- If you delete an environment and then use the same bucket for another environment, the new environment does not display Airflow task logs from the previous environment in Airflow UI. The logs remain in the bucket, but the new environment does not associate them with any DAG runs.
Create an environment with a custom bucket
Console
To specify a custom Cloud Storage bucket when you create an environment:
- On the Create environment page, in the Advanced configuration section, click Show advanced configuration.
- Select Custom bucket.
- In the Bucket name field, specify or select a bucket.
gcloud
When you create an environment, the --storage-bucket
argument specifies the
environment's custom bucket.
gcloud composer environments create ENVIRONMENT_NAME \
--location LOCATION \
--storage-bucket CUSTOM_STORAGE_BUCKET
Replace the following:
ENVIRONMENT_NAME
: the name of your environment.LOCATION
: the region where the environment is located.CUSTOM_STORAGE_BUCKET
: the name of a Cloud Storage bucket. You can also specify the bucket URI (with thegs://
prefix).
Example:
gcloud composer environments create example-environment \
--location us-cental1 \
--storage-bucket us-central1-example-bucket
API
When you create an environment, in the Environment > StorageConfig resource, specify a custom bucket for your environment.
{
"name": "projects/PROJECT_ID/locations/LOCATION/environments/ENVIRONMENT_NAME",
"storageConfig": {
"bucket": "CUSTOM_STORAGE_BUCKET"
}
}
Replace the following:
PROJECT_ID
: the Project ID.LOCATION
: the region where the environment is located.ENVIRONMENT_NAME
: the name of your environment.CUSTOM_STORAGE_BUCKET
: the name of a Cloud Storage bucket (without thegs://
prefix).
Example:
// POST https://composer.googleapis.com/v1/{parent=projects/*/locations/*}/environments
{
"name": "projects/example-project/locations/us-central1/environments/example-environment",
"storageConfig": {
"bucket": "us-central1-example-bucket"
}
}
Terraform
When you create an environment, the bucket
field in the
storage_config
block specifies the environment's custom bucket.
resource "google_composer_environment" "example" {
provider = google-beta
name = "ENVIRONMENT_NAME"
region = "LOCATION"
storage_config {
bucket = CUSTOM_STORAGE_BUCKET
}
}
Replace the following:
ENVIRONMENT_NAME
: the name of your environment.LOCATION
: the region where the environment is located.CUSTOM_STORAGE_BUCKET
: the name of a Cloud Storage bucket. You can also specify the bucket URI (with thegs://
prefix).
Example:
resource "google_composer_environment" "example" {
provider = google-beta
name = "example-environment"
region = "us-central1"
storage_config {
bucket = "gs://us-central1-example-bucket"
}
}