Use a custom environment's bucket

Cloud Composer 3 | Cloud Composer 2 | Cloud Composer 1

Note: This page is not yet revised for Cloud Composer 3 and displays content for Cloud Composer 2.

In Cloud Composer 2, you can specify a custom Cloud Storage bucket as an environment's bucket when you create an environment. Your environment uses this bucket in the same way as the default environment's bucket, and the default environment's bucket is not created.

With a custom bucket, you can:

• Use a bucket configuration that is compliant with your requirements.
• Use a continuously existing bucket for periodic development or testing purposes.
• Reuse an existing bucket from a previous environment and keep CI/CD and data integrations.
• Delete and re-create an environment without the need to transfer data between buckets.
• Pre-populate the custom bucket with required data, such as your DAG files, and then create an environment.

Before you begin

• This feature is supported starting from Cloud Composer version 2.5.0. Earlier versions of Cloud Composer 2 do not support this feature.
• The custom bucket must be located in the same region as the environment. Multi-region and dual-region buckets are not supported.
• The custom bucket and the environment must be in the same project.
• The custom bucket must have the Standard storage class.
• The custom bucket must have no retention policies or retention policy locks applied to it.
• It is not possible to use the same custom bucket with several environments at the same time. Cloud Composer generates an error if the specified bucket is already used by another environment. It is possible to delete an environment and then use the same bucket for another environment.
• If you delete an environment and then use the same bucket for another environment, the new environment does not display Airflow task logs from the previous environment in Airflow UI. The logs remain in the bucket, but the new environment does not associate them with any DAG runs.

Warning: Cloud Composer does not alter the encryption settings of a custom bucket and thus does not encrypt it with a CMEK key (even if the project's policies require encryption). Specifically, the custom bucket's encryption is not changed if your environment uses CMEK encryption. Your custom bucket follows policies of the project that it belongs to. You can manually encrypt the environment's bucket with CMEK keys before you create an environment.

Create an environment with a custom bucket

Console

To specify a custom Cloud Storage bucket when you create an environment:

1. On the Create environment page, in the Advanced configuration section, click Show advanced configuration.
2. Select Custom bucket.
3. In the Bucket name field, specify or select a bucket.

gcloud

When you create an environment, the --storage-bucket argument specifies the environment's custom bucket.

gcloud composer environments create ENVIRONMENT_NAME \
    --location LOCATION \
    --storage-bucket CUSTOM_STORAGE_BUCKET

Replace the following:

• ENVIRONMENT_NAME: the name of your environment.
• LOCATION: the region where the environment is located.
• CUSTOM_STORAGE_BUCKET: the name of a Cloud Storage bucket. You can also specify the bucket URI (with the gs:// prefix).

Example:

gcloud composer environments create example-environment \
    --location us-cental1 \
    --storage-bucket us-central1-example-bucket

API

When you create an environment, in the Environment > StorageConfig resource, specify a custom bucket for your environment.

{
  "name": "projects/PROJECT_ID/locations/LOCATION/environments/ENVIRONMENT_NAME",
  "storageConfig": {
    "bucket": "CUSTOM_STORAGE_BUCKET"
  }
}

Replace the following:

• PROJECT_ID: the Project ID.
• LOCATION: the region where the environment is located.
• ENVIRONMENT_NAME: the name of your environment.
• CUSTOM_STORAGE_BUCKET: the name of a Cloud Storage bucket (without the gs:// prefix).

Example:

// POST https://composer.googleapis.com/v1/{parent=projects/*/locations/*}/environments

{
  "name": "projects/example-project/locations/us-central1/environments/example-environment",
  "storageConfig": {
    "bucket": "us-central1-example-bucket"
  }
}

Terraform

When you create an environment, the bucket field in the storage_config block specifies the environment's custom bucket.

Caution: If you change the value of this field for an existing environment, Terraform deletes the existing environment and creates it again.

resource "google_composer_environment" "example" {
  provider = google-beta
  name = "ENVIRONMENT_NAME"
  region = "LOCATION"

  storage_config {

    bucket = CUSTOM_STORAGE_BUCKET

  }
}

Replace the following:

• ENVIRONMENT_NAME: the name of your environment.
• LOCATION: the region where the environment is located.
• CUSTOM_STORAGE_BUCKET: the name of a Cloud Storage bucket. You can also specify the bucket URI (with the gs:// prefix).

Example:

resource "google_composer_environment" "example" {
  provider = google-beta
  name = "example-environment"
  region = "us-central1"

  storage_config {

    bucket = "gs://us-central1-example-bucket"

  }
}

What's next

• About the environment's bucket
• CMEK in Cloud Composer
• Create environments