Resource location restrictions

This page shows how to configure resource location restrictions to ensure that your data stored by Cloud Composer is kept within the locations you specify.

Regions where resource location restrictions are not supported

Cloud Composer does not support resource location restrictions in the following regions:

  • Seoul (asia-northeast3)
  • Salt Lake City (us-west3)
  • Las Vegas (us-west4)

How location restrictions work

Location restrictions for Cloud Composer are determined based on the organizational policy that is applied to the project where the Cloud Composer environment is created. This policy is assigned either within the project or is inherited from the organization.

With location restrictions enabled, it is not possible to create an environment in a region that is prohibited by the policy. If a region is listed in the Deny list, or is not listed in the Allow list, then you cannot create environments in this region.

Cloud Composer is region-based. To enable the creation of environments, the policy must allow the whole region and not a specific zone within this region. For example, the europe-west3 region must be allowed by the policy in order to create Cloud Composer environments in this region.

Location restrictions are checked at:

  • Environment creation.
  • Environment upgrade, if any additional resources are created during the operation.
  • Environment update, for older environments that do not enforce location restrictions on Cloud Composer dependencies.

In addition to restrictions described above, Cloud Composer does the following:

  • Stores user-customized Airflow images in regional Artifact Registry repositories.
  • If the US multi-region is explicitly prohibited by the policy, Cloud Build use is disabled. In this case, user-customized Airflow images are built in your GKE cluster.

Installing a Python dependency to a private IP environment with resource location restrictions

Location restrictions for your project prohibit the use of some tools. In particular, Cloud Build cannot be used to install Python packages, so direct access to repositories on the public internet is disabled.

To install Python dependencies for a private IP Composer environment when your location restrictions do not allow the US multi-region, use one of the following options:

  • Use a private PyPI repository hosted in your VPC network.
  • Use a proxy server in your VPC network to connect to a PyPI repository on the public internet. Specify the proxy address in the /config/pip/pip.conf file in the Cloud Storage bucket.
  • If your security policy permits access to your VPC network from external IP addresses, you can configure Cloud NAT.
  • Vendor the Python dependencies into the dags folder in the Cloud Storage bucket, to install them as local libraries. This might not be a good option if the dependency tree is large.

Restricting locations for Cloud Composer logs

If your logs contain sensitive data, you might want to redirect Cloud Composer logs to a regional Cloud Storage bucket by using the Logs Router. Doing so prevents your logs from being sent to Cloud Logging. To get support from Cloud Customer Care, you might need to grant Google support engineers access to the Cloud Composer logs stored in Cloud Storage.

gcloud

  1. Create a new Cloud Storage bucket (for example, composer-logs-us-central1-example-environment).

    gsutil mb -l LOCATION gs://ENVIRONMENT_NAME

  2. Create a new log sink.

    gcloud logging sinks create \
composer-log-sink-ENVIRONMENT_NAME \
storage.googleapis.com/BUCKET_NAME \
--log-filter "resource.type=cloud_composer_environment AND \
resource.labels.environment_name=ENVIRONMENT_NAME AND \
resource.labels.location=LOCATION"

  3. Grant the appropriate role to the service account for this bucket (shown in the result of the previous command).

    gcloud projects add-iam-policy-binding PROJECT \
--member="serviceAccount:serviceAccountNumber@gcp-sa-logging.iam.gserviceaccount.com" \
--role='roles/storage.objectCreator' --condition=None

  4. Exclude the logs for your new environment from Monitoring.

    gcloud logging sinks update _Default \
--add-exclusion name=ENVIRONMENT_NAME-exclusion, \
filter="resource.type=cloud_composer_environment AND \
resource.labels.environment_name=ENVIRONMENT_NAME AND \
resource.labels.location=LOCATION"