This page shows how to configure resource location restrictions to ensure that your data stored by Cloud Composer is kept within the locations you specify.
How location restrictions work
Location restrictions for Cloud Composer are determined based on the organizational policy that is applied to the project where the Cloud Composer environment is created. This policy is assigned either within the project or is inherited from the organization.
With location restrictions enabled, it is not possible to create an environment in a region that is prohibited by the policy. If a region is listed in the Deny list, or is not listed in the Allow list, then you cannot create environments in this region.
Cloud Composer is region-based. To enable the creation of
environments, the policy must allow the whole region and not a specific zone within this region. For example, the
europe-west3 region must be allowed by the policy in order to create Cloud Composer environments in this region.
Location restrictions are checked at:
- Environment creation.
- Environment upgrade, if any additional resources are created during the operation.
- Environment update, for older environments that do not enforce location restrictions on Cloud Composer dependencies.
In addition to restrictions described above, Cloud Composer does the following:
- Stores user-customized Airflow images in regional Artifact Registry repositories.
- If the
USmulti-region is explicitly prohibited by the policy, Cloud Build use is disabled. In this case, user-customized Airflow images are built in your GKE cluster.
Installing a Python dependency to a private IP environment with resource location restrictions
Location restrictions for your project prohibit the use of some tools. In particular, Cloud Build cannot be used to install Python packages, so direct access to repositories on the public internet is disabled.
To install Python dependencies for a private IP Composer environment when your
location restrictions do not allow
US multi-region, use one of the following options:
- Use a private PyPI repository hosted in your VPC network.
- Use a proxy server
in your VPC network to connect to a PyPI repository on the public internet. Specify the proxy address in the
/config/pip/pip.conffile in the Cloud Storage bucket.
- If your security policy permits access to your VPC network from external IP addresses, you can configure Cloud NAT.
- Vendor the Python dependencies into the
dagsfolder in the Cloud Storage bucket, to install them as local libraries. This might not be a good option if the dependency tree is large.
Restricting locations for Cloud Composer logs
If your logs contain sensitive data, you might want to redirect Cloud Composer logs to a regional Cloud Storage bucket by using the Logs Router. Doing so prevents your logs from being sent to Cloud Logging. To get support from Cloud Customer Care, you might need to grant Google support engineers access to the Cloud Composer logs stored in Cloud Storage.
Create a new Cloud Storage bucket (for example,
gsutil mb -l LOCATION gs://ENVIRONMENT_NAME
Create a new log sink.
gcloud logging sinks create \ composer-log-sink-ENVIRONMENT_NAME \ storage.googleapis.com/BUCKET_NAME \ --log-filter "resource.type=cloud_composer_environment AND \ resource.labels.environment_name=ENVIRONMENT_NAME AND \ resource.labels.location=LOCATION"
Grant the appropriate role to the service account for this bucket (shown in the result of the previous command).
gcloud projects add-iam-policy-binding PROJECT \ --member="serviceAccount:serviceAccountNumber@gcp-sa-logging.iam.gserviceaccount.com" \ --role='roles/storage.objectCreator' --condition=None
Exclude the logs for your new environment from Monitoring.
gcloud logging sinks update _Default \ --add-exclusion name=ENVIRONMENT_NAME-exclusion, \ filter="resource.type=cloud_composer_environment AND \ resource.labels.environment_name=ENVIRONMENT_NAME AND \ resource.labels.location=LOCATION"