Private IP Cloud Composer environment

This page provides information about using a private IP Cloud Composer environment.

When you enable private IP, Cloud Composer assigns only private IP (RFC 1918) addresses to the managed Google Kubernetes Engine and Cloud SQL VMs in your environment, resulting in no inbound access to those managed VMs from the public internet.

By default, Cloud Composer workflows do not have outbound internet access. Access to GCP APIs and services is not affected by routing over Google's private network. The following sections describe the features that enable a private IP Cloud Composer environment and configuration options.

VPC-native GKE cluster

When you create an environment, Cloud Composer distributes your environment's resources between a Google-managed tenant project and your customer project.

When you enable a private IP Cloud Composer environment, Cloud Composer creates a VPC-native GKE cluster for your environment in your customer project. VPC-native clusters use Alias IP routing built into the VPC network, enabling the VPC to manage routing for pods. When you use VPC-native clusters, GKE automatically chooses a secondary range. For specific networking requirements, you can also configure the secondary ranges for your GKE pods and GKE services during environment creation.

Private IP Cloud Composer environment

You can enable a private IP Cloud Composer environment when you create an environment. Using private IP means that the GKE and Cloud SQL VMs in your environment are not assigned public IP addresses and communicate only over Google's internal network.

When you create a private IP Cloud Composer environment, the GKE cluster for your environment is configured as a private cluster, and the Cloud SQL instance is configured for private IP. Cloud Composer also creates a peering connection between your customer project's VPC network and your tenant project's VPC network.

With VPC peering and private IP enabled for your environment, the IP traffic between your environment's GKE cluster and Cloud SQL database—over the VPC peering connection—is private, isolating your workflows from the public internet.

This additional layer of security affects how you connect to these resources and how your environment accesses external resources. Using private IP does not affect how you access Cloud Storage or your Airflow webserver over the public IP.

GKE cluster

Using a private GKE cluster enables you to control access to the cluster's master endpoint (cluster nodes do not have public IP addresses).

When you create a private IP Cloud Composer environment, you specify whether or not access to the master endpoint is public and its IP range. The master IP range must not overlap with any subnetwork in your VPC network.

Option Description
Public endpoint access disabled To connect to the cluster, you must connect from a VM in the same region and same VPC network of the private IP Cloud Composer environment.

The VM instance you are connecting from requires the Access scope: Allow full access to all Cloud APIs.

From that VM, you can run Airflow commands by using the gcloud beta composer environments run command.
Public endpoint access enabled, master authorized networks enabled In this configuration, cluster nodes communicate with the master over Google's private network. Nodes can access resources in your Cloud Composer environment and in authorized networks. You can add authorized networks in GKE.

On authorized networks, you can:
  • Run Airflow commands by using the gcloud beta composer environments run command.
  • SSH to the master.
  • Run kubectl commands.

Cloud SQL

Because the Cloud SQL instance does not have a public IP address, the Cloud SQL traffic inside your private IP Cloud Composer environment is not exposed to the public internet. Cloud Composer configures Cloud SQL to accept incoming connections through private service access. You can access the Cloud SQL instance on your VPC network by using its private IP address.

Public internet access for your workflows

Operators and operations that require access to resources on unauthorized networks or on the public internet can fail. For example, the Dataflow Python operation requires a public internet connection to download Apache Beam from pip.

Allowing VMs without external IP addresses and private GKE clusters to connect to the internet requires Cloud NAT.

To use Cloud NAT, you need to create a NAT configuration using Cloud Router for the VPC network and region that your private IP Cloud Composer environment is in.

What's next

Was this page helpful? Let us know how we did:

Send feedback about...

Cloud Composer