Configuring a private IP Cloud Composer environment

This page provides information configuring a private IP Cloud Composer environment.

When you enable private IP, Cloud Composer assigns only private IP (RFC 1918) addresses to the managed Google Kubernetes Engine and Cloud SQL VMs in your environment.

For information about connecting to resources in your environment, see Private IP.

Before you begin

Access control

Network requirements

Verify that your project's VPC network meets the following requirements:

  • Make sure there are no private IP block conflicts. If your VPC network and its established VPC peers have overlapping IP blocks with the VPC network in the Google-managed tenant project, Cloud Composer cannot create your environment.
  • Ensure there are sufficient secondary IP ranges for the Cloud Composer GKE pods and services. GKE searches for secondary IP ranges for IP Aliasing. If GKE cannot find a range, Cloud Composer cannot create your environment.
  • In the VPC network for your private IP Cloud Composer environment, make sure that your existing subnets and secondary ranges are not using the IP range 172.31.255.0/24. The private IP Cloud Composer environment uses this IP range for network peerings. If there is an IP collision, environment creation might fail.
  • Ensure that the number of VPC peering connections in your VPC network does not exceed 25. Consider the following:
    • For each private IP Cloud Composer environment, Cloud Composer creates one peering connection for the tenant project network.
    • The private GKE cluster creates another VPC peering connection in your VPC network.
    • The maximum number of private IP Cloud Composer environments that Cloud Composer can support is 12.
    • The maximum number of private IP Cloud Composer environments you can create depends on the number of VPC peering connections you are using in your VPC network.
  • Ensure that the number of secondary ranges in your subnetwork does not exceed 5. Consider the following:
    • The GKE cluster for your private IP Cloud Composer environment creates two secondary ranges in the subnetwork. You can create multiple subnetworks in the same region for the same VPC network.
    • The maximum number of supported secondary ranges is 5. Because each private IP Cloud Composer environment requires two secondary ranges for the Cloud Composer GKE pods and services, each subnetwork supports up to two private IP Cloud Composer environments.

VPC Native configuration

Cloud Composer supports VPC-native GKE clusters in your environment.

During environment creation, you can enable VPC Native (using alias IP) and configure networking, such as IP allocation, without enabling private IP.

Because a VPC native cluster is required for Airflow tasks to communicate with other VMs that are reachable through private IPs, you must also enable VPC Native to configure a private IP environment.

Example private IP and secondary range configuration are provided in the following section.

Configuring a private IP Cloud Composer environment and secondary ranges

Console

  1. Open the Create Environment page in the Google Cloud Platform Console.

    Open the Create Environment page

  2. Configure your environment.
  3. Check Enable VPC-native (using alias IP).
  4. Enter the network configuration for your environment's GKE cluster, including the IP Aliasing secondary ranges, as needed.
  5. To enable a private IP Cloud Composer environment, check Private IP environment.
  6. To enable a public endpoint for the GKE cluster in your environment, check Access GKE master using its external IP address.

    Your cluster is configured as Public endpoint access disabled.

  7. In the GKE master IP range, enter a private RFC 1918 range for the master's VPC.
  8. When you have finished configuring your environment, click Create.

gcloud

Configure your environment, including the following arguments for a private IP Cloud Composer environment, as needed:

gcloud composer environments create ENVIRONMENT_NAME \
    --location LOCATION --enable-private-environment \
    OTHER_ARGUMENTS

The following parameters are required unless specified:

  • ENVIRONMENT_NAME is the name of the environment. Must match the pattern: ^[a-z](?:[-0-9a-z]{0,62}[0-9a-z])?$
  • LOCATION is the Compute Engine region where the environment is located.
  • --enable-private-environment to enable a private IP Cloud Composer environment.
  • --enable-ip-alias to enable VPC Native using alias IP addresses. This parameter is required when using --enable-private environment or when configuring secondary ranges for pods and services:
    • --cluster-secondary-range-name or --cluster-ipv4-cidr to configure the configure the secondary range for pods.
    • --services-secondary-range-name or--services-ipv4-cidr to configure the secondary range for services.
  • master-ipv4-cidr to specify a private RFC 1918 range for the master's VPC.
  • (Optional) enable-private-endpoint to enable a public endpoint for the GKE cluster in your environment. Your cluster is configured as Public endpoint access disabled.

The following example creates an environment running the latest supported Cloud Composer image version in the us-central1 region that uses the n1-standard-2 machine type with a beta environment label.

Private IP is enabled for the environment and 192.0.2.0/28 is the IP range for the GKE master's VPC.

gcloud beta composer environments create test-environment \
    --location us-central1 \
    --zone us-central1-f \
    --machine-type n1-standard-2 \
    --image-version composer-latest-airflow-x.y.z \
    --labels env=beta  \
    --enable-ip-alias \
    --enable-private-environment \
    --enable-private-endpoint \
    --master-ipv4-cidr 192.0.2.0/28 

API

To create a new Cloud Composer environment with the Cloud Composer REST API, construct an environments.create API request, filling in the Environment resource with your configuration information.

For example:

POST https://composer.googleapis.com/v1beta1/projects/test-project/locations/us-central1/environments
      {
  "config": {
    "nodeConfig": {
      "diskSizeGb": 100,
      "ipAllocationPolicy": {
        "useIpAliases": true,
        "clusterIpv4CidrBlock": "10.180.0.0/20",
        "servicesIpv4CidrBlock": "10.182.0.0/20"
      },
      "network": "projects/test-project/global/networks/default",
      "subnetwork": "projects/test-project/regions/us-central1/subnetworks/default"
    },
    "privateEnvironmentConfig": {
      "enablePrivateEnvironment": true
    },
    "softwareConfig": {
      "imageVersion": "composer-1.5.2-airflow-1.10.1"
    }
  },
  "name": "projects/test-project/locations/us-central1/environments/vpcnative-test"
}
Was this page helpful? Let us know how we did:

Send feedback about...

Cloud Composer