Configure authorized networks in Cloud Composer

Cloud Composer 3 | Cloud Composer 2 | Cloud Composer 1

This page explains how to configure authorized networks for your environment.

About authorized networks in Cloud Composer

Authorized networks allow you to specify CIDR ranges that can access your environment's cluster control plane using HTTPS.

For example, in Cloud Composer 1, such access to your environment's cluster is required in Private IP environments to run Airflow CLI commands(gcloud composer environments run). Networks from where such requests are originated must be authorized to access the control plane of your environment's cluster.

Authorized networks in Private and Public IP environments

You can specify authorized networks both for Public IP and Private IP environments.

  • In Private IP environments you can configure authorized networks to allow a certain range of IP addresses to access the control plane of your environment's cluster. For example, to allow certain IP ranges to run kubectl and Airflow CLI commands. By default, in Private IP environments you can run these commands only from a VM in the VPC subnetwork of the Private IP environment.

  • In Public IP environments, you can configure authorized networks to restrict the range of IP addresses that can access the control plane of your environment's cluster. By default, in Public IP environments there are no restrictions on IP ranges that can run kubectl and Airflow CLI commands.

Before you begin

Create environments with authorized networks

gcloud

To create an environment with authorized networks, use the --enable-master-authorized-networks argument when you create an environment. Then provide a comma-separated list of CIDR ranges in the --master-authorized-networks argument.

To specify authorized networks for a Private IP environment:

gcloud composer environments create ENVIRONMENT_NAME \
    --location LOCATION \
    --image-version composer-1.20.12-airflow-1.10.15 \
    --enable-ip-alias \
    --enable-private-environment \
    --enable-master-authorized-networks \
    --master-authorized-networks AUTHORIZED_NETWORKS_IP_RANGES

Replace:

  • ENVIRONMENT_NAME with the name of the environment.
  • LOCATION with the region where the environment is located.
  • AUTHORIZED_NETWORKS_IP_RANGES with a comma-separated list of IP address ranges in the CIDR notation. These ranges are added as authorized networks for your environment's cluster.

Example (Private IP environment):

gcloud composer environments create example-environment \
    --location us-central1 \
    --image-version composer-1.20.12-airflow-1.10.15 \
    --enable-ip-alias \
    --enable-private-environment \
    --enable-privately-used-public-ips \
    --enable-master-authorized-networks \
    --master-authorized-networks 192.0.2.0/23,192.0.4.0/23

API

Construct an environments.create API request. In the Environment resource, specify the configuration parameters for an environment with authorized networks.

To specify authorized networks for a Private IP environment:

// POST https://composer.googleapis.com/v1/{parent=projects/*/locations/*}/environments

{
  "name": "ENVIRONMENT_NAME",
  "config": {
    "nodeConfig": {
      "ipAllocationPolicy": {

        "useIpAliases": true,
      }
    },
    "privateEnvironmentConfig": {
      "enablePrivateEnvironment": true,
    },
    "masterAuthorizedNetworksConfig": {
      "enabled": true,
      "cidrBlocks": [
        {
          "displayName": "AUTHORIZED_NETWORK_NAME",
          "cidrBlock": "AUTHORIZED_NETWORK_RANGE"
        }
      ]
    }
  }
}

Replace:

  • ENVIRONMENT_NAME with the name of the environment.
  • AUTHORIZED_NETWORK_NAME with the name for the authorized network IP range. You use this name to identify this block. This field is optional.
  • AUTHORIZED_NETWORK_RANGE with an IP address range in the CIDR notation. This range is added as an authorized networks for your environment's cluster.
  • If you want to use several IP ranges, add extra ranges to cidrBlocks.

Example (Private IP environment):

// POST https://composer.googleapis.com/v1/{parent=projects/*/locations/*}/environments

{
  "name": "example-environment",
  "config": {
    "nodeConfig": {
      "ipAllocationPolicy": {
        "useIpAliases": true,
      }
    },
    "privateEnvironmentConfig": {
      "enablePrivateEnvironment": true,
    },
    "masterAuthorizedNetworksConfig": {
      "enabled": true,
      "cidrBlocks": [
        {
          "displayName": "example_range_1",
          "cidrBlock": "192.0.2.0/23"
        },
        {
          "displayName": "example_range_2",
          "cidrBlock": "192.0.4.0/23"
        }
      ]
    }
  }
}

Terraform

When you create an environment, the master_authorized_networks_config block in the config block controls authorized networks for your environment.

To specify authorized networks for a Private IP environment:

resource "google_composer_environment" "example_environment" {
  provider = google-beta
  name = "ENVIRONMENT_NAME"
  region = "LOCATION"

  config {
    node_config {
      ip_allocation_policy = [{
        use_ip_aliases = true
      }]
    }
    private_environment_config {
      // Private environment parameters
    }

    master_authorized_networks_config {

      enabled = true

      cidr_blocks {
          cidr_block = "AUTHORIZED_NETWORK_RANGE"
          display_name = "AUTHORIZED_NETWORK_NAME"
      }

    }
  }
}

Replace:

  • ENVIRONMENT_NAME with the name of the environment.
  • LOCATION with the region where the environment is located.
  • AUTHORIZED_NETWORK_RANGE with an IP address range in the CIDR notation. This range is added as an authorized networks for your environment's cluster.
  • AUTHORIZED_NETWORK_NAME with the name for the authorized network IP range. You use this name to identify this block.
  • If you want to use several IP ranges, add extra cidr_blocks blocks to to master_authorized_networks_config.

Example (Private IP environment):

resource "google_composer_environment" "example_environment" {
  provider = google-beta
  name = "example-environment"
  region = "us-central1"

  config {
    node_config {

      // Specify your network and subnetwork
      network    = google_compute_network.example_network.id
      subnetwork = google_compute_subnetwork.example_subnet.id

      ip_allocation_policy = [{
        use_ip_aliases = true
      }]
    }

    private_environment_config {
      // Private environment parameters
    }

    master_authorized_networks_config {

      enabled = true

      cidr_blocks {
          cidr_block = "192.0.2.0/23"
          display_name = "example_range_1"
      }

      cidr_blocks {
          cidr_block = "192.0.4.0/23"
          display_name = "example_range_2"
      }
    }
  }
}

Specify authorized networks for an existing environment

You can specify authorized networks for an existing environment.

gcloud

To specify authorized networks, use the --enable-master-authorized-networks argument. Then provide a comma-separated list of CIDR ranges in the --master-authorized-networks argument.

gcloud composer environments update ENVIRONMENT_NAME \
  --location LOCATION \
  --enable-master-authorized-networks \
  --master-authorized-networks AUTHORIZED_NETWORKS_IP_RANGES

Replace:

  • ENVIRONMENT_NAME with the name of the environment.
  • LOCATION with the region where the environment is located.
  • AUTHORIZED_NETWORKS_IP_RANGES with a comma-separated list of IP address ranges in the CIDR notation. These ranges are added as authorized networks for your environment's cluster.

Example:

gcloud composer environments update example-environment \
  --location us-central1 \
  --enable-master-authorized-networks \
  --master-authorized-networks 192.0.2.0/23,192.0.4.0/23

API

  1. Construct an environments.patch API request.

  2. In this request:

    1. In the updateMask parameter, specify the config.softwareConfig.masterAuthorizedNetworksConfig mask.

    2. In the request body, specify CIDR ranges for authorized networks.

"config": {

  "masterAuthorizedNetworksConfig": {
    "enabled": true,
    "cidrBlocks": [
      {
        "displayName": "AUTHORIZED_NETWORK_NAME",
        "cidrBlock": "AUTHORIZED_NETWORK_RANGE"
      }
    ]
  }
}

Replace:

  • AUTHORIZED_NETWORK_NAME with the name for the authorized network IP range. You use this name to identify this block. This field is optional.
  • AUTHORIZED_NETWORK_RANGE with an IP address range in the CIDR notation. This range is added as an authorized network for your environment's cluster.
  • If you want to use several IP ranges, add extra ranges to cidrBlocks.

Example:

// PATCH https://composer.googleapis.com/v1/projects/example-project/
// locations/us-central1/environments/example-environment?updateMask=
// config.softwareConfig.masterAuthorizedNetworksConfig

"config": {
  "masterAuthorizedNetworksConfig": {
    "enabled": true,
    "cidrBlocks": [
      {
        "displayName": "example_range_1",
        "cidrBlock": "192.0.2.0/23"
      },
      {
        "displayName": "example_range_2",
        "cidrBlock": "192.0.4.0/23"
      }
    ]
  }
}

Terraform

The master_authorized_networks_config block in the config block controls authorized networks for your environment.

To add authorized networks for a Private IP environment, add this block to your environment definition:

resource "google_composer_environment" "example_environment" {
  provider = google-beta
  name = "example_environment"
  region = "us-central1"

  config {

    // Other environment parameters

    master_authorized_networks_config {

      enabled = true

      cidr_blocks {
          cidr_block = "AUTHORIZED_NETWORK_RANGE"
          display_name = "AUTHORIZED_NETWORK_NAME"
      }

    }
  }
}

Replace:

  • AUTHORIZED_NETWORK_RANGE with an IP address range in the CIDR notation. This range is added as an authorized networks for your environment's cluster.
  • AUTHORIZED_NETWORK_NAME with the name for the authorized network IP range. You use this name to identify this block.
  • If you want to use several IP ranges, add extra cidr_blocks blocks to to master_authorized_networks_config.

Example:

resource "google_composer_environment" "example_environment" {
  provider = google-beta
  name = "example-environment"
  region = "us-central1"

  config {

    // Other environment parameters

    master_authorized_networks_config {

      enabled = true

      cidr_blocks {
          cidr_block = "192.0.2.0/23"
          display_name = "example_range_1"
      }

      cidr_blocks {
          cidr_block = "192.0.4.0/23"
          display_name = "example_range_2"
      }
    }
  }
}

Disable authorized networks

You can disable authorized networks for an existing environment:

  • For Private IP environments, this removes access for ranges that were previously added as authorized network.

  • For Public IP environments, this removes previously configured restrictions.

gcloud

To disable authorized networks, use the --disable-master-authorized-networks argument.

gcloud composer environments update ENVIRONMENT_NAME \
  --location LOCATION \
  --disable-master-authorized-networks

Replace:

  • ENVIRONMENT_NAME with the name of the environment.
  • LOCATION with the region where the environment is located.

Example:

gcloud composer environments update example-environment \
  --location us-central1 \
  --disable-master-authorized-networks

API

  1. Construct an environments.patch API request.

  2. In this request:

    1. In the updateMask parameter, specify the config.softwareConfig.masterAuthorizedNetworksConfig mask.

    2. In the request body, specify false in the enabled field.

"config": {

  "masterAuthorizedNetworksConfig": {
    "enabled": false
  }
}

Example:

// PATCH https://composer.googleapis.com/v1/projects/example-project/
// locations/us-central1/environments/example-environment?updateMask=
// config.softwareConfig.masterAuthorizedNetworksConfig

"config": {
  "masterAuthorizedNetworksConfig": {
    "enabled": false,
  }
}

Terraform

The master_authorized_networks_config block in the config block controls authorized networks for your environment.

To disable authorized networks, set the enabled field in the master_authorized_networks_config block to false.

Example:

resource "google_composer_environment" "example_environment" {
  provider = google-beta
  name = "example_environment"
  region = "us-central1"

  config {

    // Other environment parameters

    master_authorized_networks_config {
      enabled = false
    }
  }
}

What's next