Anthos clusters on Azure is generally available as of December 2. See the Release notes for more information.

Create a cluster

This page explains how to create a cluster and node pool in Anthos clusters on Azure on Kubernetes version 1.21.5-gke.2800.

Before you begin

To complete the steps on this page, do the following:

  1. Follow the steps in Configure prerequisites.

  2. Choose if you will run the control plane in multiple zones or a single zone.

  3. Select classless inter-domain routing (CIDR) ranges to provide to your cluster.

Control plane zonal placement

By default, Anthos clusters on Azure places separate control plane replicas in the same subnet across three zones in your selected region. You can choose these zones and subnets.

If you want to use the default control plane replica placement, skip to Select CIDR ranges for your cluster.

Azure NAT gateway and cluster control planes

Each control plane replica also requires connectivity to the Google-hosted management service to operate in a normal state.

If you use Azure NAT gateway to provide outbound connectivity, you must consider how a zonal failure impacts the cluster's control plane. One NAT gateway endpoint is isolated to a single zone or is regional/non-zonal, and this presents a single point of failure.

If you want to place control plane replicas in a single zone, use a single subnet and zone. If you use NAT Gateway for outbound connectivity, make sure that the endpoint is placed in the same zone.

If you want to place replicas in two or three zones, you can pass a list of subnets and zones when you create a cluster. When you pass two subnets and zones, Anthos clusters on Azure places two replicas in the first zone provided. When you pass in three subnets and zones, Anthos clusters on Azure places replicas in each subnet. For more information, see Place replicas in a specific subnet.

For more information on configuring Azure subnets and zones for high availability, see Zone isolation with zonal stacks in the Azure documentation.

Place replicas in a specific subnet

This section is optional.

To control which zones control plane replicas are placed in, you use the --replica-placements flag and pass a list of subnet IDs and zones when you create your cluster. You can use up to three subnets and zones where to place the control plane replicas.

To format the list of subnets, perform the following steps.

  1. Retrieve your Azure subnet IDs with the az command-line tool:

    az network vnet subnet show \
      --resource-group=VNET_RESOURCE_GROUP_NAME --vnet-name=VNET_NAME \
      --name SUBNET_NAME --query "id" -otsv
    

    Replace the following:

    • CLUSTER_RESOURCE_GROUP_NAME: an existing resource group name where you want to run your cluster
    • VNET_RESOURCE_GROUP_NAME: the resource group name that holds your VNet
    • VNET_NAME: your VNet name
    • SUBNET_NAME: your subnet name

    The output is the ID of the subnet. Azure subnet IDs look like the following:

    /subscriptions/SUBSCRIPTION_ID/resourceGroups/RESOURCE_GROUP/providers/Microsoft.Network/virtualNetworks/VNET_NAME/subnets/SUBNET_NAME
    

    Repeat this command for each subnet where you would like to create a control plane replica in. Copy the subnet IDs into a text editor for the following step.

  2. Create a comma-separated list of subnet IDs and Azure availability zones, with a colon separating the subnet and zone. For example, to create control plane replicas in subnet1 in zone 1, subnet2 in zone 2, and subnet3 in zone 3, you use the following string:

    /subscriptions/SUBSCRIPTION_ID/resourceGroups/RESOURCE_GROUP/providers/Microsoft.Network/virtualNetworks/VNET_NAME/subnets/subnet1:1,/subscriptions/SUBSCRIPTION_ID/resourceGroups/RESOURCE_GROUP/providers/Microsoft.Network/virtualNetworks/VNET_NAME/subnets/subnet2:2,/subscriptions/SUBSCRIPTION_ID/resourceGroups/RESOURCE_GROUP/providers/Microsoft.Network/virtualNetworks/VNET_NAME/subnets/subnet3:3
    

    Copy this string and use it as the value for the --replica-placements flag when you create a cluster.

Select CIDR ranges for your cluster

When you create a cluster in Anthos clusters on Azure, you need to provide two RFC 1918 IPv4 address ranges for Pods and Services within your Virtual Private Cloud (VPC). Because these ranges are separate, you can control access separately for internal and external cluster communications. These ranges are in CIDR notation—for example, 10.0.0.0/16.

Selecting ranges

These ranges can be any RFC 1918 range that doesn't conflict with another IP range in your VPC. For example, if you allocate the range 10.0.0.0/24 to another use, you can't use that range.

Pod address range

Kubernetes allocates addresses to Pod objects from the Pod address range. A cluster's Pod range is split into smaller ranges for each node. When a Pod is scheduled on a particular node, Kubernetes assigns a Pod IP address from the node's range.

To calculate the size of the Pod address range, you need to estimate the number of nodes that you want in your cluster and the number of Pods that you want to run on each node.

The following table provides size recommendations for Pod CIDR ranges based on the number of nodes and Pods that you intend to run.

Pod address ranges table

Pod address range Maximum Pod IP addresses Maximum nodes Maximum Pods
/24
Smallest possible Pod address range
256 addresses 1 node 110 Pods
/23 512 addresses 2 nodes 220 Pods
/22 1,024 addresses 4 nodes 440 Pods
/21 2,048 addresses 8 nodes 880 Pods
/20 4,096 addresses 16 nodes 1,760 Pods
/19 8,192 addresses 32 nodes 3,520 Pods
/18 16,384 addresses 64 nodes 7,040 Pods
/17 32,768 addresses 128 nodes 14,080 Pods
/16 65,536 addresses 256 nodes 28,160 Pods
/15 131,072 addresses 512 nodes 56,320 Pods
/14 262,144 addresses 1,024 nodes 112,640 Pods

Service address range

Kubernetes allocates virtual IP addresses for Service objects— for example, load balancers from this address range.

To calculate the size of the Service address range, you need to estimate the number of services that you want in your cluster.

The following table provides size recommendations for Service CIDR ranges based on the number of Services that you intend to run.

Service address ranges table

Service address range Maximum number of Services
/27
Smallest possible Service address range
32 Services
/26 64 Services
/25 128 Services
/24 256 Services
/23 512 Services
/22 1,024 Services
/21 2,048 Services
/20 4,096 Services
/19 8,192 Services
/18 16,384 Services
/17 32,768 Services
/16
Largest possible Service address range
65,536 Services

Create an AzureClient

The management service authenticates to Azure with an AzureClient resource. When you create a client, Google generates a key pair. You upload the public key to Azure Active Directory (Azure AD). If you already have an AzureClient, skip to Create an SSH key pair.

To create an AzureClient, run the following commands:

  1. Set environment variables with your Azure tenant and application IDs:

    export SUBSCRIPTION_ID=$(az account show --query "id" --output tsv)
    export TENANT_ID=$(az account list \
      --query "[?id=='${SUBSCRIPTION_ID}'].{tenantId:tenantId}" --output tsv)
    export APPLICATION_ID=$(az ad app list --all \
      --query "[?displayName=='APPLICATION_NAME'].appId" --output tsv)
    

    Replace APPLICATION_NAME with the Azure AD application name that you used when completing the prerequisites.

  2. Create the client:

    gcloud container azure clients create CLIENT_NAME \
      --location=GOOGLE_CLOUD_LOCATION \
      --tenant-id="${TENANT_ID}" \
      --application-id="${APPLICATION_ID}"
    

    Replace the following:

    • CLIENT_NAME: the name of your AzureClient
    • GOOGLE_CLOUD_LOCATION: the supported Google Cloud region that manages your cluster—for example, us-west1
  3. Get the certificate from an AzureClient and save it to an environment variable:

     CERT=$(gcloud container azure clients get-public-cert --location=GOOGLE_CLOUD_LOCATION \
          CLIENT_NAME)
    

    Replace the following:

    • GOOGLE_CLOUD_LOCATION
    • CLIENT_NAME
  4. Upload the certificate to your application on Azure AD:

     az ad app credential reset --id "${APPLICATION_ID}" --cert "${CERT}" --append
    
  5. After the certificate uploads and before you create a cluster, wait at least one minute for the certificate to be ready to use. If the command fails, you can try again.

After you create a client, create or select an SSH key pair for your cluster.

Create an ssh key pair

When you create a cluster, you need to provide an SSH key pair. If you already have a key pair to use, skip this step.

  1. To create a new key pair, use the ssh-keygen command-line tool:

    ssh-keygen -m PEM -t rsa -b 4096 -f KEY_PATH
    

    Replace KEY_PATH with the path to your new private key.

  2. Store the key in an environment variable:

    SSH_PUBLIC_KEY=$(cat KEY_PATH.pub)
    

    For example, to create a new key pair at ~/.ssh/anthos-multicloud-key.pub and store the public key in an environment variable, run the following command:

    ssh-keygen -m PEM -t rsa -b 4096 -f ~/.ssh/anthos-multicloud-key
    SSH_PUBLIC_KEY=$(cat ~/.ssh/anthos-multicloud-key.pub)
    

After you save the public key to an environment variable, you are ready to create a cluster.

Select your Fleet host project

Fleets are a Google Cloud concept to organize clusters into larger groups. With fleets, you can manage multiple clusters across several clouds and apply consistent policies across them. The Anthos Multi-Cloud API automatically registers your clusters with a Fleet when the cluster is created.

When you create a cluster, you specify a Fleet host project where the cluster will be managed from. Because Anthos clusters on Azure uses the cluster name as the Fleet membership name, you must ensure that your cluster names are unique across your Fleet.

Cross-project registration

If you want to use a Fleet Host project other than the Google Cloud project where the cluster is located, you must apply an additional IAM policy binding to the Multi-Cloud Service Agent service account. This allows the service account to manage Fleets with the Fleet Host Project.

  1. To add the Service Agent to your project, run this command:

    gcloud beta services identity create --service=gkemulticloud.googleapis.com \
      --project=CLUSTER_PROJECT_NUMBER
    

    Replace CLUSTER_PROJECT_NUMBER with your Google Cloud project number.

  2. Assign this binding with the following command:

    gcloud projects add-iam-policy-binding FLEET_PROJECT_ID \
      --member="service-CLUSTER_PROJECT_NUMBER@gcp-sa-gkemulticloud.iam.gserviceaccount.com \
      --role="roles/gkemulticloud.serviceAgent"
    

    Replace the following:

    • FLEET_PROJECT_ID: your Fleet host project's Google Cloud project
    • CLUSTER_PROJECT_NUMBER: your Google Cloud project number

The Multi-Cloud Service Agent account name has the following format: service-CLUSTER_PROJECT_NUMBER@gcp-sa-gkemulticloud.iam.gserviceaccount.com.

You can find your service accounts on the Google Cloud Console Service account page. For more information on how to find your project number, see Identifying projects.

Create a cluster

To create a cluster, run the following commands:

  1. Save your Azure resource group, VNet, and subnet IDs to environment variables:

    CLUSTER_RG_ID=$(az group show --resource-group=CLUSTER_RESOURCE_GROUP_NAME \
      --query "id" -otsv)
    VNET_ID=$(az network vnet show --resource-group=VNET_RESOURCE_GROUP_NAME \
      --name=VNET_NAME --query "id" -otsv)
    SUBNET_ID=$(az network vnet subnet show \
      --resource-group=VNET_RESOURCE_GROUP_NAME --vnet-name=VNET_NAME \
      --name default --query "id" -otsv)
    

    Replace the following:

    • CLUSTER_RESOURCE_GROUP_NAME: an existing resource group name where you want to run your cluster
    • VNET_RESOURCE_GROUP_NAME: the resource group name that holds your VNet
    • VNET_NAME: the name of your VNet
  2. Create a cluster with the gcloud command-line tool:

    gcloud container azure clusters create CLUSTER_NAME \
        --location GOOGLE_CLOUD_LOCATION \
        --fleet-project=FLEET_PROJECT \
        --client CLIENT_NAME \
        --azure-region AZURE_REGION \
        --pod-address-cidr-blocks POD_CIDR \
        --service-address-cidr-blocks SERVICE_CIDR \
        --vm-size VM_SIZE \
        --cluster-version 1.21.5-gke.2800 \
        --ssh-public-key "$SSH_PUBLIC_KEY" \
        --resource-group-id "$CLUSTER_RG_ID" \
        --vnet-id "$VNET_ID" \
        --subnet-id "$SUBNET_ID" # Optional, see following note
    

    Replace the following:

  3. Check the status of your cluster:

    gcloud container azure clusters describe CLUSTER_NAME  --location GOOGLE_CLOUD_LOCATION
    

    Replace the following:

    • CLUSTER_NAME
    • GOOGLE_CLOUD_LOCATION

    The output includes information about the status and configuration of your cluster.

Authorize Cloud Logging / Cloud Monitoring

In order for Anthos clusters on Azure to create and upload system logs and metrics to Google Cloud's operations suite, it must be authorized.

To authorize the Kubernetes workload identity gke-system/gke-telemetry-agent to write logs to Google Cloud Logging, and metrics to Google Cloud Monitoring, run this command:

gcloud projects add-iam-policy-binding GOOGLE_PROJECT_ID \
  --member="serviceAccount:GOOGLE_PROJECT_ID.svc.id.goog[gke-system/gke-telemetry-agent]" \
  --role=roles/gkemulticloud.telemetryWriter

Replace GOOGLE_PROJECT_ID with the cluster's Google Cloud project ID.

This IAM binding grants access for all clusters in the Google Cloud project project to upload logs and metrics. You only need to run it after creating your first cluster for the project.

Create a node pool

Before you create a node pool, you need the following:

  • Permissions to use the az command-line tool to retrieve an Azure subnet ID.
  • Access to the cluster's SSH public key.

To create a node pool, run the following commands:

  1. Save your Azure VNet subnet ID and SSH public key to environment variables:

    SUBNET_ID=$(az network vnet subnet show \
      --resource-group=VNET_RESOURCE_GROUP_NAME --vnet-name=VNET_NAME \
      --name default --query "id" -otsv)
    SSH_PUBLIC_KEY=$(cat KEY_PATH.pub)
    

    Replace the following:

    • VNET_RESOURCE_GROUP_NAME: the resource group name that holds VNet
    • VNET_NAME: the name of your VNet
    • KEY_PATH: the path to your key pair
  2. Create a node pool with the gcloud command-line tool:

    gcloud container azure node-pools create NODE_POOL_NAME \
        --cluster CLUSTER_NAME \
        --location GOOGLE_CLOUD_LOCATION \
        --node-version 1.21.5-gke.2800 \
        --vm-size VM_SIZE \
        --max-pods-per-node 110 \
        --min-nodes MIN_NODES \
        --max-nodes MAX_NODES \
        --ssh-public-key "${SSH_PUBLIC_KEY}" \
        --subnet-id "${SUBNET_ID}"
    

    Replace the following:

    • NODE_POOL_NAME: a unique name for your node pool—for example, node-pool-1
    • CLUSTER_NAME: the name of your Anthos clusters on Azure cluster
    • GOOGLE_CLOUD_LOCATION: the Google Cloud location that manages your cluster
    • VM_SIZE: a supported Azure VM size
    • MIN_NODES: the minimum number of nodes in the node pool—for more information, see Cluster autoscaler
    • MAX_NODES: the maximum number of nodes in the node pool
  3. Check the status of your node pool:

    gcloud container azure node-pools describe NODE_POOL_NAME \
        --cluster CLUSTER_NAME \
        --location GOOGLE_CLOUD_LOCATION
    

    Replace the following:

    • NODE_POOL_NAME: a unique name for your node pool—for example, node-pool-1
    • CLUSTER_NAME: the name of your Anthos clusters on Azure cluster
    • GOOGLE_CLOUD_LOCATION: the Google Cloud location that manages your cluster

    The output includes the status of your node pool, including if it is PROVISIONING or RUNNING.

What's next