Adding and managing node pools

Standard

---

This page shows you how to add and perform operations on node pools running your Google Kubernetes Engine (GKE) clusters. To learn about how node pools work, refer to Node pools.

Before you begin

Before you start, make sure you have performed the following tasks:

• Ensure that you have enabled the Google Kubernetes Engine API.
Enable Google Kubernetes Engine API
• Ensure that you have installed the Cloud SDK.

Set up default gcloud settings using one of the following methods:

• Using gcloud init, if you want to be walked through setting defaults.
• Using gcloud config, to individually set your project ID, zone, and region.

Using gcloud init

If you receive the error One of [--zone, --region] must be supplied: Please specify location, complete this section.

1. Run gcloud init and follow the directions:

   gcloud init

   If you are using SSH on a remote server, use the --console-only flag to prevent the command from launching a browser:

   gcloud init --console-only

2. Follow the instructions to authorize gcloud to use your Google Cloud account.
3. Create a new configuration or select an existing one.
4. Choose a Google Cloud project.
5. Choose a default Compute Engine zone for zonal clusters or a region for regional or Autopilot clusters.

Note: You can override these default settings in gcloud commands using the --project, --zone, and --region flags.

Using gcloud config

• Set your default project ID:

  gcloud config set project PROJECT_ID

• If you are working with zonal clusters, set your default compute zone:

  gcloud config set compute/zone COMPUTE_ZONE

• If you are working with Autopilot or regional clusters, set your default compute region:

  gcloud config set compute/region COMPUTE_REGION

• Update gcloud to the latest version:

  gcloud components update

Note: You can override these default settings in gcloud commands using the --project, --zone, and --region flags.

Note: OS Login is not supported in public GKE clusters. OS Login is supported in private GKE clusters that run node pool versions 1.20.5 or later. OS Login is explicitly disabled on GKE nodes, even if it is enabled on the Google Cloud project.

Adding a node pool

gcloud

To create a node pool, run the gcloud container node-pools create command:

gcloud container node-pools create POOL_NAME --cluster CLUSTER_NAME

Additional operational flags

Below are some optional flags that you can specify:

• --machine-type: The Compute Engine machine type to use for instances in the node pool. You can specify a predefined machine type or a custom machine type. If unspecified, the default machine type is e2-medium. Refer to gcloud container node-pools create --machine-type for information on specifying machine types.
• --num-nodes: Specifies the number of nodes to create in the node pool. Default is 3. Your project must have enough quota available for the number of nodes, as well as sufficient route quota.
• --disk-size: Size in GB for node VM boot disks. Defaults to 100GB.

• --image-type: The image type to use for the node pool. Image type specifies the base OS to run on nodes in the node pool. If an image type is specified, that is assigned to the node pool and all future upgrades use the specified image type. If it is not specified, the server chooses the default image type.

  The default image type and the list of valid image types are available using the following command:

  gcloud container get-server-config

• --min-cpu-platform: The baseline minimum CPU platform to run on the nodes.

• --node-locations=[ZONE,ZONE,...]: Enables you to specify zones for your node pools independently of setting the zone for a cluster. Multiple locations can be specified, separated by commas and you can use this flag for single zone, multi-zone, and regional clusters. You can also use this flag to change the zones of your node pool after you've created your cluster using node-pools update --node-locations=[zones] .

• --scopes=SCOPE,[SCOPE,...]: Specifies scopes for the node instances. For example:

  gcloud container node-pools create example-cluster \
    --scopes https://www.googleapis.com/auth/devstorage.read_only

  gcloud container node-pools create example-cluster \
    --scopes bigquery

  Multiple comma-delimited scopes can be specified. default scopes are assigned to all instances. For the list of default scopes, refer to the gcloud container node-pools create documentation.

  SCOPE can be either the full URI of the scope or an alias.

  Available scopes

  The following scopes are available:

  Alias	URI
  bigquery	https://www.googleapis.com/auth/bigquery
  cloud-platform	https://www.googleapis.com/auth/cloud-platform
  compute-ro	https://www.googleapis.com/auth/compute.readonly
  compute-rw	https://www.googleapis.com/auth/compute
  datastore	https://www.googleapis.com/auth/datastore
  logging-write	https://www.googleapis.com/auth/logging.write
  monitoring	https://www.googleapis.com/auth/monitoring
  monitoring-write	https://www.googleapis.com/auth/monitoring.write
  service-control	https://www.googleapis.com/auth/servicecontrol
  service-management	https://www.googleapis.com/auth/service.management.readonly
  sql	https://www.googleapis.com/discovery/v1/apis/sqladmin/v1beta4/rest
  sql-admin	https://www.googleapis.com/auth/sqlservice.admin
  storage-full	https://www.googleapis.com/auth/devstorage.full_control
  storage-ro	https://www.googleapis.com/auth/devstorage.read_only
  storage-rw	https://www.googleapis.com/auth/devstorage.read_write
  taskqueue	https://www.googleapis.com/auth/taskqueue
  useraccounts-ro	https://www.googleapis.com/auth/cloud.useraccounts.readonly
  useraccounts-rw	https://www.googleapis.com/auth/cloud.useraccounts
  userinfo-email	https://www.googleapis.com/auth/userinfo.email

For the full list of options, refer to the gcloud container node-pools create documentation.

A successful node-pools create request returns the node pool information:

Creating node pool example-pool...done.
Created [https://container.googleapis.com/v1/projects/kubernetes-engine-docs/zones/us-central1-f/clusters/example-cluster/nodePools/example-pool].
NAME          MACHINE_TYPE   DISK_SIZE_GB  NODE_VERSION
example-pool  e2-medium      100           1.2.4

The node pool is created. Occasionally, the node pool is created successfully but the gcloud command times out instead of reporting the status from the server. To check the status of all node pools, including ones not yet fully provisioned, use the following command:

gcloud container node-pools list --cluster CLUSTER_NAME

Console

To add a node pool to an existing cluster, perform the following steps:

1. Go to the Google Kubernetes Engine page in Cloud Console.

   Go to Google Kubernetes Engine

2. Beside the cluster you want to edit, click Actions, then click Edit.

3. Click Add Node Pool.

4. Configure your node pool as desired.

5. Click Create to add the node pool.

Viewing node pools in a cluster

gcloud

To list all the node pools of a cluster, run the gcloud container node-pools list command:

gcloud container node-pools list --cluster CLUSTER_NAME

To view details about a specific node pool, run the gcloud container node-pools describe command:

gcloud container node-pools describe POOL_NAME \
    --cluster CLUSTER_NAME

Replace the following:

• CLUSTER_NAME: the name of the cluster.
• POOL_NAME: the name of the node pool to view.

Console

To view node pools for a cluster, perform the following steps:

1. Go to the Google Kubernetes Engine page in Cloud Console.

   Go to Google Kubernetes Engine

2. In the cluster list, click the name of the cluster.

3. Click the Nodes tab.

4. Under Node Pools, click the name of the node pool you want to view.

Resizing a node pool

gcloud

To resize a cluster's node pools, run the gcloud container clusters resize command:

gcloud container clusters resize CLUSTER_NAME --node-pool POOL_NAME \
    --num-nodes NUM_NODES

Replace the following:

• CLUSTER_NAME: the name of the cluster to resize.
• POOL_NAME: the name of the node pool to resize.
• NUM_NODES: the number of nodes in the pool in a zonal cluster. If you use multi-zonal or regional clusters, NUM_NODES is the number of nodes for each zone the node pools is in.

Repeat this command for each node pool. If your cluster has only one node pool, omit the --node-pool flag.

Console

To resize a cluster's node pools, perform the following steps:

1. Go to the Google Kubernetes Engine page in Cloud Console.

   Go to Google Kubernetes Engine

2. Beside the cluster you want to edit, click Actions, then click Edit.

3. Click the Nodes tab.

4. In the Node Pools section, click the name of the node pool that you want to resize.

5. Click Resize.

6. In the Number of nodes field, enter how many nodes that you want in the node pool, and then click Resize.

7. Repeat for each node pool as needed.

Upgrading a node pool

By default, a cluster's nodes have auto-upgrade enabled, and it is recommended that you do not disable it.

When a node pool is upgraded, you can configure surge upgrade settings to control how many nodes GKE upgrades at once as well as how disruptive the upgrade is to workloads. By default, GKE upgrades one node at a time.

While a node is being upgraded, GKE stops scheduling new Pods onto it, and attempts to schedule its running Pods onto other nodes. This is similar to other events that re-create the node, such as enabling or disabling a feature on the node pool.

Note: During automatic or manual node upgrades, PodDisruptionBudgets (PDBs) are respected for a maximum of 1 hour. If Pods running on a node cannot be scheduled onto new nodes within 1 hour, the upgrade is initiated, regardless.
For more information about what to expect during node termination in general, see the topic about Pods.

The upgrade is only complete when all nodes have been recreated and the cluster is in the desired state. When a newly-upgraded node registers with the control plane, GKE marks the node as schedulable.

New node instances run the desired Kubernetes version as well as:

• The node image
• The Docker daemon, if applicable
• kubelet
• kube-proxy

Manually upgrade a node pool

You can manually upgrade a node pool version to match the version of the control plane or to a previous version that is still available and is compatible with the control plane. The Kubernetes version and version skew support policy guarantees that control planes are compatible with nodes up to two minor versions older than the control plane. For example, Kubernetes 1.13 control planes are compatible with Kubernetes 1.11 nodes.

When you manually upgrade a node pool, GKE removes any labels you added to individual nodes using kubectl. To avoid this, apply labels to node pools (Preview) instead.

Note: Upgrading a node pool may disrupt workloads running in that node pool. To avoid this, you can create a new node pool with the desired version and migrate the workload. After migration, you can delete the old node pool.

Note: If you upgrade a node pool with an Ingress in an errored state, the instance group does not sync. To work around this issue, first check the status using the kubectl get ing command. If the instance group is not synced, you can work around the problem by re-applying the manifest used to create the ingress.

You can manually upgrade your node pools to a version compatible with the control plane, using the Google Cloud Console or the gcloud command-line tool.

gcloud

To update all nodes to the same version as the control plane, run the gcloud container clusters upgrade command:

gcloud container clusters upgrade CLUSTER_NAME

Replace CLUSTER_NAME with the name of the cluster to be upgraded.

To update a specific node pool, specify the --node-pool flag:

gcloud container clusters upgrade CLUSTER_NAME \
  --node-pool=NODE_POOL_NAME

To specify a different version of GKE on nodes, use the optional --cluster-version flag:

gcloud container clusters upgrade CLUSTER_NAME \
  --node-pool=NODE_POOL_NAME \
  --cluster-version VERSION

Replace VERSION with the Kubernetes version to which the nodes are upgraded. For example, --cluster-version=1.7.2 or cluster-version=latest.

For more information about specifying versions, see Versioning.

For more information, refer to the gcloud container clusters upgrade documentation.

Console

To upgrade a node pool using Cloud Console, perform the following steps:

To upgrade a node pool, perform the following steps:

1. Go to the Google Kubernetes Engine page in Cloud Console.

   Go to Google Kubernetes Engine

2. Next to the cluster you want to edit, click Actions, then click Edit.

3. On the Cluster details page, click the Nodes tab.

4. In the Node Pools section, click the name of the node pool that you want to upgrade.

5. Click Edit.

6. Click Change under Node version.

7. Select the desired version from the Node version drop-down list, then click Change.

Note: It may take several minutes for the node version to change.

Deploying a Pod to a specific node pool

You can explicitly deploy a Pod to a specific node pool by using a nodeSelector in your Pod manifest. nodeSelector schedules Pods onto nodes with a matching label.

All GKE node pools have labels with the following format: cloud.google.com/gke-nodepool: POOL_NAME. Add this label to the nodeSelector field in your Pod as shown in the following example:

apiVersion: v1
kind: Pod
metadata:
  name: nginx
  labels:
    env: test
spec:
  containers:
  - name: nginx
    image: nginx
    imagePullPolicy: IfNotPresent
  nodeSelector:
    cloud.google.com/gke-nodepool: POOL_NAME

For more information, see Assigning Pods to Nodes.

As an alternative to node selector, you can use node affinity. Use node affinity if you want a "soft" rule where the Pod attempts to meet the constraint, but is still scheduled even if the constraint can't be satisfied. For more information, see Node affinity. You can also specify resource requests for the containers.

Downgrading node pools

Note: Downgrading node pools is not recommended. You should only attempt to downgrade a node pool to mitigate an unsuccessful node pool upgrade.

Deleting a node pool

Deleting a node pool deletes the nodes and routes to them. Any Pods running on those nodes are evicted and rescheduled. If the Pods have specific node selectors, the Pods might remain in an unschedulable condition if no other node in the cluster satisfies the criteria.

gcloud

To delete a node pool, run the gcloud container node-pools delete command:

gcloud container node-pools delete POOL_NAME --cluster CLUSTER_NAME

Console

To delete a node pool, perform the following steps:

1. Go to the Google Kubernetes Engine page in Cloud Console.

   Go to Google Kubernetes Engine

2. Beside the cluster you want to edit, click Actions, then click Edit.

3. Click the Nodes tab.

4. In the Node Pools section, click next to the node pool you want to delete.

5. When prompted to confirm, click Delete.

What's next

• Read about how node pools work.
• Learn about auto-provisioning node pools.
• Learn how to configure kubelet and sysctl using Node System Configuration.