Migrate your container runtime to containerd

Standard

---

This page shows you how to migrate your Google Kubernetes Engine (GKE) Standard cluster and node pools to node images that use the containerd container runtime.

Overview

Kubernetes nodes use the container runtime to launch, manage, and stop containers running in Pods. The containerd runtime is an industry-standard container runtime that's supported by GKE.

The containerd runtime provides the layering abstraction that allows for the implementation of a rich set of features like gVisor and Image streaming to extend GKE functionality. The containerd runtime is considered more resource efficient and secure than the Docker runtime.

Note: GKE 1.24 and later only support node images that use the containerd runtime. To use these versions of GKE, you must migrate to a containerd node image. To learn more about the removal of Docker support and the potential impact of migrating, refer to Migrating to containerd.

To migrate your container runtime:

• Identify nodes that use the Docker runtime
• Verify the impact of migration
• Change the node image

Before you begin

Before you start, make sure you have performed the following tasks:

• Ensure that you have enabled the Google Kubernetes Engine API.
Enable Google Kubernetes Engine API
• Ensure that you have installed the Google Cloud CLI.
• Set up default Google Cloud CLI settings for your project by using one of the following methods:
• Use gcloud init, if you want to be walked through setting project defaults.
• Use gcloud config, to individually set your project ID, zone, and region.

gcloud init

1. Run gcloud init and follow the directions:

   gcloud init

   If you are using SSH on a remote server, use the --console-only flag to prevent the command from launching a browser:

   gcloud init --console-only

2. Follow the instructions to authorize the gcloud CLI to use your Google Cloud account.
3. Create a new configuration or select an existing one.
4. Choose a Google Cloud project.
5. Choose a default Compute Engine zone.
6. Choose a default Compute Engine region.

Note: You can override these default settings in the gcloud CLI by using the --project, --zone, and --region flags.

gcloud config

1. Set your default project ID:

   gcloud config set project PROJECT_ID

2. Set your default Compute Engine region (for example, us-central1):

   gcloud config set compute/region COMPUTE_REGION

3. Set your default Compute Engine zone (for example, us-central1-c):

   gcloud config set compute/zone COMPUTE_ZONE

4. Update gcloud to the latest version:

   gcloud components update

Note: You can override these default settings in the gcloud CLI by using the --project, --zone, and --region flags.

By setting default locations, you can avoid errors in gcloud CLI like the following: One of [--zone, --region] must be supplied: Please specify location.

Identify nodes that use the Docker runtime

You can check which nodes use Docker-based node images using the following methods:

• Use a script to iterate over all nodes in all GKE clusters across your Google Cloud project.
• Use the Google Cloud CLI, kubectl, or the Google Cloud console to identify node images.

We recommend that you use a script to quickly identify all the nodes that you need to migrate.

Use a script to identify Docker nodes

The following script iterates over every node in every cluster in your available projects and provides you with actionable recommendations, such as:

• Whether node auto-provisioning is configured for Docker images
• Suggested containerd node image equivalents for migration
• Suggested node image versions for migration
• Suggested commands to run to migrate the identified nodes and settings

The script ignores GKE Autopilot clusters, which use the Container-Optimized OS with containerd node image by default.

Run the following script:

migrating-to-containerd/find-nodepools-to-migrate.sh

View on GitHub Feedback

for project in  $(gcloud projects list --format="value(projectId)")
do
  echo "ProjectId:  $project"
  for clusters in $( \
    gcloud container clusters list \
      --project $project \
      --format="csv[no-heading](name,location,autopilot.enabled,currentMasterVersion,autoscaling.enableNodeAutoprovisioning,autoscaling.autoprovisioningNodePoolDefaults.imageType)")
  do
    IFS=',' read -r -a clustersArray <<< "$clusters"
    cluster_name="${clustersArray[0]}"
    cluster_zone="${clustersArray[1]}"
    cluster_isAutopilot="${clustersArray[2]}"
    cluster_version="${clustersArray[3]}"
    cluster_minorVersion=${cluster_version:0:4}
    cluster_autoprovisioning="${clustersArray[4]}"
    cluster_autoprovisioningImageType="${clustersArray[5]}"

    if [ "$cluster_isAutopilot" = "True" ]; then
      echo "  Cluster: $cluster_name (autopilot) (zone: $cluster_zone)"
      echo "    Autopilot clusters are running Containerd."
    else
      echo "  Cluster: $cluster_name (zone: $cluster_zone)"

      if [ "$cluster_autoprovisioning" = "True" ]; then
        if [ "$cluster_minorVersion"  \< "1.20" ]; then
          echo "    Node autoprovisioning is enabled, and new node pools will have image type 'COS'."
          echo "    This settings is not configurable on the current version of a cluster."
          echo "    Please upgrade you cluster and configure the default node autoprovisioning image type."
          echo "    "
        else
          if [ "$cluster_autoprovisioningImageType" = "COS" ]; then
            echo "    Node autoprovisioning is configured to create new node pools of type 'COS'."
            echo "    Run the following command to update:"
            echo "    gcloud container clusters update '$cluster_name' --project '$project' --zone '$cluster_zone' --enable-autoprovisioning --autoprovisioning-image-type='COS_CONTAINERD'"
            echo "    "
          fi

          if [ "$cluster_autoprovisioningImageType" = "UBUNTU" ]; then
            echo "    Node autoprovisioning is configured to create new node pools of type 'UBUNTU'."
            echo "    Run the following command to update:"
            echo "    gcloud container clusters update '$cluster_name' --project '$project' --zone '$cluster_zone' --enable-autoprovisioning --autoprovisioning-image-type='UBUNTU_CONTAINERD'"
            echo "    "
          fi
        fi
      fi

      for nodepools in $( \
        gcloud container node-pools list \
          --project $project \
          --cluster $cluster_name \
          --zone $cluster_zone \
          --format="csv[no-heading](name,version,config.imageType)")
      do
        IFS=',' read -r -a nodepoolsArray <<< "$nodepools"
        nodepool_name="${nodepoolsArray[0]}"
        nodepool_version="${nodepoolsArray[1]}"
        nodepool_imageType="${nodepoolsArray[2]}"

        nodepool_minorVersion=${nodepool_version:0:4}

        echo "    Nodepool: $nodepool_name, version: $nodepool_version ($nodepool_minorVersion), image: $nodepool_imageType"

        minorVersionWithRev="${nodepool_version/-gke./.}"
        linuxGkeMinVersion="1.14"
        windowsGkeMinVersion="1.21.1.2200"

        suggestedImageType="COS_CONTAINERD"

        if [ "$nodepool_imageType" = "UBUNTU" ]; then
          suggestedImageType="UBUNTU_CONTAINERD"
        elif [ "$nodepool_imageType" = "WINDOWS_LTSC" ]; then
          suggestedImageType="WINDOWS_LTSC_CONTAINERD"
        elif [ "$nodepool_imageType" = "WINDOWS_SAC" ]; then
          suggestedImageType="WINDOWS_SAC_CONTAINERD"
        fi

        tab=$'\n      ';
        nodepool_message="$tab Please update the nodepool to use Containerd."
        nodepool_message+="$tab Make sure to consult with the list of known issues https://cloud.google.com/kubernetes-engine/docs/concepts/using-containerd#known_issues."
        nodepool_message+="$tab Run the following command to upgrade:"
        nodepool_message+="$tab "
        nodepool_message+="$tab gcloud container clusters upgrade '$cluster_name' --project '$project' --zone '$cluster_zone' --image-type '$suggestedImageType' --node-pool '$nodepool_name'"
        nodepool_message+="$tab "

        # see https://cloud.google.com/kubernetes-engine/docs/concepts/node-images
        if [ "$nodepool_imageType" = "COS_CONTAINERD" ] || [ "$nodepool_imageType" = "UBUNTU_CONTAINERD" ] ||
           [ "$nodepool_imageType" = "WINDOWS_LTSC_CONTAINERD" ] || [ "$nodepool_imageType" = "WINDOWS_SAC_CONTAINERD" ]; then
          nodepool_message="$tab Nodepool is using Containerd already"
        elif ( [ "$nodepool_imageType" = "WINDOWS_LTSC" ] || [ "$nodepool_imageType" = "WINDOWS_SAC" ] ) &&
               !( [ "$(printf '%s\n' "$windowsGkeMinVersion" "$minorVersionWithRev" | sort -V | head -n1)" = "$windowsGkeMinVersion" ] ); then
          nodepool_message="$tab Upgrade nodepool to the version that supports Containerd for Windows"
        elif !( [ "$(printf '%s\n' "$linuxGkeMinVersion" "$minorVersionWithRev" | sort -V | head -n1)" = "$linuxGkeMinVersion" ] ); then
          nodepool_message="$tab Upgrade nodepool to the version that supports Containerd"
        fi
        echo "$nodepool_message"
      done
    fi # not autopilot
  done
done

# Sample output:
#
# ProjectId:  my-project-id
#  Cluster: autopilot-cluster-1 (autopilot) (zone: us-central1)
#    Autopilot clusters are running Containerd.
#  Cluster: cluster-1 (zone: us-central1-c)
#    Nodepool: default-pool, version: 1.18.12-gke.1210 (1.18), image: COS
#
#       Please update the nodepool to use Containerd.
#       Make sure to consult with the list of known issues https://cloud.google.com/kubernetes-engine/docs/concepts/using-containerd#known_issues.
#       Run the following command to upgrade:
#
#       gcloud container clusters upgrade 'cluster-1' --project 'my-project-id' --zone 'us-central1-c' --image-type 'COS_CONTAINERD' --node-pool 'default-pool'
#
#    Nodepool: pool-1, version: 1.18.12-gke.1210 (1.18), image: COS
#
#       Please update the nodepool to use Containerd.
#       Make sure to consult with the list of known issues https://cloud.google.com/kubernetes-engine/docs/concepts/using-containerd#known_issues.
#       Run the following command to upgrade:
#
#       gcloud container clusters upgrade 'cluster-1' --project 'my-project-id' --zone 'us-central1-c' --image-type 'COS_CONTAINERD' --node-pool 'pool-1'
#
#    Nodepool: winpool, version: 1.18.12-gke.1210 (1.18), image: WINDOWS_SAC
#
#       Upgrade nodepool to the version that supports Containerd for Windows
#
#  Cluster: another-test-cluster (zone: us-central1-c)
#    Nodepool: default-pool, version: 1.20.4-gke.400 (1.20), image: COS_CONTAINERD
#
#      Nodepool is using Containerd already
#

Identify node images using Google Cloud

You can check the node images for existing nodes using the Google Cloud CLI, kubectl, or the Google Cloud console.

gcloud

Run the following command:

gcloud container node-pools list \
    --cluster=CLUSTER_NAME \
    --format="table(name,version,config.imageType)"

Replace CLUSTER_NAME with the name of your cluster.

The output is similar to the following:

NAME          NODE_VERSION    IMAGE_TYPE
default-pool  1.19.6-gke.600  UBUNTU

Console

1. Go to the Google Kubernetes Engine page in Google Cloud console.

   Go to Google Kubernetes Engine

2. In the cluster list, click the name of the cluster you want to verify.

3. Select the Nodes tab.

4. In the Node pools section, check the value in the Image type column.

kubectl

Run the following command:

kubectl get nodes -o wide

The output is similar to the following:

# For Docker runtime
NAME         STATUS   VERSION             OS-IMAGE                             CONTAINER-RUNTIME
gke-node-1   Ready    v1.16.15-gke.6000   Container-Optimized OS from Google   docker://19.3.1
gke-node-2   Ready    v1.16.15-gke.6000   Container-Optimized OS from Google   docker://19.3.1
gke-node-3   Ready    v1.16.15-gke.6000   Container-Optimized OS from Google   docker://19.3.1

The value in the CONTAINER-RUNTIME column shows the runtime and version.

Verify the impact of migrating

Before you migrate your production clusters and node pools to node images that use containerd, we strongly recommend that you test the impact of the migration in a staging environment to minimize the risk of issues.

Change the node image to a containerd image

If you used the script to identify your Docker nodes, you can use the suggested commands returned by the script to change your node auto-provisioning settings and your node images to the containerd equivalents.

You can also migrate nodes from a Docker image type to a containerd image type by updating the node pool and setting a different image using the gcloud CLI or the Google Cloud console.

gcloud

Run the following command:

gcloud container clusters upgrade CLUSTER_NAME \
    --image-type=NODE_IMAGE \
    --node-pool=POOL_NAME

Replace the following:

• NODE_IMAGE: the node image that you want the nodes to use.
• POOL_NAME: the name of the node pool to migrate.

The output is similar to the following:

NAME          NODE_VERSION    IMAGE_TYPE
default-pool  1.19.6-gke.600  UBUNTU_CONTAINERD

Note: After updating your node image, if you notice a problem and need to revert to the Docker image, run the same command and replace NODE_IMAGE with the previous Docker image.

Console

1. Go to the Google Kubernetes Engine page in Google Cloud console.

   Go to Google Kubernetes Engine

2. In the cluster list, click the name of the cluster you want to verify.

3. Click the Nodes tab.

4. In the Node pools section, click the name of the node pool that you want to modify.

5. On the Node pool details page, click Edit.

6. In the Nodes section, under Image type, click Change.

7. Select one of the containerd image types.

8. Click Change.

Note: After updating your node image, if you notice a problem and need to revert to the Docker image, perform the previous steps and select the previous Docker image.

Change the default node image for node auto-provisioning

If you use node auto-provisioning in your cluster, change the default image type to a containerd node image. Changing the default image type only applies to new auto-provisioned node pools. You must manually change the node image for existing auto-provisioned node pools.

You can change the default node auto-provisioning image type using the gcloud CLI or a configuration file.

gcloud

Run the following command:

gcloud container clusters update CLUSTER_NAME \
    --enable-autoprovisioning \
    --autoprovisioning-image-type=IMAGE_TYPE

Replace the following:

• CLUSTER_NAME: the name of the cluster to update.

• IMAGE_TYPE: the node image type, which can be one of the following:

  • cos_containerd

  • ubuntu_containerd

    Note: For ubuntu_containerd, specify --autoprovisioning-scopes=https://www.googleapis.com/auth/devstorage.read_only in the command.

File

You can use a YAML configuration file to change the default node image type for node auto-provisioning. When you use a file, you must also specify maximum values for CPU and memory resources.

1. Save the following YAML file:

   resourceLimits:
     - resourceType: 'cpu'
         minimum: 4
         maximum: 10
     - resourceType: 'memory'
         maximum: 64
   autoprovisioningNodePoolDefaults:
     imageType: 'IMAGE_TYPE'
   

   Replace IMAGE_TYPE with the containerd image type.

   Note: For ubuntu_containerd, add scopes: https://www.googleapis.com/auth/devstorage.read_only to the configuration file.

2. Apply the configuration:

   gcloud container clusters update CLUSTER_NAME \
       --enable-autoprovisioning \
       --autoprovisioning-config-file=FILE_NAME
   

   Replace FILE_NAME with the path to the configuration file.

Troubleshooting

For troubleshooting and for known issues with workarounds, refer to Troubleshooting the container runtime.

What's next

• Learn more about node auto-provisioning.
• Learn more about containerd node images.