Migrate from Docker to containerd node images

Standard

---

This page shows you how to migrate your Google Kubernetes Engine (GKE) Standard cluster and node pools to node images that use the containerd container runtime.

Important: GKE begins automatically migrating clusters to GKE version 1.24 after version 1.23 reaches end-of-life on July 31, 2023. To learn more about how the migration process works, and how you can use a maintenance exclusion to temporarily prevent your nodes from being migrated to containerd node images after version 1.23 reaches end-of-life, see Temporarily delay the automatic migration to containerd node images.

Overview

Kubernetes nodes use the container runtime to launch, manage, and stop containers running in Pods. The containerd runtime is an industry-standard container runtime that's supported by GKE.

The containerd runtime provides the layering abstraction that allows for the implementation of a rich set of features like gVisor and Image streaming to extend GKE functionality. The containerd runtime is considered more resource efficient and secure than the Docker runtime.

Note: GKE version 1.24 and later only support node images that use the containerd runtime. For clusters using Docker node images, before version 1.23 is end-of-life, automatic upgrades to 1.24 are paused and manual upgrades are blocked until all nodes are migrated to containerd node images. To learn more about the removal of Docker support and the potential impact of migrating, refer to About the Docker node image deprecation.

To migrate your container runtime:

• Identify nodes that use the Docker runtime
• Verify the impact of migration
• Change the node image

Before you begin

Before you start, make sure you have performed the following tasks:

• Enable the Google Kubernetes Engine API.
Enable Google Kubernetes Engine API
• If you want to use the Google Cloud CLI for this task, install and then initialize the gcloud CLI. If you previously installed the gcloud CLI, get the latest version by running gcloud components update.
  Note: For existing gcloud CLI installations, make sure to set the compute/region and compute/zone properties. By setting default locations, you can avoid errors in gcloud CLI like the following: One of [--zone, --region] must be supplied: Please specify location.

Identify nodes that use the Docker runtime

You can check which nodes use Docker-based node images using the following methods:

• Use a script to iterate over all nodes in all GKE clusters across your Google Cloud project.
• Use the Google Cloud CLI, kubectl, or the Google Cloud console to identify node images.
• Use deprecation insights and recommendations to identify clusters and nodes in specific zones or regions in a Google Cloud project.

We recommend that you use a script to quickly identify all the nodes that you need to migrate.

Use a script to identify Docker nodes

The following script iterates over every node in every cluster in your available projects and provides you with actionable recommendations, such as:

• Whether node auto-provisioning is configured for Docker images
• Suggested containerd node image equivalents for migration
• Suggested node image versions for migration
• Suggested commands to run to migrate the identified nodes and settings

The script ignores GKE Autopilot clusters, which use the Container-Optimized OS with containerd node image by default.

Run the following script:

migrating-to-containerd/find-nodepools-to-migrate.sh

View on GitHub Feedback

for project in $(gcloud projects list --format="value(projectId)")
do
  echo "ProjectId: $project"
  for clusters in $( \
    gcloud container clusters list \
      --project $project \
      --format="csv[no-heading](name,location,autopilot.enabled,currentMasterVersion,autoscaling.enableNodeAutoprovisioning,autoscaling.autoprovisioningNodePoolDefaults.imageType)")
  do
    IFS=',' read -r -a clustersArray <<< "$clusters"
    cluster_name="${clustersArray[0]}"
    cluster_zone="${clustersArray[1]}"
    cluster_isAutopilot="${clustersArray[2]}"
    cluster_version="${clustersArray[3]}"
    cluster_minorVersion=${cluster_version:0:4}
    cluster_autoprovisioning="${clustersArray[4]}"
    cluster_autoprovisioningImageType="${clustersArray[5]}"

    if [ "$cluster_isAutopilot" = "True" ]; then
      echo "  Cluster: $cluster_name (autopilot) (zone: $cluster_zone)"
      echo "    Autopilot clusters are running Containerd."
    else
      echo "  Cluster: $cluster_name (zone: $cluster_zone)"

      if [ "$cluster_autoprovisioning" = "True" ]; then
        if [ "$cluster_minorVersion"  \< "1.20" ]; then
          echo "    Node autoprovisioning is enabled, and new node pools will have image type 'COS'."
          echo "    This settings is not configurable on the current version of a cluster."
          echo "    Please upgrade you cluster and configure the default node autoprovisioning image type."
          echo "    "
        else
          if [ "$cluster_autoprovisioningImageType" = "COS" ]; then
            echo "    Node autoprovisioning is configured to create new node pools of type 'COS'."
            echo "    Run the following command to update:"
            echo "    gcloud container clusters update '$cluster_name' --project '$project' --zone '$cluster_zone' --enable-autoprovisioning --autoprovisioning-image-type='COS_CONTAINERD'"
            echo "    "
          fi

          if [ "$cluster_autoprovisioningImageType" = "UBUNTU" ]; then
            echo "    Node autoprovisioning is configured to create new node pools of type 'UBUNTU'."
            echo "    Run the following command to update:"
            echo "    gcloud container clusters update '$cluster_name' --project '$project' --zone '$cluster_zone' --enable-autoprovisioning --autoprovisioning-image-type='UBUNTU_CONTAINERD'"
            echo "    "
          fi
        fi
      fi

      for nodepools in $( \
        gcloud container node-pools list \
          --project $project \
          --cluster $cluster_name \
          --zone $cluster_zone \
          --format="csv[no-heading](name,version,config.imageType)")
      do
        IFS=',' read -r -a nodepoolsArray <<< "$nodepools"
        nodepool_name="${nodepoolsArray[0]}"
        nodepool_version="${nodepoolsArray[1]}"
        nodepool_imageType="${nodepoolsArray[2]}"

        nodepool_minorVersion=${nodepool_version:0:4}

        echo "    Nodepool: $nodepool_name, version: $nodepool_version ($nodepool_minorVersion), image: $nodepool_imageType"

        minorVersionWithRev="${nodepool_version/-gke./.}"
        linuxGkeMinVersion="1.14"
        windowsGkeMinVersion="1.21.1.2200"

        suggestedImageType="COS_CONTAINERD"

        if [ "$nodepool_imageType" = "UBUNTU" ]; then
          suggestedImageType="UBUNTU_CONTAINERD"
        elif [ "$nodepool_imageType" = "WINDOWS_LTSC" ]; then
          suggestedImageType="WINDOWS_LTSC_CONTAINERD"
        elif [ "$nodepool_imageType" = "WINDOWS_SAC" ]; then
          suggestedImageType="WINDOWS_SAC_CONTAINERD"
        fi

        tab=$'\n      ';
        nodepool_message="$tab Please update the nodepool to use Containerd."
        nodepool_message+="$tab Make sure to consult with the list of known issues https://cloud.google.com/kubernetes-engine/docs/concepts/using-containerd#known_issues."
        nodepool_message+="$tab Run the following command to upgrade:"
        nodepool_message+="$tab "
        nodepool_message+="$tab gcloud container clusters upgrade '$cluster_name' --project '$project' --zone '$cluster_zone' --image-type '$suggestedImageType' --node-pool '$nodepool_name'"
        nodepool_message+="$tab "

        # see https://cloud.google.com/kubernetes-engine/docs/concepts/node-images
        if [ "$nodepool_imageType" = "COS_CONTAINERD" ] || [ "$nodepool_imageType" = "UBUNTU_CONTAINERD" ] ||
           [ "$nodepool_imageType" = "WINDOWS_LTSC_CONTAINERD" ] || [ "$nodepool_imageType" = "WINDOWS_SAC_CONTAINERD" ]; then
          nodepool_message="$tab Nodepool is using Containerd already"
        elif ( [ "$nodepool_imageType" = "WINDOWS_LTSC" ] || [ "$nodepool_imageType" = "WINDOWS_SAC" ] ) &&
               [ "$(printf '%s\n' "$windowsGkeMinVersion" "$minorVersionWithRev" | sort -V | head -n1)" != "$windowsGkeMinVersion" ]; then
          nodepool_message="$tab Upgrade nodepool to the version that supports Containerd for Windows"
        elif [ "$(printf '%s\n' "$linuxGkeMinVersion" "$minorVersionWithRev" | sort -V | head -n1)" != "$linuxGkeMinVersion" ]; then
          nodepool_message="$tab Upgrade nodepool to the version that supports Containerd"
        fi
        echo "$nodepool_message"
      done
    fi # not autopilot
  done
done

# Sample output:
#
# ProjectId:  my-project-id
#  Cluster: autopilot-cluster-1 (autopilot) (zone: us-central1)
#    Autopilot clusters are running Containerd.
#  Cluster: cluster-1 (zone: us-central1-c)
#    Nodepool: default-pool, version: 1.18.12-gke.1210 (1.18), image: COS
#
#       Please update the nodepool to use Containerd.
#       Make sure to consult with the list of known issues https://cloud.google.com/kubernetes-engine/docs/concepts/using-containerd#known_issues.
#       Run the following command to upgrade:
#
#       gcloud container clusters upgrade 'cluster-1' --project 'my-project-id' --zone 'us-central1-c' --image-type 'COS_CONTAINERD' --node-pool 'default-pool'
#
#    Nodepool: pool-1, version: 1.18.12-gke.1210 (1.18), image: COS
#
#       Please update the nodepool to use Containerd.
#       Make sure to consult with the list of known issues https://cloud.google.com/kubernetes-engine/docs/concepts/using-containerd#known_issues.
#       Run the following command to upgrade:
#
#       gcloud container clusters upgrade 'cluster-1' --project 'my-project-id' --zone 'us-central1-c' --image-type 'COS_CONTAINERD' --node-pool 'pool-1'
#
#    Nodepool: winpool, version: 1.18.12-gke.1210 (1.18), image: WINDOWS_SAC
#
#       Upgrade nodepool to the version that supports Containerd for Windows
#
#  Cluster: another-test-cluster (zone: us-central1-c)
#    Nodepool: default-pool, version: 1.20.4-gke.400 (1.20), image: COS_CONTAINERD
#
#      Nodepool is using Containerd already
#

Identify node images using Google Cloud

You can check the node images for existing nodes using the Google Cloud CLI, kubectl, or the Google Cloud console.

gcloud

Run the following command:

gcloud container node-pools list \
    --cluster=CLUSTER_NAME \
    --format="table(name,version,config.imageType)"

Replace CLUSTER_NAME with the name of your cluster.

The output is similar to the following:

NAME          NODE_VERSION    IMAGE_TYPE
default-pool  1.19.6-gke.600  UBUNTU

Console

1. Go to the Google Kubernetes Engine page in the Google Cloud console.

   Go to Google Kubernetes Engine

2. In the cluster list, click the name of the cluster you want to verify.

3. Select the Nodes tab.

4. In the Node pools section, check the value in the Image type column.

kubectl

Run the following command:

kubectl get nodes -o wide

The output is similar to the following:

# For Docker runtime
NAME         STATUS   VERSION             OS-IMAGE                             CONTAINER-RUNTIME
gke-node-1   Ready    v1.16.15-gke.6000   Container-Optimized OS from Google   docker://19.3.1
gke-node-2   Ready    v1.16.15-gke.6000   Container-Optimized OS from Google   docker://19.3.1
gke-node-3   Ready    v1.16.15-gke.6000   Container-Optimized OS from Google   docker://19.3.1

The value in the CONTAINER-RUNTIME column shows the runtime and version.

Identify clusters using deprecation insights and recommendations

GKE detects the usage of some deprecated features and APIs, including Docker-based node images. To learn more, see GKE deprecations.

As part of detecting the usage of deprecations, GKE generates insights and recommendations which identify usage of Docker-based node images with the DEPRECATION_K8S_1_24_DOCKERSHIM insight subtype.

One insight and recommendation pairing identifies a cluster that has nodes that are using Docker-based node images. Each insight and recommendation provides a list of the specific node pools in a cluster that are using Docker-based node images and must be migrated to containerd node images.

To get started, follow the instructions to view deprecation insights and recommendations. For the gcloud CLI commands, use the following flag to just see insights for this deprecation:

--filter="insightSubtype:DEPRECATION_K8S_1_24_DOCKERSHIM"

Once you've identified which cluster's node pools are using Docker-based node images, follow the instructions to change the node image to a containerd node image.

Verify the impact of migrating

Before you migrate your production clusters and node pools to node images that use containerd, we strongly recommend that you test the impact of the migration in a staging environment to minimize the risk of issues.

We recommend that when you migrate the nodes, you do so independently of upgrading the nodes, so that you can isolate variables when verifying the workloads function with the new configuration. Additionally, if you upgrade the node pool to version 1.24 at the same time, you cannot roll back the change as 1.24 does not support Docker nodes and you cannot downgrade minor versions.

Change the node image to a containerd image

If you used the script to identify your Docker nodes, you can use the suggested commands returned by the script to change your node auto-provisioning settings and your node images to the containerd equivalents.

You can also migrate nodes from a Docker image type to a containerd image type by updating the node pool and setting a different image using the gcloud CLI or the Google Cloud console.

GKE uses the selected node pool upgrade strategy and configuration to migrate a node's image. For this migration, we recommend that you use the blue-green upgrade strategy because if your workloads experience issues with the new node image during the upgrade, you can roll back to the previous environment with the original node image configuration.

gcloud

Run the following command:

gcloud container clusters upgrade CLUSTER_NAME \
    --image-type=NODE_IMAGE \
    --node-pool=POOL_NAME \
    --cluster-version=NODE_VERSION

Replace the following:

• NODE_IMAGE: the node image that you want the nodes to use.
• POOL_NAME: the name of the node pool to migrate.
• NODE_VERSION: the existing version of the node pool. We recommend setting this flag because otherwise, GKE attempts to upgrade the version of the node pool to the control plane version and update the node image in the same operation. If the control plane is running 1.24, the command fails. If the control plane is running 1.23, the command succeeds, which prevents you from testing the two changes (version upgrade and image update) in isolation.

The output is similar to the following:

NAME          NODE_VERSION    IMAGE_TYPE
default-pool  1.23.6-gke.600  UBUNTU_CONTAINERD

Note: After updating your node image, if you notice a problem and need to revert to the Docker image, run the same command and replace NODE_IMAGE with the previous Docker image.

Console

1. Go to the Google Kubernetes Engine page in the Google Cloud console.

   Go to Google Kubernetes Engine

2. In the cluster list, click the name of the cluster you want to verify.

3. Click the Nodes tab.

4. In the Node pools section, click the name of the node pool that you want to modify.

5. On the Node pool details page, click Edit.

6. In the Nodes section, under Image type, click Change.

7. Select one of the containerd image types.

8. Click Change.

Note: After updating your node image, if you notice a problem and need to revert to the Docker image, perform the previous steps and select the previous Docker image.

Roll back to Docker node images

If your nodes were automatically or manually migrated to containerd nodes and you saw an issue with your workloads, perform the following steps to revert to Docker node images:

1. Choose the following step depending on the state of the operation:
   • If the operation is still in progress, you can cancel and roll back.
   • If the operation is complete, see the previous section's steps and select the equivalent Docker image.
2. Configure a maintenance exclusion to temporarily prevent GKE from retrying the migration.
3. Investigate the root cause of the issue so that you can migrate from Docker and ensure that your cluster is running a supported version of GKE.
4. Try again to change the node image to a containerd image. If you remove the maintenance exclusion, GKE will trigger the operation again.

Update Infrastructure as Code (IaC) tool configurations

If you use IaC tools such as Terraform, Ansible, or Pulumi to manage GKE clusters, ensure that you update your configurations to use containerd node images to prevent the tools from reconciling the previously desired state with the new actual state. For example, the GKE Terraform provider supports configurable image types.

Update any configurations so that the tool does not update the node image back to a Docker-based node image after you've migrated to containerd node images.

Change the default node image for node auto-provisioning

If you use node auto-provisioning in your cluster, change the default image type to a containerd node image. Changing the default image type only applies to new auto-provisioned node pools. You must manually change the node image for existing auto-provisioned node pools.

You can change the default node auto-provisioning image type using the gcloud CLI or a configuration file.

gcloud

Run the following command:

gcloud container clusters update CLUSTER_NAME \
    --enable-autoprovisioning \
    --autoprovisioning-image-type=IMAGE_TYPE

Replace the following:

• CLUSTER_NAME: the name of the cluster to update.

• IMAGE_TYPE: the node image type, which can be one of the following:

  • cos_containerd

  • ubuntu_containerd

    Note: For ubuntu_containerd, specify --autoprovisioning-scopes=https://www.googleapis.com/auth/devstorage.read_only in the command.

File

You can use a YAML configuration file to change the default node image type for node auto-provisioning. When you use a file, you must also specify maximum values for CPU and memory resources.

1. Save the following YAML file:

   resourceLimits:
     - resourceType: 'cpu'
         minimum: 4
         maximum: 10
     - resourceType: 'memory'
         maximum: 64
   autoprovisioningNodePoolDefaults:
     imageType: 'IMAGE_TYPE'
   

   Replace IMAGE_TYPE with the containerd image type.

   Note: For ubuntu_containerd, add scopes: https://www.googleapis.com/auth/devstorage.read_only to the configuration file.

2. Apply the configuration:

   gcloud container clusters update CLUSTER_NAME \
       --enable-autoprovisioning \
       --autoprovisioning-config-file=FILE_NAME
   

   Replace FILE_NAME with the path to the configuration file.

Troubleshooting

For troubleshooting and for known issues with workarounds, refer to Troubleshooting the container runtime.

What's next

• Learn more about node auto-provisioning.
• Learn more about containerd node images.