Containerd node images

This page provides information about node images that use containerd as the container runtime in your Google Kubernetes Engine (GKE) nodes.

About containerd

The container runtime is software that is responsible for running containers, and abstracts container management for Kubernetes. There are a few different container runtimes. The containerd runtime is an industry-standard container runtime that's supported by Kubernetes, and used by many other projects. The containerd runtime provides the layering abstraction that allows for the implementation of a rich set of features like gVisor to extend Kubernetes functionality. The containerd runtime is considered more resource efficient and secure than the Docker runtime.

Using containerd images in GKE clusters

When you create a new GKE cluster, a new node pool in an existing cluster, or when you upgrade an existing cluster, you can choose to use a containerd node image. GKE Autopilot clusters always use Container-Optimized OS with containerd.

The following table describes the supported containerd node images based on your cluster mode and node pool OS:

Cluster mode Node pool OS Node image
Autopilot Linux cos_containerd
Standard Linux
  • cos_containerd
  • ubuntu_containerd

These images require GKE node version 1.14.3 or later.

Standard Windows Server
  • windows_ltsc_containerd
  • windows_sac_containerd

These images require GKE version 1.21.1-gke.2200 or later.

Checking the node image type

You can check which image type is used for existing nodes by using the Google Cloud Console, the gcloud tool, or kubectl. Also, refer to the Sample migration script, which iterates over all node pools, and displays any suggested node pool migrations.

Console

  1. Go to the Google Kubernetes Engine page in Cloud Console.

    Go to Google Kubernetes Engine

  2. In the cluster list, click the name of the cluster you want to verify.

  3. Select the Nodes tab.

  4. In the Node pools section, check the value in the Image type column.

gcloud

Run the following command:

gcloud container node-pools list \
    --cluster=CLUSTER_NAME \
    --format="table(name,version,config.imageType)"

Replace CLUSTER_NAME with the name of your cluster.

The output is similar to the following:

NAME          NODE_VERSION    IMAGE_TYPE
default-pool  1.19.6-gke.600  UBUNTU_CONTAINERD

Refer to the gcloud container node-pools list documentation for more details.

kubectl

Run the following command:

kubectl get nodes -o wide

The output is similar to the following:

# For Docker runtime
NAME         STATUS   VERSION             OS-IMAGE                             CONTAINER-RUNTIME
gke-node-1   Ready    v1.16.15-gke.6000   Container-Optimized OS from Google   docker://19.3.1
gke-node-2   Ready    v1.16.15-gke.6000   Container-Optimized OS from Google   docker://19.3.1
gke-node-3   Ready    v1.16.15-gke.6000   Container-Optimized OS from Google   docker://19.3.1
# For containerd runtime
NAME         STATUS   VERSION           OS-IMAGE                             CONTAINER-RUNTIME
gke-node-1   Ready    v1.19.6-gke.600   Container-Optimized OS from Google   containerd://1.4.1
gke-node-2   Ready    v1.19.6-gke.600   Container-Optimized OS from Google   containerd://1.4.1
gke-node-3   Ready    v1.19.6-gke.600   Container-Optimized OS from Google   containerd://1.4.1

The column CONTAINER-RUNTIME shows the runtime and its version.

Refer to the kubectl get nodes documentation for more details.

Migrating from the Docker runtime to the containerd runtime

Most user workloads don't have a dependency on the container runtime. The container runtime is what runs the containers in your Pods, and the Docker runtime actually uses containerd under the hood, so both runtimes behave similarly.

Even if you use Docker on your developer machine, or as part of a build pipeline that runs outside of your cluster to build and push your images, this itself is not a dependency on the Docker runtime (as these actions happen outside of the cluster).

There are a few instances when you might have a dependency on Docker, such as the following:

  • Running privileged Pods executing Docker commands.
  • Running scripts on nodes outside of Kubernetes infrastructure (for example, using ssh to troubleshoot issues).
  • Running third-party tools that perform similarly privileged operations.

You might also have an indirect dependency on Docker if some of your tooling was configured to react to Docker-specific log messages in your monitoring system.

You can find information on potential dependencies on the Docker runtime in Migrating from dockershim. To confirm compatibility with containerd, you might also want to consult with any vendors who supply logging and monitoring, security, or continuous integration tooling that you deploy inside your cluster.

We recommend that you first deploy your workload on a test node pool with containerd to verify that everything runs as expected. If you have a canary or staging cluster, we recommend migrating this first. You might also want to migrate nodes in stages using the approach explained in Migrating workloads to different machine types.

Updating your node images

You can migrate nodes from a Docker runtime image to a containerd image by updating the node pool and setting a different image. This migration can be done using the Google Cloud Console or the gcloud tool.

Console

  1. Go to the Google Kubernetes Engine page in Cloud Console.

    Go to Google Kubernetes Engine

  2. Click the name of the cluster you want to modify.

  3. On the Cluster details page, click the Nodes tab.

  4. Under Node Pools, click the name of the node pool you want to modify.

  5. On the Node pools details page, click Edit.

  6. In the Nodes section, under Image type, click Change.

  7. Select one of the containerd image variants for your operating system.

  8. Click Change.

gcloud

In the gcloud tool, you can update a node pool by using the gcloud container clusters upgrade command and specifying the --image-type parameter.

For example, to change a node pool's image to Container-Optimized OS with containerd, run the following command:

gcloud container clusters upgrade CLUSTER_NAME \
    --image-type COS_CONTAINERD \
    --node-pool POOL_NAME

Replace POOL_NAME with the name of your node pool.

If after updating your node image you notice a problem and need to revert back to the Docker image variants, run the same command with a Docker image variant.

Refer to the gcloud container clusters upgrade documentation for more details.

Running Docker commands on containerd nodes

While the Docker binary is currently available on Linux nodes that use containerd as a runtime, we do not recommend using it after you migrate to containerd. Docker does not manage the containers Kubernetes runs on containerd nodes, thus you cannot use it to view or interact with running Kubernetes containers using Docker commands or the Docker API.

The Docker binary is not available on Windows Server nodes.

Troubleshooting containers on containerd nodes

For debugging or troubleshooting on Linux nodes, you can interact with containerd using the portable command-line tool built for Kubernetes container runtimes: crictl. crictl supports common functionalities to view containers and images, read logs, and execute commands in the containers. Refer to the crictl user guide for the complete set of supported features and usage information.

For Windows Server nodes, the containerd daemon runs as a Windows service named containerd. Logs are available in the following logs directory: C:\etc\kubernetes\logs\containerd.log and are shown in Logs Explorer under LOG NAME: "container-runtime".

Accessing Docker Engine from privileged Pods

Some users currently access Docker Engine on a node from within privileged Pods. It is recommended that you update your workloads so that they do not rely on Docker directly. For example, if you currently extract application logs or monitoring data from Docker Engine, consider using GKE system add-ons for logging and monitoring instead.

Building images

The containerd runtime does not support building images, because the feature is not supported by Kubernetes itself.

Kubernetes is not aware of system resources used by local processes outside the scope of Kubernetes, and the Kubernetes control plane cannot account for those processes when allocating resources. This can starve your GKE workloads of resources or cause instability on the node. For this reason, it is not recommended to run commands on local nodes. Instead, consider accomplishing these tasks using other services outside the scope of the individual container, such as Cloud Build, or use a tool such as kaniko to build images as a Kubernetes workload.

If none of these suggestions work for you, and you understand the risks, you can continue using Docker to build images. You need to push the images to a registry before attempting to use them in a GKE cluster. Kubernetes is not aware of locally-built images.

Known issues

Some filesystem metrics are missing and the metrics format is different

Affected GKE versions: all

The Kubelet /metrics/cadvisor endpoint provides Prometheus metrics, as documented in Metrics for Kubernetes system components. If you install a metrics collector that depends on that endpoint, you might see the following issues:

  • The metrics format on the Docker node is k8s_<container-name>_<pod-name>_<namespace>_<pod-uid>_<restart-count> but the format on the containerd node is <container-id>.
  • Some filesystem metrics are missing on the containerd node, as follows:

    container_fs_inodes_free
    container_fs_inodes_total
    container_fs_io_current
    container_fs_io_time_seconds_total
    container_fs_io_time_weighted_seconds_total
    container_fs_limit_bytes
    container_fs_read_seconds_total
    container_fs_reads_merged_total
    container_fs_sector_reads_total
    container_fs_sector_writes_total
    container_fs_usage_bytes
    container_fs_write_seconds_total
    container_fs_writes_merged_total
    

Workarounds

You can mitigate this issue by using cAdvisor as a standalone daemonset.

  1. Find the latest cAdvisor release with the name pattern vX.Y.Z-containerd-cri (for example, v0.42.0-containerd-cri).
  2. Follow the steps in cAdvisor Kubernetes Daemonset to create the daemonset.
  3. Point the installed metrics collector to use the cAdvisor /metrics endpoint which provides the full set of Prometheus container metrics.

Alternatives

  1. Migrating your monitoring solution to Cloud Monitoring, which provides the full set of container metrics.
  2. Collect metrics from the Kubelet summary API with an endpoint of /stats/summary.

Attach-based operations do not function correctly after container-runtime restarts on GKE Windows

Affected GKE versions: 1.21 to 1.21.5-gke.1802, 1.22 to 1.22.3-gke.700

GKE clusters running Windows Server node pools that use the containerd runtime (version 1.5.4 and 1.5.7-gke.0) might experience issues if the container runtime is forcibly restarted, with attach operations to existing running containers not being able to bind IO again. The issue will not cause failures in API calls, however data will not be sent or received. This includes data for attach and logs CLIs and APIs through the cluster API server.

A patched container runtime version (1.5.7-gke.1) with newer GKE releases address the issue.

Pods display failed to allocate for range 0: no IP addresses available in range set error message

Affected GKE versions: 1.18, 1.19.0 to 1.19.14-gke.1400, 1.20.0 to 1.20.10-gke.1400, 1.21.0 to 1.21.4-gke.1500

GKE clusters running node pools that use containerd might experience IP leak issues and exhaust all the Pod IPs on a node. A Pod scheduled on an affected node displays an error message similar to the following:

failed to allocate for range 0: no IP addresses available in range set: 10.48.131.1-10.48.131.62

For more information about the issue, see containerd issue #5438 and issue #5768.

Workarounds

You can mitigate this issue using the following workarounds:

  • Use Docker-based node pools instead of containerd.
  • Clean up the Pod IP range on the affected node.

To clean up the IP range, connect to the affected node and perform the following steps:

  1. Stop the kubelet and containerd:

    systemctl stop kubelet
    systemctl stop containerd
    
  2. Rename the /var/lib/cni/networks directory to remove the old range:

    mv /var/lib/cni/networks /var/lib/cni/networks.backups
    
  3. Recreate the /var/libs/cni/networks directory:

    mkdir /var/lib/cni/networks
    
  4. Restart containerd and the kubelet:

    systemctl start containerd
    systemctl start kubelet
    

Exec probe behavior difference when probe exceeds the timeout

Affected GKE versions: all

Exec probe behavior on containerd images is different from the behavior on dockershim images. When exec probe, defined for the Pod, exceeds the declared timeoutSeconds threshold, on dockershim images, it is treated as a probe failure. On containerd images, probe results returned after the declared timeoutSeconds threshold are ignored.

Insecure registry option is not configured for local network (10.0.0.0/8)

Affected GKE versions: all

On containerd images the insecure registry option is not configured for local network 10.0.0.0/8. When migrating from images with Docker, which were using the private image registry, ensure the correct certificate is installed on the registry, or the registry is configured to use http.

containerd ignores any device mappings for privileged pods

Affected GKE versions: all

For privileged Pods, the container runtime ignores any device mappings that volumeDevices.devicePath passed to it, and instead makes every device on the host available to the container under /dev.

IPv6 address family is enabled on pods running containerd

Affected GKE versions: 1.18, 1.19, 1.20.0 to 1.20.9

IPv6 image family is enabled for Pods running with containerd. The dockershim image disables IPv6 on all Pods, while the containerd image does not. For example, localhost resolves to IPv6 address ::1 first. This is typically not a problem, however, this might result in unexpected behavior in certain cases.

Currently, the workaround is to use IPv4 address like 127.0.0.1 explicitly, or configure an application running in the Pod to work on both address families.

Node auto-provisioning only provisions Container-Optimized OS with Docker node pools

Affected GKE versions: 1.18, 1.19, 1.20.0 to 1.20.6-gke.1800

Node auto-provisioning allows auto-scaling node pools with any supported image type, but can only create new node pools with the Container-Optimized OS with Docker image type.

Starting in GKE version 1.20.6-gke.1800 and later, the default image type can be set for the cluster.

Conflict with 172.17/16 range

Affected GKE versions: 1.18.0 to 1.18.14

The 172.17/16 IP range is occupied by the docker0 interface on the node VM with containerd enabled. Traffic sending to or originating from that range might not be routed correctly (for example, a Pod might not be able to connect to a VPN-connected host with an IP within 172.17/16).

GPU metrics not collected

Affected GKE versions: 1.18.0 to 1.18.18

GPU usage metrics are not collected when using containerd as a runtime on GKE versions before 1.18.18.

Images with config.mediaType set to application/octet-stream cannot be used on containerd

Affected GKE node versions: All

Images with config.mediaType set to "application/octet-stream" cannot be used on containerd. See https://github.com/containerd/containerd/issues/4756. These images are not compatible with the Open Container Initiative specification and are considered incorrect. These images work with Docker to provide backward compatibility, while in containerd these images are not supported.

Symptom and diagnosis

Example error in node logs:

Error syncing pod <pod-uid> ("<pod-name>_<namespace>(<pod-uid>)"), skipping: failed to "StartContainer" for "<container-name>" with CreateContainerError: "failed to create containerd container: error unpacking image: failed to extract layer sha256:<some id>: failed to get reader from content store: content digest sha256:<some id>: not found"

The image manifest can be usually found in the registry where it is hosted. Once you have the manifest, look at config.mediaType to determine if you have this issue:

"mediaType": "application/octet-stream",

Fix

As the containerd community decided to not support such images, all versions of containerd are affected and there is no fix. The container image must be rebuilt with Docker version 1.11 or later and you must ensure that the config.mediaType field is not set to "application/octet-stream".

Sample migration script

The following sample script iterates over all node pools across available projects, and for each node pool outputs the suggestion on whether the node pool should be migrated to containerd. This script also outputs the node pool version and suggested migration command as listed in the updating your node images section. Make sure that you review the known issues for a node pool version

Sample script: iterate over all node pools for containerd migration

for project in  $(gcloud projects list --format="value(projectId)")
do
  echo "ProjectId:  $project"
  for clusters in $( \
    gcloud container clusters list \
      --project $project \
      --format="csv[no-heading](name,location,autopilot.enabled,currentMasterVersion,autoscaling.enableNodeAutoprovisioning,autoscaling.autoprovisioningNodePoolDefaults.imageType)")
  do
    IFS=',' read -r -a clustersArray <<< "$clusters"
    cluster_name="${clustersArray[0]}"
    cluster_zone="${clustersArray[1]}"
    cluster_isAutopilot="${clustersArray[2]}"
    cluster_version="${clustersArray[3]}"
    cluster_minorVersion=${cluster_version:0:4}
    cluster_autoprovisioning="${clustersArray[4]}"
    cluster_autoprovisioningImageType="${clustersArray[5]}"

    if [ "$cluster_isAutopilot" = "True" ]; then
      echo "  Cluster: $cluster_name (autopilot) (zone: $cluster_zone)"
      echo "    Autopilot clusters are running Containerd."
    else
      echo "  Cluster: $cluster_name (zone: $cluster_zone)"

      if [ "$cluster_autoprovisioning" = "True" ]; then
        if [ "$cluster_minorVersion"  \< "1.20" ]; then
          echo "    Node autoprovisioning is enabled, and new node pools will have image type 'COS'."
          echo "    This settings is not configurable on the current version of a cluster."
          echo "    Please upgrade you cluster and configure the default node autoprovisioning image type."
          echo "    "
        else
          if [ "$cluster_autoprovisioningImageType" = "COS" ]; then
            echo "    Node autoprovisioning is configured to create new node pools of type 'COS'."
            echo "    Run the following command to update:"
            echo "    gcloud container clusters update '$cluster_name' --project '$project' --zone '$cluster_zone' --enable-autoprovisioning --autoprovisioning-image-type='COS_CONTAINERD'"
            echo "    "
          fi

          if [ "$cluster_autoprovisioningImageType" = "UBUNTU" ]; then
            echo "    Node autoprovisioning is configured to create new node pools of type 'UBUNTU'."
            echo "    Run the following command to update:"
            echo "    gcloud container clusters update '$cluster_name' --project '$project' --zone '$cluster_zone' --enable-autoprovisioning --autoprovisioning-image-type='UBUNTU_CONTAINERD'"
            echo "    "
          fi
        fi
      fi

      for nodepools in $( \
        gcloud container node-pools list \
          --project $project \
          --cluster $cluster_name \
          --zone $cluster_zone \
          --format="csv[no-heading](name,version,config.imageType)")
      do
        IFS=',' read -r -a nodepoolsArray <<< "$nodepools"
        nodepool_name="${nodepoolsArray[0]}"
        nodepool_version="${nodepoolsArray[1]}"
        nodepool_imageType="${nodepoolsArray[2]}"

        nodepool_minorVersion=${nodepool_version:0:4}

        echo "    Nodepool: $nodepool_name, version: $nodepool_version ($nodepool_minorVersion), image: $nodepool_imageType"

        minorVersionWithRev="${nodepool_version/-gke./.}"
        linuxGkeMinVersion="1.14"
        windowsGkeMinVersion="1.21.1.2200"

        suggestedImageType="COS_CONTAINERD"

        if [ "$nodepool_imageType" = "UBUNTU" ]; then
          suggestedImageType="UBUNTU_CONTAINERD"
        elif [ "$nodepool_imageType" = "WINDOWS_LTSC" ]; then
          suggestedImageType="WINDOWS_LTSC_CONTAINERD"
        elif [ "$nodepool_imageType" = "WINDOWS_SAC" ]; then
          suggestedImageType="WINDOWS_SAC_CONTAINERD"
        fi

        tab=$'\n      ';
        nodepool_message="$tab Please update the nodepool to use Containerd."
        nodepool_message+="$tab Make sure to consult with the list of known issues https://cloud.google.com/kubernetes-engine/docs/concepts/using-containerd#known_issues."
        nodepool_message+="$tab Run the following command to upgrade:"
        nodepool_message+="$tab "
        nodepool_message+="$tab gcloud container clusters upgrade '$cluster_name' --project '$project' --zone '$cluster_zone' --image-type '$suggestedImageType' --node-pool '$nodepool_name'"
        nodepool_message+="$tab "

        # see https://cloud.google.com/kubernetes-engine/docs/concepts/node-images
        if [ "$nodepool_imageType" = "COS_CONTAINERD" ] || [ "$nodepool_imageType" = "UBUNTU_CONTAINERD" ] ||
           [ "$nodepool_imageType" = "WINDOWS_LTSC_CONTAINERD" ] || [ "$nodepool_imageType" = "WINDOWS_SAC_CONTAINERD" ]; then
          nodepool_message="$tab Nodepool is using Containerd already"
        elif ( [ "$nodepool_imageType" = "WINDOWS_LTSC" ] || [ "$nodepool_imageType" = "WINDOWS_SAC" ] ) &&
               !( [ "$(printf '%s\n' "$windowsGkeMinVersion" "$minorVersionWithRev" | sort -V | head -n1)" = "$windowsGkeMinVersion" ] ); then
          nodepool_message="$tab Upgrade nodepool to the version that supports Containerd for Windows"
        elif !( [ "$(printf '%s\n' "$linuxGkeMinVersion" "$minorVersionWithRev" | sort -V | head -n1)" = "$linuxGkeMinVersion" ] ); then
          nodepool_message="$tab Upgrade nodepool to the version that supports Containerd"
        fi
        echo "$nodepool_message"
      done
    fi # not autopilot
  done
done

# Sample output:
#
# ProjectId:  my-project-id
#  Cluster: autopilot-cluster-1 (autopilot) (zone: us-central1)
#    Autopilot clusters are running Containerd.
#  Cluster: cluster-1 (zone: us-central1-c)
#    Nodepool: default-pool, version: 1.18.12-gke.1210 (1.18), image: COS
#
#       Please update the nodepool to use Containerd.
#       Make sure to consult with the list of known issues https://cloud.google.com/kubernetes-engine/docs/concepts/using-containerd#known_issues.
#       Run the following command to upgrade:
#
#       gcloud container clusters upgrade 'cluster-1' --project 'my-project-id' --zone 'us-central1-c' --image-type 'COS_CONTAINERD' --node-pool 'default-pool'
#
#    Nodepool: pool-1, version: 1.18.12-gke.1210 (1.18), image: COS
#
#       Please update the nodepool to use Containerd.
#       Make sure to consult with the list of known issues https://cloud.google.com/kubernetes-engine/docs/concepts/using-containerd#known_issues.
#       Run the following command to upgrade:
#
#       gcloud container clusters upgrade 'cluster-1' --project 'my-project-id' --zone 'us-central1-c' --image-type 'COS_CONTAINERD' --node-pool 'pool-1'
#
#    Nodepool: winpool, version: 1.18.12-gke.1210 (1.18), image: WINDOWS_SAC
#
#       Upgrade nodepool to the version that supports Containerd for Windows
#
#  Cluster: another-test-cluster (zone: us-central1-c)
#    Nodepool: default-pool, version: 1.20.4-gke.400 (1.20), image: COS_CONTAINERD
#
#      Nodepool is using Containerd already
#

What's next