This page provides additional information about using
Container-Optimized OS with Containerd (cos_containerd
)
and
Ubuntu with Containerd (ubuntu_containerd
)
on Google Kubernetes Engine (GKE) nodes. The cos_containerd
and ubuntu_containerd
images let you use Containerd
as the container runtime in your GKE cluster.
About Containerd
The container runtime is software that is responsible for running containers, and abstracts container management for Kubernetes. There are a few different container runtimes. Containerd is an industry-standard container runtime that's supported by Kubernetes, and used by many other projects. Containerd provides the layering abstraction that allows for the implementation of a rich set of features like gVisor to extend Kubernetes functionality. Containerd is considered more resource efficient and secure when compared to the Docker runtime.
Using Containerd images in GKE clusters
You can select cos_containerd
or ubuntu_containerd
as the image type when
you create a new GKE cluster, create a new node pool in
an existing cluster, or when you upgrade an existing cluster. Both Containerd
images require GKE version 1.14.3 or later.
Migrating from the Docker runtime to the Containerd runtime
Most user workloads don't have a dependency on the container runtime. The container runtime is what runs the containers in your Pods, and the Docker runtime actually uses Containerd under the hood, so both runtimes behave similarly.
Even if you use Docker on your developer machine, or as part of a build pipeline that runs outside of your cluster to build and push your images, this itself is not a dependency on the Docker runtime (as these actions happen outside of the cluster).
There are a few instances when you might have a dependency on Docker: running privileged Pods executing Docker commands, running scripts on nodes outside of Kubernetes infrastructure (for example, using ssh to troubleshoot issues), or through third-party tools that perform such similarly privileged operations. You might also have an indirect dependency on Docker if some of your tooling was configured to react to Docker-specific log messages in your monitoring system.
We recommend first deploying your workload on a node pool with Containerd to verify if everything runs as expected. If you have a canary, or staging cluster, this would be a great choice to migrate first. You may also want to consult with any vendors who supply logging and monitoring, or continuous integration tooling that you deploy inside your cluster, to confirm compatibility with Containerd.
Updating your node images
You can migrate nodes from a Docker runtime image to a Containerd image by
updating the node pool and setting a different image. This migration can be done
using the Google Cloud Console or the gcloud
tool.
Console
Visit the Google Kubernetes Engine menu in Cloud Console.
Beside your cluster, click more_vert Actions, then click Edit.
In the Node pools section, select the desired node pool.
In the Node pools details page, click Edit.
In the Image type section, click Change.
Select one of the Containerd image variants for your operating system.
Click Change.
Wait for the nodes to upgrade.
gcloud
In the gcloud
tool, you can update a node pool by using the
gcloud container clusters upgrade
command and specifying the --image-type
parameter.
For example, to change a node pool's image to Container-Optimized OS with Containerd, run the following command:
gcloud container clusters upgrade CLUSTER_NAME --image-type COS_CONTAINERD \
--node-pool POOL_NAME
If after updating your node image you notice a problem and need to revert back to the Docker image variants, you can perform this same command but select a Docker image variant.
Refer to the gcloud container clusters upgrade
API documentation for more details.
Running Docker commands on Containerd nodes
While the Docker binary is currently available on Containerd nodes, we do not recommend using it after you migrate to Containerd. Docker does not manage the containers Kubernetes runs on Containerd nodes, thus you cannot use it to view or interact with running Kubernetes containers using Docker commands or the Docker API.
Troubleshooting containers on Containerd nodes
For debugging or troubleshooting on the node, you can interact with Containerd
using the portable command-line tool built for Kubernetes container runtimes:
crictl
. crictl
supports common functionalities to view containers and images,
read logs, and execute commands in the containers. Refer to the
crictl user guide
for the complete set of supported features and usage information.
Accessing Docker Engine from privileged Pods
Some users currently access Docker Engine on a node from within privileged Pods. It is recommended that you update your workloads so that they do not rely on Docker directly. For example, if you currently extract application logs or monitoring data from Docker Engine, consider using GKE system add-ons for logging and monitoring instead.
Building images
Containerd does not support building images, because the feature is not supported by Kubernetes itself.
Kubernetes is not aware of system resources used by local processes outside the scope of Kubernetes, and the Kubernetes control plane cannot account for those processes when allocating resources. This can starve your GKE workloads of resources or cause instability on the node. For this reason, it is not recommended to run commands on local nodes. Instead, consider accomplishing these tasks using other services outside the scope of the individual container, such as Cloud Build, or use a tool such as kaniko to build images as a Kubernetes workload.
If none of these suggestions work for you, and you understand the risks, you can continue using Docker to build images. You need to push the images to a registry before attempting to use them in a GKE cluster. Kubernetes is not aware of locally-built images.
Known issues
Conflict with 172.17/16 range
The 172.17/16 IP range is occupied by the docker0
interface on the node VM
with Containerd enabled. Traffic sending to or originating from that range might
not be routed correctly (for example, a Pod might not be able to connect to a
VPN-connected host with an IP within 172.17/16).
GPU metrics not collected
GPU usage metrics are not collected when using containerd as a runtime.
Issue fixed on GKE v1.19.4-gke.1300+.
Image metrics missing labels
Image metrics container_fs_usage_bytes
and container_tasks_state
don't
show labels like image
, container_name
, name namespace
.
What's next
- Learn more about the Containerd integration in the Kubernetes 1.11 announcement. For even more information, visit the documentation for Containerd and the CRI plugins.
- Read about DockerShim deprecation by Kubernetes.
- Learn how you can secure your apps with the gVisor on Containerd.
- Read about the benefits of using Cloud Build to build images securely and reliably on Google Cloud to replace a custom solution that might require Docker.