This page describes the node images available for Google Kubernetes Engine nodes. To learn how to choose a node image, refer to Specifying a Node Image.
Overview
When you create a GKE cluster or node pool, you can choose the operating system image that runs on each node. You can also upgrade an existing cluster to use a different node image type.
Available node images
GKE offers the following node image options for your cluster:
- Container-Optimized OS from Google
- Container-Optimized OS with containerd (
cos_containerd) - Ubuntu
Container-Optimized OS
The Container-Optimized OS node image is based on a recent version of the Linux kernel and is optimized to enhance node security. It is backed by a team at Google that can quickly patch it for security and iterate on features. The Container-Optimized OS image provides better support, security, and stability than other images.
Container-Optimized OS with containerd (cos_containerd)
Containerd is an important building block, and the core runtime component of Docker.
cos_containerd is a variant of the Container-Optimized OS image with containerd
as the main container runtime directly integrated with Kubernetes.
For debugging or troubleshooting on the node, you can interact with containerd
using the portable command-line tool built for Kubernetes container runtimes:
crictl. crictl supports common functionalities to view containers and images,
read logs, and execute commands in the containers. Please see the crictl user guide
for the complete set of supported features and usage information.
cos_containerd requires Kubernetes version 1.11.0 or higher.
Ubuntu
The Ubuntu node image has been validated against GKE's node image requirements. You should use the Ubuntu node image if your nodes require support for XFS, CephFS, Sysdig, or Debian packages.
Node image comparison
The following sections compare the operational aspects of the Container-Optimized OS and Ubuntu node images, including:
- Automatic upgrade and repair
- Software package management
- System initialization
- Logs collection
- File system layout
- Sysdig support
- Storage driver support
Automatic upgrade and repair
Container-Optimized OS supports GKE's node auto-upgrade and node auto-repair features.
The Ubuntu node image does not support node auto-upgrade or node auto-repair, but those features are currently under development.
Software package manager
The cos and cos_containerd node images use a minimal root file system with built-in support for
the Docker (containerd) container runtime, which also serves as the software package manager
for installing software on the host. The Ubuntu image uses the
Aptitude package manager.
Managing software on Container-Optimized OS
You cannot install software packages on a host with the Container-Optimized OS
image (that is, outside of containers) or upgrade software packages
independently. However, the Container-Optimized OS node image includes
some common debugging tools in the image and provides a toolbox wrapper to run
debugging tools of your choice. Some examples include:
sudo toolbox ping www.google.com
sudo toolbox apt-get install psmisc
sudo toolbox pstree -p
For additional examples of how to use the wrapper to install additional software
on a host with the cos node image, see the
Container-Optimized OS how-to guides.
Managing software on Ubuntu
The Ubuntu image has the Aptitude package manager pre-installed. You can use
the apt-get command to install packages on these images. For example, to
install ceph packages:
sudo apt-get update
sudo apt-get install ceph
System initialization
Both the Container-Optimized OS and Ubuntu node image use systemd to manage
system resources and services during the system initialization process.
Both node images use systemd service files
to define services on the node, and
systemd.targets
to group boot targets via dependencies.
Logs collection
The Container-Optimized OS and Ubuntu node images use systemd-journald
for collecting system-wide logs.
Viewing logs on Container-Optimized OS and Ubuntu
To view logs on a node with the Container-Optimized OS or Ubuntu node image, you
must use the journalctl
command. For example, to view Docker daemon logs:
sudo journalctl -u docker
To view kubelet logs:
sudo journalctl -u kubelet
File system layout
The Ubuntu node image uses the standard Linux file system layout.
The Container-Optimized OS node image file system layout is optimized to enhance node security. The boot disk space is split into three types of partitions:
- Root partition, which is mounted as read-only
- Stateful partitions, which are writable and stateful
- Stateless partitions, which are writable but the contents do not persist across reboots
When using Container-Optimized OS, be aware of the partitioning if you run your own services that have certain expectations about the filesystem layout outside of containers.
Working with the Container-Optimized OS file system
The following is a list of paths in the Container-Optimized OS node image file system, along with their properties and recommended usage:
| Path | Properties | Purpose |
|---|---|---|
| / |
|
The root filesystem is mounted as read-only to maintain integrity. The kernel verifies integrity root filesystem during boot up, and refuses to boot in case of errors. |
| /home /var |
|
These paths are meant for storing data that persists for the lifetime of the boot disk. They are mounted from /mnt/stateful_partition. |
| /var/lib/google /var/lib/cloud /var/lib/docker /var/lib/kubelet /var/lib/toolbox |
|
These paths are working directories for Compute Engine packages (for example, the accounts manager service), cloud-init, Docker, Kubelet, and Toolbox respectively. |
| /etc |
|
/etc typically holds your configuration (for example,
systemd services defined via cloud-init).
It's a good idea to capture the desired state of your instances in
cloud-init, as cloud-init is applied when an
instance is newly created as well as when an instance is
restarted. |
| /tmp |
|
/tmp is typically used as a scratch space and should not be used to store persistent data. |
| /mnt/disks |
|
You can mount Persistent Disks at directories under /mnt/disks. |
Sysdig support
Sysdig is not currently supported on Container-Optimized OS.
Sysdig is supported and tested on Ubuntu but is not installed by default. For more information about installing Sysdig on an Ubuntu node, refer to the Sysdig installation instructions on Kubernetes.
Storage driver support
Each node image differs in the kinds of storage plugins it supports. The following terms apply when describing a node image's support for a particular storage driver:
- Yes - Fully Tested/Supported: This storage plugin is fully supported and tested with the specified node image.
- Yes - Limited Testing: This storage plugin works with the specified node image, but have been tested only in a limited fashion; you might encounter unexpected behavior. For Container-Optimized OS, these plugins will eventually be fully tested and supported.
- Unsupported: This storage plugin has not been tested or used with the specified node image and GKE cannot provide any guarantee of functionality. There are no plans to test this storage plugin.
- No: This storage plugin does not work with the specified node image due to a limitation inherent to the node OS or Google Cloud Platform.
The following matrix describes how each GKE node image supports some common storage plugins.
| Volume Type | Does it work on Container-Optimized OS (cos)? |
Does it work on Ubuntu? |
|---|---|---|
| Google Compute Engine Persistent Disk (EXT4 or XFS) |
Yes - Fully Tested/Supported (XFS is not supported.) |
Yes - Fully Tested/Supported |
| GlusterFS | Yes - Fully Tested/Supported (XFS is not supported.) |
Yes - Fully Tested/Supported |
| NFSv3 | Yes - Fully Tested/Supported | Yes - Fully Tested/Supported |
| NFSv4 | Yes - Fully Tested/Supported | Yes - Fully Tested/Supported |
| CephFS | No | Yes - Limited Testing (Driver is not installed by default. You must install the ceph client, preferably via DaemonSet.) |
| Cinder | No | No |
| Fibre Channel | No | No |
| Flocker | Unsupported | Unsupported |
| iSCSI | No | No |
| RBD | No | No |
Container-Optimized OS documentation and release notes
Google provides comprehensive documentation for Container-Optimized OS:
Ubuntu node image release notes and package manifest
Periodically, Google updates the Ubuntu images that are available for use on your cluster's Nodes. Refer to the GKE release notes for information about these updates, including a link to a manifest listing the packages that are installed by default.
