Node images

This page describes the node images available for Google Kubernetes Engine (GKE) nodes.

When you create a GKE cluster or node pool, you can choose the operating system image that runs on each node. You can also upgrade an existing cluster to use a different node image type. For instructions on how to set the node image, see Specifying a node image.

Available node images

GKE offers the following node image options per OS for your cluster:

OS Node images
Container-Optimized OS
Ubuntu
Windows Server

Container-Optimized OS

The Container-Optimized OS from Google node images are based on a recent version of the Linux kernel and are optimized to enhance node security. Container-Optimized OS images are backed by a team at Google that can quickly patch images for security and iterate on features. The Container-Optimized OS images provides better support, security, and stability than other images.

For information about the image project and family, see Feature support by operating system.

Container-Optimized OS variants

Two container runtimes are offered with Container-Optimized OS. The images are the same, other than the choice of container runtime.

  • Container-Optimized OS with Containerd (cos_containerd): The cos_containerd image uses Containerd as the container runtime directly integrated with Kubernetes. For more information, see Using Containerd images.
  • Container-Optimized OS with Docker (cos): The cos image uses the Docker container runtime.

Ubuntu

The Ubuntu node images has been validated against GKE's node image requirements. You should use the Ubuntu node images if your nodes require support for XFS, CephFS, or Debian packages.

For information about the image project and family, see Feature support by operating system.

Ubuntu variants

Two container runtimes are offered with Ubuntu. The images are the same, other than the choice of container runtime.

  • Ubuntu with Containerd (ubuntu_containerd): The ubuntu_containerd image uses Containerd as the container runtime. For more information, see Using Containerd images.

  • Ubuntu with Docker (ubuntu): The ubuntu image uses Docker as the container runtime.

Windows Server LTSC and Windows Server SAC

When creating a cluster using Windows Server node pools you can use a Windows Server Semi-Annual Channel (SAC) or Windows Server Long-Term Servicing Channel (LTSC) node image. All Windows node images are Windows Server Datacenter Core images. A single cluster can have multiple Windows Server node pools using different Windows Server versions, but each individual node pool can only use one Windows Server version. For more information, see Choose your Windows node image.

These images are only available for clusters with versions that are 1.16.8-gke.9 or higher.

For information about the image project and family, see Feature support by operating system.

Linux node image comparison

The following sections compare the operational aspects of the Container-Optimized OS and Ubuntu node images, including:

  • Software package management
  • System initialization
  • Logs collection
  • File system layout
  • Storage driver support

Software package manager

The cos and cos_containerd node images use a minimal root file system with built-in support for the Docker (containerd) container runtime, which also serves as the software package manager for installing software on the host. The Ubuntu image uses the APT package manager.

Managing software on Container-Optimized OS

The Container-Optimized OS image does not provide package management software such as apt-get. You can't install arbitrary software onto the nodes using conventional mechanisms. Instead, create a container image that contains the software you need.

For debugging purposes only, Container-Optimized OS includes the CoreOS Toolbox for installing and running common debugging tools such as ping, psmisc, or pstree.

For more information about debugging Container-Optimized OS nodes, see the Container-Optimized OS how-to guides.

Managing software on Ubuntu

The Ubuntu image uses the APT package manager. You can use the apt-get command to install packages on these images. For example, to install ceph packages:

sudo apt-get update
sudo apt-get install ceph

System initialization

Both the Container-Optimized OS and Ubuntu node image use systemd to manage system resources and services during the system initialization process.

Both node images use systemd service files to define services on the node, and systemd.targets to group boot targets via dependencies.

Logs collection

The Container-Optimized OS and Ubuntu node images use systemd-journald for collecting system-wide logs.

Viewing logs on Container-Optimized OS and Ubuntu

To view logs on a node with the Container-Optimized OS or Ubuntu node image, you must use the journalctl command. For example, to view Docker daemon logs:

sudo journalctl -u docker

To view kubelet logs:

sudo journalctl -u kubelet

File system layout

The Ubuntu node image uses the standard Linux file system layout.

The Container-Optimized OS node image file system layout is optimized to enhance node security. The boot disk space is split into three types of partitions:

  • Root partition, which is mounted as read-only
  • Stateful partitions, which are writable and stateful
  • Stateless partitions, which are writable but the contents do not persist across reboots

When using Container-Optimized OS, be aware of the partitioning if you run your own services that have certain expectations about the filesystem layout outside of containers.

Working with the Container-Optimized OS file system

The following is a list of paths in the Container-Optimized OS node image file system, along with their properties and recommended usage:

Path Properties Purpose
/
  • read-only
  • executable
The root filesystem is mounted as read-only to maintain integrity. The kernel verifies integrity root filesystem during boot up, and refuses to boot in case of errors.
/home
/var
  • writable
  • non-executable
  • stateful
These paths are meant for storing data that persists for the lifetime of the boot disk. They are mounted from /mnt/stateful_partition.
/var/lib/google
/var/lib/docker
/var/lib/toolbox
  • writable
  • executable
  • stateful
These paths are working directories for Compute Engine packages (for example, the accounts manager service), Docker, and Toolbox respectively.
/var/lib/cloud
  • writable
  • executable
  • stateless
  • tmpfs
This path is the working directory of the cloud-init package.
/etc
  • writable
  • non-executable
  • stateless
  • tmpfs
/etc typically holds your configuration (for example, systemd services defined via cloud-init). It's a good idea to capture the desired state of your instances in cloud-init, as cloud-init is applied when an instance is newly created as well as when an instance is restarted.
/tmp
  • writable
  • non-executable
  • stateless
  • tmpfs
/tmp is typically used as a scratch space and should not be used to store persistent data.
/mnt/disks
  • writable
  • executable
  • stateless
  • tmpfs
You can mount Persistent Disks at directories under /mnt/disks.

Storage driver support

Each node image differs in the kinds of storage plugins it supports. The following terms apply when describing a node image's support for a particular storage driver:

  • Yes - Fully Tested/Supported: This storage plugin is fully supported and tested with the specified node image.
  • Yes - Limited Testing: This storage plugin works with the specified node image, but have been tested only in a limited fashion; you might encounter unexpected behavior. For Container-Optimized OS, these plugins will eventually be fully tested and supported.
  • Unsupported: This storage plugin has not been tested or used with the specified node image and GKE cannot provide any guarantee of functionality. There are no plans to test this storage plugin.
  • No: This storage plugin does not work with the specified node image due to a limitation inherent to the node OS or Google Cloud Platform.

The following matrix describes how each GKE node image supports some common storage plugins.

Volume Type Does it work on Container-Optimized OS (cos)? Does it work on Ubuntu?
Google Compute Engine
Persistent Disk (EXT4 or XFS)
Yes - Fully Tested/Supported
(XFS is not supported.)
Yes - Fully Tested/Supported
NFSv3 Yes - Fully Tested/Supported Yes - Fully Tested/Supported
NFSv4 Yes - Fully Tested/Supported Yes - Fully Tested/Supported
CephFS No Yes - Limited Testing
(Driver is not installed by default. You must install the ceph client, preferably via DaemonSet.)
Cinder No No
Fibre Channel No No
Flocker Unsupported Unsupported
iSCSI No No
RBD No No

Node VM modifications

Modifications on the boot disk of a node VM do not persist across node re-creations. Nodes are re-created during manual upgrade, auto-upgrade, auto-repair, and auto-scaling. In addition, nodes are re-created when you enable a feature that requires node re-creation, such as GKE sandbox, intranode visibility, and shielded nodes.

To preserve modifications across node re-creation, use a DaemonSet.

It's not recommended to manage critical software provided by a node image, such as the kernel or container runtime (whether containerd or docker). Node images are tested extensively, and modifying critical software provided in the node image puts the node into an unknown and untestable state.

Node images release notes

Container-Optimized OS

Google provides comprehensive documentation for Container-Optimized OS:

Ubuntu

Periodically, Google updates the Ubuntu images that are available for use on your cluster's Nodes. Refer to the GKE release notes for information about these updates, including a link to a manifest listing the packages that are installed by default.

What's next