Node Images

This page describes the node images available for Kubernetes Engine nodes.

Overview

When you create a Kubernetes Engine cluster or node pool, you can choose the operating system image that runs on each node. You can also upgrade an existing cluster to use a different node image type.

Available node images

Kubernetes Engine offers the following node image options for your cluster:

Container-Optimized OS

The Container-Optimized OS node image is based on a recent version of the Linux kernel (4.4 as of December 2016) and is optimized to enhance node security. It is backed by a team at Google that can quickly patch it for security and iterate on features. The Container-Optimized OS image provides better support, security, and stability than previous images.

Container-Optimized OS requires Kubernetes version 1.4.0 or higher.

Ubuntu

The Ubuntu node image has been validated against Kubernetes Engine's node image requirements. You should use the Ubuntu node image if your nodes require support for NFS, glusterfs, Sysdig, or Debian packages. The Ubuntu image replaces the deprecated container-vm image.

Ubuntu requires Kubernetes version 1.6.4 or higher.

container-vm (deprecated)

The container-vm node image is a legacy node image based on Debian 7. It is minimally supported and only patched with security fixes that depend on patches from Debian Long Term Support.

container-vm is deprecated and not recommended for use. The Ubuntu node image replaces container-vm for use cases that require NFS, glusterfs, or Sysdig support.

Specifying a node image

You can select the node image you want to use when you create a new cluster, or you can upgrade an existing cluster to use a different node type.

Creating a new cluster

Console

  1. Visit the Kubernetes Engine menu in the Google Cloud Platform Console.

    Visit the Kubernetes Engine menu

  2. Click Create cluster.

  3. Configure your cluster as desired. Then, from the Node image drop-down menu, select the desired node image.
  4. Click Create.

gcloud

Container-Optimized OS is the default option for a cluster node image. You can specify the Ubuntu or container-vm node image by including the image-type option when you use the gcloud container clusters create command.

To create a new cluster with Container-Optimized OS as the node image:

gcloud container clusters create [CLUSTER_NAME]

To create a new cluster with Ubuntu as the node image:

gcloud container clusters create [CLUSTER_NAME] --image-type=ubuntu --cluster-version=1.6.4

If you attempt to create a cluster running a Kuberentes version lower than 1.6.4 and the Ubuntu node image, you might encounter the following error:

ERROR: (gcloud.container.clusters.create) ResponseError: code=400, message=The required node image is not available.

To create a new cluster with container-vm as the node image:

gcloud container clusters create [CLUSTER_NAME] --image-type=container_vm

Upgrading an existing cluster

Console

  1. Visit the Kubernetes Engine menu in the Google Cloud Platform Console.

    Visit the Kubernetes Engine menu

  2. Select the desired cluster.

  3. Click Edit.
  4. From the Node image drop-down menu, select the desired node image.
  5. Click Save.

gcloud

You can upgrade an existing cluster to use the Container-Optimized OS or Ubuntu node images by using the gcloud container clusters upgrade command.

To upgrade an existing cluster to use the Container-Optimized OS node image:

gcloud container clusters upgrade --image-type=cos [CLUSTER_NAME] [--node-pool=POOL_NAME]

To upgrade an existing cluster to use the Ubuntu node image:

gcloud container clusters upgrade --image-type=ubuntu [CLUSTER_NAME] [--node-pool=POOL_NAME]

Node image comparison

The following sections compare the operational aspects of the Container-Optimized OS, Ubuntu, and container-vm node images, including:

  • Software package management
  • System initialization
  • Logs collection
  • File system layout
  • Automatic upgrade and repair
  • CPU limits
  • Sysdig support
  • Storage driver support

Software package manager

The cos node image uses a minimal root file system with built-in support for the Docker container runtime, which also serves as the software package manager for installing software on the host. The Ubuntu and container-vm images use the Aptitude package manager.

Managing software on Container-Optimized OS

You cannot install software packages on a host with the Container-Optimized OS image (that is, outside of containers) or upgrade software packages independently. However, the Container-Optimized OS node image includes some common debugging tools in the image and provides a toolbox wrapper to run debugging tools of your choice. Some examples include:

sudo toolbox ping www.google.com
sudo toolbox apt-get install psmisc
sudo toolbox pstree -p

For additional examples of how to use the wrapper to install additional software on a host with the cos node image, see the Container-Optimized OS how-to guides.

Managing software on Ubuntu or container-vm

Both the Ubuntu and container-vm images have the Aptitude package manager pre-installed. You can use the apt-get command to install packages on these images. For example, to install ceph packages:

sudo apt-get update
sudo apt-get install ceph

System initialization

Both the Container-Optimized OS and Ubuntu node image use systemd to manage system resources and services during the system initialization process.

Both node images use systemd service files to define services on the node, and systemd.targets to group boot targets via dependencies.

The container-vm node image can use Sysvinit services, runlevels, or rc.d scripts for initialization.

Logs collection

The Container-Optimized OS and Ubuntu node images use systemd-journald for collecting system-wide logs.

The container-vm node image writes system log files to the /var/log directory on the node.

Viewing logs on Container-Optimized OS and Ubuntu

To view logs on a node with the Container-Optimized OS or Ubuntu node image, you must use the journalctl command. For example, to view Docker daemon logs:

sudo journalctl -u docker

To view kubelet logs:

sudo journalctl -u kubelet

File system layout

The Ubuntu and container-vm node images use the standard Linux file system layout.

The Container-Optimized OS node image file system layout is optimized to enhance node security. The boot disk space is split into three types of partitions:

  • Root partition, which is mounted as read-only
  • Stateful partitions, which are writable and stateful
  • Stateless partitions, which are writable but the contents do not persist across reboots

When using Container-Optimized OS, be aware of the partitioning if you run your own services that have certain expectations about the filesystem layout outside of containers.

Working with the Container-Optimized OS file system

The following is a list of paths in the Container-Optimized OS node image file system, along with their properties and recommended usage:

Path Properties Purpose
/
  • read-only
  • executable
The root filesystem is mounted as read-only to maintain integrity. The kernel verifies integrity root filesystem during boot up, and refuses to boot in case of errors.
/home
/mnt/stateful_partition
/var
  • writable
  • non-executable
  • stateful
These paths are meant for storing data that persists for the lifetime of the boot disk.
/var/lib/google
/var/lib/cloud
/var/lib/docker
/var/lib/kubelet
  • writable
  • executable
  • stateful
These paths are working directories for Compute Engine packages (for example, the accounts manager service), cloud-init, Docker, and Kubelet respectively.
/etc
  • writable
  • non-executable
  • stateless
  • tmpfs
/etc typically holds your configuration (for example, `systemd` services defined via `cloud-init`). It's a good idea to capture the desired state of your instances in `cloud-init`, as `cloud-init` is applied when an instance is newly created as well as when an instance is restarted.
/tmp
  • writable
  • non-executable
  • stateless
  • tmpfs
/tmp is typically used as a scratch space and should not be used to store persistent data.
/mnt/disks
  • writable
  • non-executable
  • stateless
You can mount Persistent Disks at directories under /mnt/disks.

Automatic upgrade and repair

Container-Optimized OS supports Kubernetes' and Kubernetes Engine's node auto-upgrade and node auto-repair features. The Ubuntu and container-vm node images do not support node auto-upgrade or node auto-repair, but those features are currently under development for the Ubuntu node image.

CPU limits

CPU limits are enforced on Container-Optimized OS and Ubuntu nodes. They are not enforced on container-vm nodes.

Sysdig support

Sysdig is not currently supported on Container-Optimized OS. Sysdig is supported and tested on Ubuntu and container-vm, but is not installed by default. For more information about installing Sysdig on an Ubuntu or container-vm node, refer to the Sysdig installation instructions.

Storage driver support

Each node image differs in the kinds of storage plugins it supports. The following terms apply when describing a node image's support for a particular storage driver:

  • Yes - Fully Tested/Supported: This storage plugin is fully supported and tested with the specified node image.
  • Yes - Limited Testing: This storage plugin works with the specified node image, but have been tested only in a limited fashion; you might encounter unexpected behavior. For Container-Optimized OS, these plugins will eventually be fully tested and supported (container-vm is deprecated).
  • Unsupported: This storage plugin has not been tested or used with the specified node image and Kubernetes Engine cannot provide any guarantee of functionality. There are no plans to test this storage plugin.
  • No: This storage plugin does not work with the specified node image due to a limitation inherent to the node OS or Google Cloud Platform.

The following matrix describes how each Kubernetes Engine node image supports some common storage plugins.

Volume Type Does it work on container-vm? Does it work on Container-Optimized OS (cos)? Does it work on Ubuntu? Version Notes
Google Compute Engine
Persistent Disk (ext4 or xfs)
Yes - Fully Tested/Supported Yes - Fully Tested/Supported Yes - Fully Tested/Supported Requires Kubernetes 1.4+
GlusterFS Yes - Limited Testing Yes Yes - Fully Tested/Supported Requires Kubernetes 1.4.7+
NFSv3 Yes - Limited Testing Yes Yes - Fully Tested/Supported On Container-Optimized OS, requires Kubernetes 1.6.0+
NFSv4 Yes - Limited Testing Yes Yes - Fully Tested/Supported Requires Kubernetes 1.4.7+
CephFS No No Yes - Fully Tested/Supported On Ubuntu, you must manually install the ceph package
Cinder No No No n/a
Fibre Channel No No No n/a
Flocker Unsupported Unsupported Unsupported n/a
iSCSI Unsupported No No Removed from container-vm in v20161208
RBD No No No n/a

Container-Optimized OS documentation and release notes

Google provides comprehensive documentation for Container-Optimized OS:

Ubuntu node image release notes and package manifest

  • 28 August 2017: ubuntu-gke-1604-xenial-v20170816-1 (package manifest)

    • This patch release is based on Ubuntu 16.04.3 LTS.
    • It includes a fix for the Stackdriver Logging issues in ubuntu-gke-1604-xenial-v20170420-1.
  • 2 August 2017: ubuntu-gke-1604-xenial-v20170420-1 (package manifest)

    • Beta release of Ubuntu node image, based on version 16.04.2 LTS.
    • See the package manifest for more information.
    • Known issue: When Stackdriver Logging is enabled in the cluster, kubelet and docker logs are not exported from the nodes running Ubuntu image. This issue is resolved in the next Ubuntu node image release.

What's next

Monitor your resources on the go

Get the Google Cloud Console app to help you manage your projects.

Send feedback about...

Kubernetes Engine