When you create a Container Engine cluster or node pool, you can choose the operating system image that runs on each node. You can also upgrade an existing cluster to use a different node image type.
Available node images
Container Engine offers the following node image options for your cluster:
- Container-Optimized OS from Google
- Ubuntu (beta)
container-vmopen preview (deprecated)
The Container-Optimized OS node image is based on a recent version of the Linux kernel (4.4 as of December 2016) and is optimized to enhance node security. It is backed by a team at Google that can quickly patch it for security and iterate on features. The Container-Optimized OS image provides better support, security, and stability than previous images.
Container-Optimized OS requires Kubernetes version 1.4.0 or higher.
The Ubuntu node image has been validated against Container Engine's node
image requirements. You should use the Ubuntu node image if your nodes require
support for NFS, glusterfs, Sysdig, or Debian packages. The Ubuntu image
replaces the deprecated
Ubuntu requires Kubernetes version 1.6.4 or higher.
container-vm node image is a legacy node image based on Debian 7. It is
minimally supported and only patched with security fixes that depend on patches
from Debian Long Term Support.
container-vm is deprecated and not recommended for use. The Ubuntu node
container-vm for use cases that require NFS, glusterfs, or
Specifying a node image
You can select the node image you want to use when you create a new cluster, or you can upgrade an existing cluster to use a different node type.
Creating a new cluster
Container-Optimized OS is the default option for a cluster node image. You can
specify the Ubuntu or
container-vm node image by including the
image-type option when you use the
gcloud container clusters create command.
To create a new cluster with Container-Optimized OS as the node image:
gcloud container clusters create [CLUSTER_NAME]
To create a new cluster with Ubuntu as the node image:
gcloud container clusters create [CLUSTER_NAME] --image-type=ubuntu --cluster-version=1.6.4
If you attempt to create a cluster running a Kuberentes version lower than 1.6.4 and the Ubuntu node image, you might encounter the following error:
ERROR: (gcloud.container.clusters.create) ResponseError: code=400, message=The required node image is not available.
To create a new cluster with
container-vm as the node image:
gcloud container clusters create [CLUSTER_NAME] --image-type=container_vm
Upgrading an existing cluster
You can upgrade an existing cluster to use the Container-Optimized OS or Ubuntu
node images by using the
gcloud container clusters upgrade command.
To upgrade an existing cluster to use the Container-Optimized OS node image:
gcloud container clusters upgrade --image-type=cos [CLUSTER_NAME] [--node-pool=POOL_NAME]
To upgrade an existing cluster to use the Ubuntu node image:
gcloud container clusters upgrade --image-type=ubuntu [CLUSTER_NAME] [--node-pool=POOL_NAME]
Node image comparison
The following sections compare the operational aspects of the
Container-Optimized OS, Ubuntu, and
container-vm node images, including:
- Software package management
- System initialization
- Logs collection
- File system layout
- Automatic upgrade and repair
- CPU limits
- Sysdig support
- Storage driver support
Software package manager
cos node image uses a minimal root file system with built-in support for
the Docker container runtime, which also serves as the software package manager
for installing software on the host. The Ubuntu and
container-vm images use
the Aptitude package manager.
Managing software on Container-Optimized OS
You cannot install software packages on a host with the Container-Optimized OS
image (that is, outside of containers) or upgrade software packages
independently. However, the Container-Optimized OS node image includes
some common debugging tools in the image and provides a
toolbox wrapper to run
debugging tools of your choice. Some examples include:
sudo toolbox ping www.google.com sudo toolbox apt-get install psmisc sudo toolbox pstree -p
For additional examples of how to use the wrapper to install additional software
on a host with the
cos node image, see the Container-Optimized OS how-to guides.
Managing software on Ubuntu or
Both the Ubuntu and
container-vm images have the Aptitude package manager
pre-installed. You can use the
apt-get command to install packages on these
images. For example, to install
sudo apt-get update sudo apt-get install ceph
Both the Container-Optimized OS and Ubuntu node image use
systemd to manage
system resources and services during the system initialization process.
container-vm node image can use
rc.d scripts for initialization.
The Container-Optimized OS and Ubuntu node images use
for collecting system-wide logs.
container-vm node image writes system log files to the
directory on the node.
Viewing logs on Container-Optimized OS and Ubuntu
To view logs on a node with the Container-Optimized OS or Ubuntu node image, you
must use the
command. For example, to view Docker daemon logs:
sudo journalctl -u docker
To view kubelet logs:
sudo journalctl -u kubelet
File system layout
The Ubuntu and
container-vm node images use the standard Linux file system
The Container-Optimized OS node image file system layout is optimized to enhance node security. The boot disk space is split into three types of partitions:
- Root partition, which is mounted as read-only
- Stateful partitions, which are writable and stateful
- Stateless partitions, which are writable but the contents do not persist across reboots
When using Container-Optimized OS, be aware of the partitioning if you run your own services that have certain expectations about the filesystem layout outside of containers.
Working with the Container-Optimized OS file system
The following is a list of paths in the Container-Optimized OS node image file system, along with their properties and recommended usage:
||The root filesystem is mounted as read-only to maintain integrity. The kernel verifies integrity root filesystem during boot up, and refuses to boot in case of errors.|
||These paths are meant for storing data that persists for the lifetime of the boot disk.|
||These paths are working directories for Compute Engine packages (for example, the accounts manager service), cloud-init, Docker, and Kubelet respectively.|
||/etc typically holds your configuration (for example, `systemd` services defined via `cloud-init`). It's a good idea to capture the desired state of your instances in `cloud-init`, as `cloud-init` is applied when an instance is newly created as well as when an instance is restarted.|
||/tmp is typically used as a scratch space and should not be used to store persistent data.|
||You can mount Persistent Disks at directories under /mnt/disks.|
Automatic upgrade and repair
Container-Optimized OS supports Kubernetes' and Container Engine's node
auto-upgrade and node auto-repair features. The Ubuntu and
images do not support node auto-upgrade or node auto-repair, but those
features are currently under development for the Ubuntu node image.
CPU limits are
enforced on Container-Optimized OS and Ubuntu nodes. They are not enforced on
Sysdig is not currently supported on
Container-Optimized OS. Sysdig is supported and tested on Ubuntu and
container-vm, but is not installed by default. See the
Sysdig installation instructions for help
installing Sysdig on an Ubuntu or
Storage driver support
Each node image differs in the kinds of storage plugins it supports. The following terms apply when describing a node image's support for a particular storage driver:
- Yes - Fully Tested/Supported: This storage plugin is fully supported and tested with the specified node image.
- Yes - Limited Testing: This storage plugin works with the
specified node image, but have been tested only in a limited fashion; you might
encounter unexpected behavior. For Container-Optimized OS, these plugins will
eventually be fully tested and supported (
- Unsupported: This storage plugin has not been tested or used with the specified node image and Container Engine cannot provide any guarantee of functionality. There are no plans to test this storage plugin.
- No: This storage plugin does not work with the specified node image due to a limitation inherent to the node OS or Google Cloud Platform.
The following matrix describes how each Container Engine node image supports some common storage plugins.
|Volume Type||Does it work on
||Does it work on Container-Optimized OS (
||Does it work on Ubuntu?||Version Notes|
|Google Compute Engine
Persistent Disk (ext4 or xfs)
|Yes - Fully Tested/Supported||Yes - Fully Tested/Supported||Yes - Fully Tested/Supported||Requires Kubernetes 1.4+|
|GlusterFS||Yes - Limited Testing||Yes||Yes - Fully Tested/Supported||Requires Kubernetes 1.4.7+|
|NFSv3||Yes - Limited Testing||Yes||Yes - Fully Tested/Supported||On Container-Optimized OS, requires Kubernetes 1.6.0+|
|NFSv4||Yes - Limited Testing||Yes||Yes - Fully Tested/Supported||Requires Kubernetes 1.4.7+|
|CephFS||No||No||Yes - Fully Tested/Supported||On Ubuntu, you must manually install the
Container-Optimized OS documentation and release notes
Google provides comprehensive documentation for Container-Optimized OS:
Ubuntu node image release notes and package manifest
28 August 2017: ubuntu-gke-1604-xenial-v20170816-1 (package manifest)
- This patch release is based on Ubuntu 16.04.3 LTS.
- It includes a fix for the Stackdriver Logging issues in
2 August 2017: ubuntu-gke-1604-xenial-v20170420-1 (package manifest)
- Beta release of Ubuntu node image, based on version 16.04.2 LTS.
- See the package manifest for more information.
- Known issue: When Stackdriver Logging is enabled in the cluster,
dockerlogs are not exported from the nodes running Ubuntu image. This issue is resolved in the next Ubuntu node image release.