Starting with Google Container Engine version 1.4, the default node
image for all Container Engine clusters and node pools has been changed. In
previous versions, the default node image was
container-vm, based on the
Debian 7 operating system. In version 1.4, Container Engine uses a node
image called Container-Optimized OS from Google,
with the image name
cos node image is built on the open source
Chromium OS project, and is optimized for running containers on Google Cloud Platform.
The open preview version of the Debian 7
container-vm is now deprecated.
However, during the lifetime of the Kubernetes 1.4 release, you may opt out of
cos node images in your clusters and continue using the Debian 7-based
container-vm instead. See the opt out instructions for more
The new default node image is primarily a change to the underlying infrastructure
of Container Engine, and should not affect most users in the course of normal
operations. However, the migration to the
cos node image changes some aspects
of how you interact with the nodes in your node pools and clusters.
To upgrade existing clusters to
cos, follow the upgrade instructions.
Why should I migrate?
- The new Container-Optimized OS is based on a recent version of the Linux kernel (4.4 as of December 2016) and is optimized to enhance node security. It is backed by a team at Google that can quickly patch it for security and iterate on features. The Container-Optimized OS image provides better support, security, and stability than the old image based on Debian 7.
- The old
container-vmimage is minimally supported. The
container-vmimage is only patched with security fixes that depend on patches from Debian Long Term Support (LTS), and Debian LTS officially ends support for Debian 7 starting in 2018. Additionally, there is no SLA on how quickly security fixes are applied to Debian LTS. See the LTS page for more detail on Debian LTS.
- CPU limits are enforced in
cos. They were not enforced in
- Sysdig is not yet supported.
Storage Driver Support
container-vm node images differ in the kinds of storage plugins
they support. The following matrix describes how each Container Engine node image
supports using some common storage plugins:
The following terms apply when describing a node image's support for a particular storage driver:
- Yes - Fully Tested/Supported: This storage plugin is fully supported and tested with the specified node image.
- Yes - Limited Testing: This storage plugin works with the
specified node image, but have been tested only in a limited fashion; you might
encounter unexpected behavior. For Container-Optimized OS, these plugins will
eventually be fully tested and supported (
- Unsupported: This storage plugin has not been tested or used with the specified node image and Container Engine cannot provide any guarantee of functionality. There are no plans to test this storage plugin.
- No: This storage plugin does not work with the specified node image due to a limitation inherent to the node OS or Google Cloud Platform.
|Volume Type||Does it work on
||Does it work on Container-Optimized OS (
|Google Compute Engine
|Yes - Fully Tested/Supported||Yes - Fully Tested/Supported||Requires Kubernetes 1.4+|
|GlusterFS||Yes - Limited Testing||Yes||Requires Kubernetes 1.4.7+|
|NFSv3||Yes - Limited Testing||Yes||On COS, requires Kubernetes 1.6.0+|
|NFSv4||Yes - Limited Testing||Yes||Requires Kubernetes 1.4.7+|
Features Updated with Container-Optimized OS
The following features have been updated or changed in the
cos node image:
- Software Package Manager
- System Initialization
- Logs Collection
- Filesystem Layout
Software Package Manager
cos node image uses a minimal root file system with built-in support for the Docker
container runtime, which also serves as the software package manager for
installing software on the host. By contrast, the Debian 7-based
container-vm image used
the Aptitude package manager.
This change may affect you if: You use the Aptitude package manager to
install additional software on your Debian 7-based
Managing Software on Container-Optimized OS
You cannot install software packages on a host with the
cos image (that is, outside of
containers) or upgrade software packages independently. However, the
cos node image includes
some common debugging tools in the image and provides a
toolbox wrapper to run
debugging tools of your choice. Some examples include:
sudo toolbox ping www.google.com sudo toolbox apt-get install psmisc sudo toolbox pstree -p
For additional examples of how to use the wrapper to install additional software
on a host with the
cos node image, see the Container-Optimized OS how-to guides.
cos node image uses
systemd to manage system resources and services during the system
This change may affect you if: Your nodes use DaemonSets or Debian packages
that rely on
Initializing a Container-Optimized OS Node
cos node image uses
for collecting system-wide logs.
This change may affect you if: You access system log files in the
directory on your nodes.
Viewing Logs on Container-Optimized OS
To view logs on a node with the
cos node image, you must use the
journalctl command. For example, to view Docker daemon logs:
sudo journalctl -u docker
To view kubelet logs:
sudo journalctl -u kubelet
File system Layout
cos node image file system layout is optimized to enhance node security. The boot disk
space is split into three types of partitions:
- Root partition, which is mounted as read-only
- Stateful partitions, which are writable and stateful
- Stateless partitions, which are writable but the contents do not persist across reboots
This change may affect you if: You run your own services that have certain expectations about the filesystem layout outside of containers.
Working with the Container-Optimized OS File system
The following is a list of paths in the
cos node image file system, along with their
properties and recommended usage:
||The root filesystem is mounted as read-only to maintain integrity. The kernel verifies integrity root filesystem during boot up, and refuses to boot in case of errors.|
||These paths are meant for storing data that persists for the lifetime of the boot disk.|
||These paths are working directories for Compute Engine packages (for example, the accounts manager service), cloud-init, Docker and Kubelet respectively.|
||/etc typically holds your configuration (for example, `systemd` services defined via `cloud-init`). It's a good idea to capture the desired state of your instances in `cloud-init`, as `cloud-init` is applied when an instance is newly created as well as when an instance is restarted.|
||/tmp is typically used as a scratch space and not for storing persistent data.|
||You can mount Persistent Disks at directories under /mnt/disks.|
Opting out of using Container-Optimized OS
You can opt out of using the
cos node image on your nodes and continue using the
container-vm. To opt out, specify
container_vm as the
when you use the
gcloud command-line tool to create or upgrade a cluster.
To opt out with a new cluster:
gcloud container clusters create --image-type=container_vm [CLUSTER_NAME]
To opt out with an existing cluster:
gcloud container clusters upgrade --image-type=container_vm [CLUSTER_NAME]
Upgrading existing clusters to Container-Optimized OS
The recommended strategy for upgrading existing Container Engine clusters to use the
cos node image is:
Upgrade master version to v1.4.0+
Create a new
Migrate existing workloads to the new
node-poolby including a new node selector selector to your pods. Follow the instructions here to add the new node selector.
Tear down the old
Remove the node selector added in step 3
Alternatively, you can upgrade an existing Container Engine cluster to use the
cos node image by using the
upgrade command specifying
cos for the
gcloud container clusters upgrade --image-type=cos [CLUSTER_NAME] [--node-pool=POOL_NAME]