Container-Optimized OS Migration Guide

Starting with Google Container Engine version 1.4, the default node image for all Container Engine clusters and node pools has been changed. In previous versions, the default node image was container-vm, based on the Debian 7 operating system. In version 1.4, Container Engine uses a node image called Container-Optimized OS from Google, with the image name cos. The cos node image is built on the open source Chromium OS project, and is optimized for running containers on Google Cloud Platform.

The open preview version of the Debian 7 container-vm is now deprecated. However, during the lifetime of the Kubernetes 1.4 release, you may opt out of using cos node images in your clusters and continue using the Debian 7-based container-vm instead. See the opt out instructions for more information.

The new default node image is primarily a change to the underlying infrastructure of Container Engine, and should not affect most users in the course of normal operations. However, the migration to the cos node image changes some aspects of how you interact with the nodes in your node pools and clusters.

To upgrade existing clusters to cos, follow the upgrade instructions.

Why should I migrate?

  • The new Container-Optimized OS is based on a recent version of the Linux kernel (4.4 as of December 2016) and is optimized to enhance node security. It is backed by a team at Google that can quickly patch it for security and iterate on features. The Container-Optimized OS image provides better support, security, and stability than the old image based on Debian 7.
  • The old container-vm image is minimally supported. The container-vm image is only patched with security fixes that depend on patches from Debian Long Term Support (LTS), and Debian LTS officially ends support for Debian 7 starting in 2018. Additionally, there is no SLA on how quickly security fixes are applied to Debian LTS. See the LTS page for more detail on Debian LTS.

Behavioral Differences

  • CPU limits are enforced in cos. They were not enforced in container-vm.

Known Limitations

  • Sysdig is not yet supported.

Storage Driver Support

The cos and container-vm node images differ in the kinds of storage plugins they support. The following matrix describes how each Container Engine node image supports using some common storage plugins:

The following terms apply when describing a node image's support for a particular storage driver:

  • Yes - Fully Tested/Supported: This storage plugin is fully supported and tested with the specified node image.
  • Yes - Limited Testing: This storage plugin works with the specified node image, but have been tested only in a limited fashion; you might encounter unexpected behavior. For Container-Optimized OS, these plugins will eventually be fully tested and supported (container-vm is deprecated).
  • Unsupported: This storage plugin has not been tested or used with the specified node image and Container Engine cannot provide any guarantee of functionality. There are no plans to test this storage plugin.
  • No: This storage plugin does not work with the specified node image due to a limitation inherent to the node OS or Google Cloud Platform.
Volume Type Does it work on container-vm? Does it work on Container-Optimized OS (cos)? Version Notes
Google Compute Engine
Persistent Disk
Yes - Fully Tested/Supported Yes - Fully Tested/Supported Requires Kubernetes 1.4+
GlusterFS Yes - Limited Testing Yes Requires Kubernetes 1.4.7+
NFSv3 Yes - Limited Testing Yes On COS, requires Kubernetes 1.6.0+
NFSv4 Yes - Limited Testing Yes Requires Kubernetes 1.4.7+
CephFS No No n/a
Cinder No No n/a
Fibre Channel No No n/a
Flocker Unsupported Unsupported n/a
iSCSI Unsupported No Removed from container-vm in v20161208
RBD No No n/a

Features Updated with Container-Optimized OS

The following features have been updated or changed in the cos node image:

  • Software Package Manager
  • System Initialization
  • Logs Collection
  • Filesystem Layout

Software Package Manager

The cos node image uses a minimal root file system with built-in support for the Docker container runtime, which also serves as the software package manager for installing software on the host. By contrast, the Debian 7-based container-vm image used the Aptitude package manager.

This change may affect you if: You use the Aptitude package manager to install additional software on your Debian 7-based container-vm nodes.

Managing Software on Container-Optimized OS

You cannot install software packages on a host with the cos image (that is, outside of containers) or upgrade software packages independently. However, the cos node image includes some common debugging tools in the image and provides a toolbox wrapper to run debugging tools of your choice. Some examples include:

sudo toolbox ping
sudo toolbox apt-get install psmisc
sudo toolbox pstree -p

For additional examples of how to use the wrapper to install additional software on a host with the cos node image, see the Container-Optimized OS how-to guides.

System Initialization

The cos node image uses systemd to manage system resources and services during the system initialization process.

This change may affect you if: Your nodes use DaemonSets or Debian packages that rely on Sysvinit services, runlevels, or rc.d scripts.

Initializing a Container-Optimized OS Node

The cos node image uses systemd service files to define services on the node, and systemd.targets to group boot targets via dependencies.

Logs Collection

The cos node image uses systemd-journald for collecting system-wide logs.

This change may affect you if: You access system log files in the /var/log directory on your nodes.

Viewing Logs on Container-Optimized OS

To view logs on a node with the cos node image, you must use the journalctl command. For example, to view Docker daemon logs:

sudo journalctl -u docker

To view kubelet logs:

sudo journalctl -u kubelet

File system Layout

The cos node image file system layout is optimized to enhance node security. The boot disk space is split into three types of partitions:

  • Root partition, which is mounted as read-only
  • Stateful partitions, which are writable and stateful
  • Stateless partitions, which are writable but the contents do not persist across reboots

This change may affect you if: You run your own services that have certain expectations about the filesystem layout outside of containers.

Working with the Container-Optimized OS File system

The following is a list of paths in the cos node image file system, along with their properties and recommended usage:

Path Properties Purpose
  • read-only
  • executable
The root filesystem is mounted as read-only to maintain integrity. The kernel verifies integrity root filesystem during boot up, and refuses to boot in case of errors.
  • writable
  • non-executable
  • stateful
These paths are meant for storing data that persists for the lifetime of the boot disk.
  • writable
  • executable
  • stateful
These paths are working directories for Compute Engine packages (for example, the accounts manager service), cloud-init, Docker and Kubelet respectively.
  • writable
  • non-executable
  • stateless
  • tmpfs
/etc typically holds your configuration (for example, `systemd` services defined via `cloud-init`). It's a good idea to capture the desired state of your instances in `cloud-init`, as `cloud-init` is applied when an instance is newly created as well as when an instance is restarted.
  • writable
  • non-executable
  • stateless
  • tmpfs
/tmp is typically used as a scratch space and not for storing persistent data.
  • writable
  • non-executable
  • stateless
You can mount Persistent Disks at directories under /mnt/disks.

Opting out of using Container-Optimized OS

You can opt out of using the cos node image on your nodes and continue using the Debian 7-based container-vm. To opt out, specify container_vm as the image-type when you use the gcloud command-line tool to create or upgrade a cluster.

  • To opt out with a new cluster:

    gcloud container clusters create --image-type=container_vm [CLUSTER_NAME]
  • To opt out with an existing cluster:

    gcloud container clusters upgrade --image-type=container_vm [CLUSTER_NAME]

Upgrading existing clusters to Container-Optimized OS

The recommended strategy for upgrading existing Container Engine clusters to use the cos node image is:

  1. Upgrade master version to v1.4.0+

  2. Create a new node-pool

  3. Migrate existing workloads to the new node-pool by including a new node selector selector to your pods. Follow the instructions here to add the new node selector.

  4. Tear down the old node-pool

  5. Remove the node selector added in step 3

Alternatively, you can upgrade an existing Container Engine cluster to use the cos node image by using the upgrade command specifying cos for the image-type option.

gcloud container clusters upgrade --image-type=cos [CLUSTER_NAME] [--node-pool=POOL_NAME]

Send feedback about...

Container Engine