Debugging node issues using toolbox

You might need to install additional packages or tools on Container-Optimized OS for certain tasks, such as debugging. Although Container-Optimized OS does not include a package manager, you can use the pre-installed toolbox utility to install any additional packages or tools you require. Using /usr/bin/toolbox is the preferred method for installing and running one-off debugging tools.

/usr/bin/toolbox essentially provides you a shell in a Debian chroot-like environment. When you invoke /usr/bin/toolbox, it runs following commands:

  1. docker pull and docker create to set up the environment. These are only run the first time you invoke /usr/bin/toolbox.
  2. systemd-nspawn to run the given command or (in absence of any command) provides you a shell

toolbox has some other properties to keep in mind:

  • Invoking toolbox after the first invocation does not require a working Docker daemon, nor does it incur any network/disk overhead.
  • The toolbox environment is set up once for each user invoking it. Running sudo toolbox sets it up for root user.
  • The toolbox environment is created under /var/lib/toolbox and is persistent across reboots.
  • You can access sections of the root filesystem, such as user home directories, from inside the toolbox environment.

Customizing toolbox for your deployment

In the following examples, we assume that your user name is USER and your Container-Optimized OS instance name is cos-dev.

You can customize the Docker image that toolbox uses, as well as the paths available to toolbox in the root filesystem. These settings are located in the file /etc/default/toolbox. The default /etc/default/toolbox file typically resembles the following:

USER@cos-dev ~ $ cat /etc/default/toolbox
# Copyright 2016 The Chromium OS Authors. All rights reserved.
# Use of this source code is governed by a BSD-style license that can be
# found in the LICENSE file.

TOOLBOX_BIND="--bind=/:/media/root/ --bind=/mnt/disks/:/media/root/mnt/disks/ --bind=/var/:/media/root/var/ --bind=/home:/media/root/home/ --bind=/etc/resolv.conf:/etc/resolv.conf"
: ${USER:=root}
  • The TOOLBOX_DOCKER_IMAGE and TOOLBOX_DOCKER_TAG variables specify the Docker image to be used. The default comes with some of the common tools like the gcloud command-line tool pre-installed.
  • The TOOLBOX_BIND variable specifies the paths from rootfs to be made available inside the toolbox environment.

To change the default settings, modify the /etc/default/toolbox file, or specify new values for the variables in ${HOME}/.toolboxrc for the appropriate user. For example, assuming that you want toolbox to use fedora:latest as its container, you can run the following commands:

USER@cos-dev ~ $ echo "TOOLBOX_DOCKER_IMAGE=fedora" > "${HOME}/.toolboxrc"
USER@cos-dev ~ $ echo "TOOLBOX_DOCKER_TAG=latest" >> "${HOME}/.toolboxrc"
USER@cos-dev ~ $ toolbox

Installing and running tools from toolbox

Once you've invoked the toolbox utility to start the shell, you can use apt-get inside the resulting container to install packages. For example:

# Inside the toolbox shell
USER@cos-dev ~ $ toolbox
root@cos-dev:~# apt-get update && apt-get install -y htop psmisc
root@cos-dev:~# htop
root@cos-dev:~# pstree -p
root@cos-dev:~# exit

You do not always have to enter toolbox to run commands in it. You can simply prefix your commands with toolbox. For example, to install and run the strace utility to trace the execution of the Docker daemon, you can do the following:

USER@cos-dev ~ $ toolbox apt-get install -y strace
USER@cos-dev ~ $ toolbox strace -p `pidof dockerd`

The Cloud SDK is pre-installed in toolbox:

USER@cos-dev ~ $ toolbox
root@cos-dev:~# which gcloud
# View installed components
root@cos-dev:~# gcloud components list

Your current Cloud SDK version is: 287.0.0
The latest available version is: 295.0.0

Keep in mind that gcloud can only access resources that the VM instance is authorized to access. Read the Compute Engine documentation to learn more about provisioning VM instances with access to other resources.

Read the Cloud SDK documentation to learn more about using gcloud.

Getting files into and out of toolbox

The root filesystem of the host is accessible inside toolbox through the /media/root path. On the host, the root directory of toolbox is accessible through the following path: /var/lib/toolbox/USER-gcr.io_google-containers_toolbox-VERSION/root where USER is your user name and VERSION is the toolbox version number (for example, 20200404-00).

The following examples show how to get files into and out of toolbox:

# Access the host filesystem inside the toolbox
USER@cos-dev ~ $ toolbox
root@cos-dev:~# ls /media/root
bin  boot  dev  etc  home  lib  lib64  ...
root@cos-dev:~# cp /media/root/home/USER/some-file .

# Access toolbox directory from the host
USER@cos-dev ~ $ sudo cp some-file /var/lib/toolbox/USER-gcr.io_google-containers_toolbox-20200404-00/root

# Run a command inside toolbox and save its output in your home directory
USER@cos-dev ~ $ toolbox strace -o /media/root/$HOME/ls.strace ls
USER@cos-dev ~ $ more $HOME/ls.strace