Deploying containers on VMs and MIGs

You can configure a virtual machine (VM) instance or an instance template to deploy and launch a Docker container. Compute Engine supplies an up-to-date Container-Optimized OS (COS) image with Docker installed and launches your container when your VM starts.

Before you begin

• If you want to use the command-line examples in this guide:
  1. Install or update to the latest version of the gcloud command-line tool.
  2. Set a default region and zone.
• If you aren't familiar with containers, read What are containers and their benefits.
• If you aren't familiar with Docker, read the Docker documentation.
• Read about Container-Optimized OS.
• Read about managed instance groups (MIGs).

Choosing to deploy containers on VMs and MIGs

By deploying containers on Compute Engine, you can simplify app deployment while controlling your VM infrastructure.

• Manage VMs that are running containers in the same way you would treat any other VM when configuring and managing your Compute Engine infrastructure.
• Create scalable services using managed instance groups (MIGs) running containers, which offer features like autoscaling, autohealing, rolling updates, multi-zone deployments, and load balancing.
• Use familiar processes and tools such as the gcloud command-line tool or the Compute Engine API to manage your VMs with containers.

Alternatively, you might consider deploying to Google Kubernetes Engine to:

• Run a large number of microservices
• Have faster container startup time
• Take advantage of Kubernetes automated orchestration, including auto upgrades, node auto repair, and autoscaling

Running each microservice on a separate virtual machine (VM) on Compute Engine could make the operating system overhead a significant part of your cost. Google Kubernetes Engine lets you deploy multiple containers and groups of containers for each VM instance, which can allocate host VM resources more efficiently to microservices with a smaller footprint.

How deploying containers on Compute Engine works

The common methods of deploying software onto a Compute Engine VM instance include:

• Deploying software on VM boot using a startup script or cloud-init.
• Creating a custom boot disk image with software pre-installed.

Both of the above methods combine the tasks of configuring the app and setting up the operating system environment. As the developer, you must carefully track and resolve any runtime dependencies. For example, if two apps running on a VM use different versions of the same library, you must install both versions and point to them through system variables.

A VM instance with apps deployed directly to the operating system

Alternatively, you can deploy software in a container onto a VM instance or to a MIG. A container carries both application software and the required libraries and is isolated from OS apps and libraries. A container can be easily moved between deployment environments without dealing with conflicting library versions in the container and its OS.

A VM instance with apps deployed in a container

The following process describes how you deploy a container on Compute Engine:

1. You bundle your app and required libraries into a Docker image and publish the image to Artifact Registry, Container Registry, or a third-party registry such Docker Hub.
2. You specify a Docker image name and the docker run configuration when creating a VM instance or an instance template for a MIG.

Note: As of November 1, 2020, Docker Hub rate limits apply to unauthenticated or authenticated pull requests on the Docker Free plan. To avoid disruptions and have greater control over your software supply chain, you can migrate your dependencies to Container Registry or Artifact Registry.

Compute Engine executes the following tasks after you make a request to create a VM instance:

1. Compute Engine creates a VM instance that uses a Google-provided Container-Optimized OS image. This image includes a Docker runtime and additional software that is responsible for starting your container.
2. Compute Engine stores your container settings in instance metadata under the gce-container-declaration metadata key.
3. When the VM starts, the Container-Optimized OS image uses the docker run command configuration that is stored in the instance's metadata, pulls the container image from the repository, and starts the container.

Steps to create a VM instance or a managed instance group running a container

Limitations

• You can only deploy one container for each VM instance. Consider Google Kubernetes Engine if you need to deploy multiple containers per VM instance.
• You can only deploy containers from a public repository or from a private Container Registry repository that you have access to. Other private repositories are not supported.
• You can't map a VM instance's ports to the container's ports (Docker's -p option).
• You can only use Container-Optimized OS images with this deployment method.
• You can only use this feature through the Google Cloud Console or the gcloud command-line tool, not the API.

Preparing a container for deployment

Choose one of the following approaches to make your container image accessible to Compute Engine:

• Upload your Docker image to Container Registry.
• Use any publicly available container images from Docker Hub or other registries.

Note: Consider running your container image builds on Cloud Build, with none of the overhead of managing your own build servers.

Deploying a container on a new VM instance

You can deploy a container on a new VM instance by using the Google Cloud Console or the gcloud command-line tool.

Console

The following example deploys a container from a Google-provided Nginx Docker image, https://gcr.io/cloud-marketplace/google/nginx1:latest, to a VM instance. To use a different Docker image, specify the image you want instead in the examples below.

1. In the Google Cloud Console, go to the VM instances page.

   Go to the VM instances page

2. Click Create instance to create an instance.

3. Under Container, select Deploy container image.

4. Under Container image, specify a container image name and configure options to run the container if desired. For example, you can specify gcr.io/cloud-marketplace/google/nginx1:latest for the container image.

5. Click Create.

gcloud

Use the gcloud compute instances create-with-container command:

gcloud compute instances create-with-container VM_NAME \
    --container-image DOCKER_IMAGE

For example, the following command creates a new VM instance named nginx-vm, which launches and runs the Docker image gcr.io/cloud-marketplace/google/nginx1:latest.

gcloud compute instances create-with-container nginx-vm \
    --container-image gcr.io/cloud-marketplace/google/nginx1:latest

Learn more about the gcloud compute instances create-with-container command.

You must always specify a full Docker image name when using a public image from Docker Hub. For example, specify the following image name to deploy an Apache container image:

docker.io/httpd:2.4

Updating a container on a VM instance

You can update a Docker image and configuration options to run the container on a VM instance using Google Cloud Console or the gcloud command-line tool.

When you update a VM running a container, Compute Engine performs two steps:

• Updates container declaration on the instance. Compute Engine stores the updated container declaration in instance metadata under the gce-container-declaration metadata key.
• Stops and restarts the instance to actuate the updated configuration, if the instance is running. If the instance is stopped, updates the container declaration and keeps the instance stopped. The VM instance downloads the new image and launches the container on VM start.

Note: If you specify a Docker image labeled with latest, the VM instance downloads the latest image and launches a container from the new image each time the VM starts. Do not use the latest label in production if you depend on a specific image version.

Console

1. In the Google Cloud Console, go to the VM instances page.

   Go to the VM instances page

2. Click the name of the instance to update.

3. On the instance details page, click Edit.

4. Specify the new container image and update the options to run the container as needed.

5. Click Save and restart to save your changes. Compute Engine saves the changes and restarts the instance automatically to make the update. After the VM restarts, it downloads the new image and starts the container with the updated configuration.

gcloud

Update the container declaration by using the gcloud compute instances update-container command. For example:

gcloud compute instances update-container nginx-vm \
    --container-image gcr.io/cloud-marketplace/google/nginx1:latest

This command sets the container image to gcr.io/cloud-marketplace/google/nginx1:latest and restarts the instance to actuate the changes. You can also update any of the properties described in Configuring options to run your container by adding corresponding flags.

After the instance restarts, it downloads the new container image and starts the container with the new configuration.

Deploying a container on a managed instance group

You can deploy a container to a new managed instance group (MIG) using Google Cloud Console or the gcloud command-line tool by following these steps:

1. Create an instance template that is based on a Docker image.

   Note: To maintain identical instances in your group, include a specific Docker image version in your instance template, such as nginx1:15. For more information, see Deterministic instance templates.

2. Create a MIG from the new instance template.

Console

The following example creates an instance template that deploys a container from a Google-provided Nginx (gcr.io/cloud-marketplace/google/nginx1:15) Docker image to a MIG. To use other Docker images, specify your desired image instead of gcr.io/cloud-marketplace/google/nginx1:15 in the following example.

1. In the Google Cloud Console, go to the Instance templates page.

   Go to the Instance templates page

2. To create an instance template, click Create instance template.

3. Under Container, select Deploy container image.

4. Under Container image, specify the Docker image name and configure options to run the container if desired. For example, you can specify gcr.io/cloud-marketplace/google/nginx1:15 for the container image.

5. Click Create.

Next, create a MIG that uses the new instance template.

gcloud

Create an instance template for running Docker images using the gcloud compute instance-templates create-with-container command:

gcloud compute instance-templates create-with-container TEMPLATE_NAME \
  --container-image DOCKER_IMAGE

You can also configure options to run your container if desired.

For example, the following command creates a new instance template with name nginx-template, which includes information about the Docker image. A VM instance created from this template launches and runs the Docker image gcr.io/cloud-marketplace/google/nginx1:15 when the VM starts.

gcloud compute instance-templates create-with-container nginx-template \
    --container-image gcr.io/cloud-marketplace/google/nginx1:15

Next, create a MIG using the new instance template.

Now that you have an instance template, you can create a MIG that uses the instance template. For example, to create a MIG by using the gcloud tool and the nginx-template that you just created, run the following command:

gcloud compute instance-groups managed create example-group \
    --base-instance-name nginx-vm \
    --size 3 \
    --template nginx-template

Updating a managed instance group running a container

Beta

This feature is covered by the Pre-GA Offerings Terms of the Google Cloud Platform Terms of Service. Pre-GA features may have limited support, and changes to pre-GA features may not be compatible with other pre-GA versions. For more information, see the launch stage descriptions.

You can update a managed instance group (MIG) to deploy a new version of a Docker image or a new version of the Container-Optimized OS image.

Updating a MIG to a new version of a container image

You can deploy a new version of a Docker image to a MIG by using the Managed Instance Group Updater, in three steps:

1. Prepare a new Docker image for deployment.
2. Create an instance template based on the new Docker image in the same way you create a container-based template.
3. Update a MIG to the new instance template by using the Managed Instance Group Updater.

Updating a managed instance group to a new version of Container-Optimized OS image

Google updates Container-Optimized OS images regularly, and you might want to apply those updates to your containerized MIGs without changing your Docker image. You can update a MIG to a new version of a Container-Optimized OS image by using Google Cloud Console or the gcloud command-line tool in two steps:

1. Create an instance template based on the current version of your Docker image, the same way you create a container-based template for a new MIG. The latest supported version of a Container-Optimized OS image is used by default.
2. Update a MIG with the new instance template by using Managed Instance Group Updater.

Connecting to a container using SSH

You can connect to a container on a VM by using SSH. Use the gcloud tool to run gcloud compute ssh with the --container flag:

gcloud compute ssh VM_NAME --container CONTAINER_NAME

Replace the following:

• VM_NAME: the name of the VM instance
• CONTAINER_NAME: the name of the container

Learn more about the gcloud compute ssh command and its arguments.

Viewing logs

You can view three types of logs related to containers:

1. Startup agent logs, also known as konlet{ class="external" target="github" } logs. The startup agent parses the container's configuration and runs tasks to start the container on a Compute Engine VM instance.

2. Docker event logs report container events, including container start and stop events. These logs are available if you use COS 69 or higher.

3. Logs from your container include the STDOUT from apps that run in your container.

Viewing startup agent logs

Startup agent logs are available in the serial console, through the journald system service included in the OS image, and through Cloud Logging.

Viewing startup agent logs in the serial console

Console

1. In the Google Cloud Console, go to the VM instances page.

   Go to the VM instances page

2. Select the VM instance for which you want to view startup agent logs.

3. Under Logs, click Serial port 1 (console) to view serial console logs.

   

gcloud

Use the get-serial-port-output command to view logs on the instance's serial port.

gcloud compute instances get-serial-port-output VM_NAME

Replace VM_NAME with the name of the VM instance.

For example, use the following command to view the serial port output of a VM instance named nginx-vm:

gcloud compute instances get-serial-port-output nginx-vm

Viewing startup agent logs in journald

1. Connect to your instance with a container by using SSH.

2. Execute the sudo journalctl command to see the VM startup and container startup logs. Use the following command to filter for container startup agent logs (konlet).

   sudo journalctl -u konlet*
   

Viewing startup agent logs in Logging

Console

1. In the Google Cloud Console, go to the VM instances page.

   Go to the VM instances page

2. Select the VM instance for which you want to view startup agent logs.

3. Under Logs, click Cloud Logging to view Cloud Logging logs.

   

4. Enter a search filter to retrieve startup agent logs.

   If you use COS 68 or lower, use the following filter.

   resource.type="global"
   logName="projects/PROJECT_ID/logs/gcplogs-docker-driver"
   jsonPayload.container.imageName:"gcr.io/gce-containers/konlet"
   jsonPayload.instance.name="VM_NAME"
   

   If you use COS 69 or higher, use the following filter.

   resource.type="gce_instance"
   logName="projects/PROJECT_ID/logs/cos_system"
   jsonPayload.SYSLOG_IDENTIFIER="konlet-startup"
   jsonPayload._HOSTNAME="VM_NAME"
   

   Replace the following:

   • PROJECT_ID: the project ID that contains the instance
   • VM_NAME: the name of the instance you want to get logs for

gcloud

Use the gcloud logging read command with an appropriate filter to view container startup agent logs.

If you use COS 68 or lower, use the following command and filter.

gcloud logging read "resource.type=global AND \
    logName=projects/PROJECT_ID/logs/gcplogs-docker-driver AND \
    jsonPayload.instance.name=VM_NAME"

If you use COS 69 or higher, use the following command and filter.

gcloud logging read "resource.type=gce_instance AND \
    logName=projects/PROJECT_ID/logs/cos_system AND \
    jsonPayload.SYSLOG_IDENTIFIER=konlet-startup AND \
    jsonPayload._HOSTNAME=VM_NAME"

Replace the following:

• PROJECT_ID: the project ID that contains the instance
• VM_NAME: the name of the instance that you want to get logs for

For example, use the following command to view the last 10 startup agent logs in Logging for a VM instance named nginx-vm that's running COS 70 and that exists in my-project.

gcloud logging read "resource.type=gce_instance AND \
    logName=projects/my-project/logs/cos_system AND \
    jsonPayload.SYSLOG_IDENTIFIER=konlet-startup AND \
    jsonPayload._HOSTNAME=nginx-vm" \
    --limit 10

Viewing Docker event logs

With COS 69 or higher, you can view Docker event logs in journald and in Cloud Logging.

Viewing Docker event logs in journald

1. Connect to your instance with a container using SSH.

2. Execute the sudo journalctl command with the following filter to see Docker event logs.

   sudo journalctl -u docker-events-collector
   

Viewing Docker event logs in Logging

Console

1. Go to the VM instances page.

   Go to the VM instances page

2. Select the VM instance for which you want to view startup agent logs.

3. Under Logs, click Cloud Logging to view Cloud Logging logs.

   

4. Enter the following search filter to retrieve Docker event logs.

   resource.type="gce_instance"
   logName="projects/PROJECT_ID/logs/cos_system"
   jsonPayload._HOSTNAME="VM_NAME"
   jsonPayload.SYSLOG_IDENTIFIER="docker"
   

   Replace the following:

   • PROJECT_ID: the project ID that contains the instance
   • VM_NAME: the name of the instance you want to get logs for

gcloud

Use the gcloud logging read command with an appropriate filter to view Docker event logs.

gcloud logging read "resource.type=gce_instance AND \
    logName=projects/PROJECT_ID/logs/cos_system AND \
    jsonPayload._HOSTNAME=VM_NAME AND \
    jsonPayload.SYSLOG_IDENTIFIER=docker"

Replace the following:

• PROJECT_ID: the project ID that contains the instance
• VM_NAME: the name of the instance you want to get logs for

For example, use the following command to view the last 10 Docker event logs in Logging for a VM instance named nginx-vm that's running COS 70 and that exists in my-project.

gcloud logging read "resource.type=gce_instance AND \
    logName=projects/my-project/logs/cos_system AND \
    jsonPayload._HOSTNAME=nginx-vm AND \
    jsonPayload.SYSLOG_IDENTIFIER=docker" \
    --limit 10

Viewing container logs

Console

1. Go to the VM instances page.

   Go to the VM instances page

2. Select the VM instance for which you want to view startup agent logs.

3. Under Logs, click Cloud Logging to view Cloud Logging logs.

   

4. The Cloud Logging page loads with a default search filter. If you are using COS 69 or higher, examine the filter and copy the value for resource.labels.instance_id. You will use it later.

5. Update the search filter to retrieve container logs.

   If you use COS 68 or lower, use the following filter.

   logName="projects/PROJECT_ID/logs/gcplogs-docker-driver"
   jsonPayload.container.name="/VM_NAME"
   

   Replace the following:

   • PROJECT_ID: the project ID that contains the instance
   • VM_NAME: the name of the instance you want to get logs for

   If you use COS 69 or higher, use the following filter:

   resource.type="gce_instance"
   logName="projects/PROJECT_ID/logs/cos_containers"
   resource.labels.instance_id="INSTANCE_ID"
   

   Replace the following:

   • PROJECT_ID: the project ID that contains the instance
   • INSTANCE_ID: the ID of the instance that you want to get logs for

gcloud

Use the gcloud logging read command with an appropriate filter to view container logs.

If you use COS 68 or lower, use the following command and filter:

gcloud logging read "logName=projects/PROJECT_ID/logs/gcplogs-docker-driver AND \
    jsonPayload.container.name=/VM_NAME"

If you use COS 69 or higher:

1. Determine the ID for the instance that you want to get logs for.

   gcloud compute instances describe VM_NAME \
      --zone ZONE \
      --format="value(id)"
   

   Replace the following:

   • `<ZONEVM_NAME: the zone where the instance is located
   • `VM_NAMEVM_NAME: the name of the instance that you want to get logs for

2. Use the following command and filter to view the instance's container logs.

   gcloud logging read "resource.type=gce_instance AND \
       logName=projects/PROJECT_ID/logs/cos_containers AND \
       resource.labels.instance_id=INSTANCE_ID"
   

   Replace the following:

   • PROJECT_ID: the project ID that contains the instance.
   • INSTANCE_ID: the ID of the instance.

   For example, use the following command to view the last 10 container logs in Cloud Logging for a VM instance that is running COS 70, that exists in my-project, and that has an instance ID of 555123456789012345.

   gcloud logging read "resource.type=gce_instance AND \
       logName=projects/my-project/logs/cos_containers AND \
       resource.labels.instance_id=555123456789012345" \
       --limit 10
   

Specifying container-optimized images or image families

Containerized VM instances or instance templates are created to use the latest supported container-optimized image by default. The image belongs to the cos-cloud project.

You can override this default with another image from the cos-stable, cos-beta, or cos-dev image families (all in the cos-cloud project).

These image families include container-optimized images from three different release channels:

• The dev channel has all the latest changes and is updated frequently.
• The beta channel is a qualified image that is in beta and is updated less frequently.
• The stable channel offers a longer tested and supported release with only rare and critical updates (if needed).

Note: Read about container-optimized image release channels and update schedule and the container-optimized image support policy.

For example, using the gcloud tool, provide the --image flag to override the default container-optimized image or the --image-family flag to pick the latest image from the specified family at VM creation time.

The following example creates a containerized VM instance that uses the latest image from the cos-dev image family:

gcloud compute instances create-with-container nginx-vm \
    --image-family cos-dev \
    --image-project cos-cloud \
    --container-image gcr.io/cloud-marketplace/google/nginx1:1.15

Configuring firewall rules

Containerized VMs launch containers whose network is set to host mode. A container shares the host network stack, and all interfaces from the host are available to the container.

Note: A VM instance starts a container with the --network="host" flag of the docker run command. Learn about container network settings and host mode.

By default, Google Cloud firewall rules block all incoming connections to a VM instance and allow all outgoing connections from a VM instance.

Create firewall rules to allow incoming connections to your instance and therefore to the container.

Note: This method of VM creation overrides the default host firewall configuration of container-optimized images and opens all ports for incoming TCP connections. The ports are still not accessible from outside the VM because of the default firewall settings of the Virtual Private Cloud (VPC) network.

Configuring options to run a container

You can configure the following options to run your container:

• Specify a container restart policy.
• Override container ENTRYPOINT (default command to be executed on container start).
• Pass arguments to container ENTRYPOINT command.
• Run a container in a privileged mode.
• Mount a host directory or tmpfs as a data volume inside the container.
• Set environment variables.
• Allocate a buffer for STDIN in the container runtime.
• Allocate a pseudo-TTY.

Learn more about configuring options to run your container.

What's next

• Learn about configuring options to run your container.
• Learn more about managed instance groups.
• Learn about Container-Optimized OS.