Deploying containers on VMs and MIGs

---

You can configure a virtual machine (VM) instance or an instance template to deploy and launch a Docker container. Compute Engine supplies an up-to-date Container-Optimized OS (COS) image with Docker installed and launches your container when your VM starts.

Before you begin

• If you aren't familiar with containers, read What are containers and their benefits.
• If you aren't familiar with Docker, read the Docker documentation.
• Read about Container-Optimized OS.
• Read about managed instance groups (MIGs).
• If you haven't already, then set up authentication. Authentication is the process by which your identity is verified for access to Google Cloud services and APIs. To run code or samples from a local development environment, you can authenticate to Compute Engine by selecting one of the following options:
  

  Select the tab for how you plan to use the samples on this page:

  Console

  When you use the Google Cloud console to access Google Cloud services and APIs, you don't need to set up authentication.

  gcloud

  1. Install the Google Cloud CLI, then initialize it by running the following command:

     gcloud init

     Note: If you installed the gcloud CLI previously, make sure you have the latest version by running gcloud components update.
  2. Set a default region and zone.

Choosing to deploy containers on VMs and MIGs

By deploying containers on Compute Engine, you can simplify app deployment while controlling your VM infrastructure.

• Manage VMs that are running containers in the same way you would treat any other VM when configuring and managing your Compute Engine infrastructure.
• Use familiar processes and tools such as the Google Cloud CLI or the Compute Engine API to manage your VMs with containers.

• Create scalable services using managed instance groups (MIGs) running containers, which offer features like autoscaling, autohealing, rolling updates, multi-zone deployments, and load balancing.

  Note: The Mexico, Montreal, and Osaka regions have three zones within one or two physical data centers. These regions are in the process of expanding to at least three physical data centers. For more information, see Cloud locations and Google Cloud Platform SLAs. To help improve the reliability of your workloads, consider a multi-regional deployment.

Alternatively, you might consider deploying to Google Kubernetes Engine to:

• Run a large number of microservices
• Have faster container startup time
• Take advantage of Kubernetes automated orchestration, including auto upgrades, node auto repair, and autoscaling

Running each microservice on a separate virtual machine (VM) on Compute Engine could make the operating system overhead a significant part of your cost. Google Kubernetes Engine lets you deploy multiple containers and groups of containers for each VM instance, which can allocate host VM resources more efficiently to microservices with a smaller footprint.

How deploying containers on Compute Engine works

The common methods of deploying software onto a Compute Engine VM instance include:

• Deploying software on VM boot using a startup script or cloud-init.
• Creating a custom boot disk image with software pre-installed.

Both of the above methods combine the tasks of configuring the app and setting up the operating system environment. As the developer, you must carefully track and resolve any runtime dependencies. For example, if two apps running on a VM use different versions of the same library, you must install both versions and point to them through system variables.

A VM instance with apps deployed directly to the operating system

Alternatively, you can deploy software in a container onto a VM instance or to a MIG. A container carries both application software and the required libraries and is isolated from OS apps and libraries. A container can be easily moved between deployment environments without dealing with conflicting library versions in the container and its OS.

A VM instance with apps deployed in a container

The following process describes how you deploy a container on Compute Engine:

1. You bundle your app and required libraries into a Docker image and publish the image to Artifact Registry, or a third-party registry such as Docker Hub.
2. You specify a Docker image name and the docker run configuration when creating a VM instance or an instance template for a MIG.

Note: As of November 1, 2020, Docker Hub rate limits apply to unauthenticated or authenticated pull requests on the Docker Free plan. To avoid disruptions and have greater control over your software supply chain, you can migrate your dependencies to Artifact Registry.

Compute Engine executes the following tasks after you make a request to create a VM instance:

1. Compute Engine creates a VM instance that uses a Google-provided Container-Optimized OS image. This image includes a Docker runtime and additional software that is responsible for starting your container.
2. Compute Engine stores your container settings in instance metadata under the gce-container-declaration metadata key.
3. When the VM starts, the Container-Optimized OS image uses the docker run command configuration that is stored in the instance's metadata, pulls the container image from the repository, and starts the container.

Steps to create a VM instance or a managed instance group running a container

Limitations

• You can only deploy one container for each VM instance. Consider Google Kubernetes Engine if you need to deploy multiple containers per VM instance.

• You can only deploy containers from a public repository or from a private Artifact Registry or Container Registry repository that you can access. Other private repositories are not supported.

  See the access control documentation for Artifact Registry or Container Registry for information about private registry permissions.

  Caution: Container Registry is deprecated and scheduled for shutdown. Organizations that haven't used Container Registry prior to January 8, 2024 have new gcr.io repositories hosted on Artifact Registry by default. Effective May 15, 2024, Google Cloud projects without previous usage of Container Registry will only support hosting and managing images for the gcr.io domain in Artifact Registry.

  Container Registry is scheduled for shutdown on March 18, 2025. For details on the deprecation, see Container Registry deprecation.

• You can't map a VM instance's ports to the container's ports (Docker's -p option). To enable access to your containers, see Publishing container ports.

• You can only use Container-Optimized OS images with this deployment method.

• You can only use this feature through the Google Cloud console or the Google Cloud CLI, not the API.

Preparing a container for deployment

Choose one of the following approaches to make your container image accessible to Compute Engine:

• Upload your Docker image to Artifact Registry.
• Use any publicly available container images from Docker Hub or other registries.

Note: Consider running your container image builds on Cloud Build, with none of the overhead of managing your own build servers.

Deploying a container on a new VM instance

You can deploy a container on a new VM instance by using the Google Cloud console or the Google Cloud CLI.

Console

The following example deploys a container from a Google-provided Nginx Docker image, https://gcr.io/cloud-marketplace/google/nginx1:latest, to a VM instance. To use a different Docker image, specify the image you want instead in the examples below.

1. Go to the Create an instance page.

   Go to Create an instance

2. Specify the VM details.

3. In the Container section, click Deploy container.

4. On the Configure container page, specify a container image name and configure options to run the container for your needs. For example, you can specify gcr.io/cloud-marketplace/google/nginx1:latest for the container image.

5. Continue with the VM creation process.

gcloud

Use the gcloud compute instances create-with-container command:

gcloud compute instances create-with-container VM_NAME \
    --container-image DOCKER_IMAGE

For example, the following command creates a new VM instance named nginx-vm, which launches and runs the Docker image gcr.io/cloud-marketplace/google/nginx1:latest.

gcloud compute instances create-with-container nginx-vm \
    --container-image gcr.io/cloud-marketplace/google/nginx1:latest

Learn more about the gcloud compute instances create-with-container command.

You must always specify a full Docker image name when using a public image from Docker Hub. For example, specify the following image name to deploy an Apache container image:

docker.io/httpd:2.4

Updating a container on a VM instance

You can update a Docker image and configuration options to run the container on a VM instance using Google Cloud console or the Google Cloud CLI.

When you update a VM running a container, Compute Engine performs two steps:

• Updates container declaration on the instance. Compute Engine stores the updated container declaration in instance metadata under the gce-container-declaration metadata key.
• Stops and restarts the instance to actuate the updated configuration, if the instance is running. If the instance is stopped, updates the container declaration and keeps the instance stopped. The VM instance downloads the new image and launches the container on VM start.

Note: If you specify a Docker image labeled with latest, the VM instance downloads the latest image and launches a container from the new image each time the VM starts. Do not use the latest label in production if you depend on a specific image version.

Console

1. Go to the VM instances page.

   Go to VM instances

2. Click the name of the VM to update.

3. On the instance details page, click Edit.

4. Specify the new container image and update the options to run the container as needed.

5. To save your changes, click Save and restart. Compute Engine saves the changes and restarts the instance automatically to make the update. After the VM restarts, it downloads the new image and starts the container with the updated configuration.

gcloud

Update the container declaration by using the gcloud compute instances update-container command. For example:

gcloud compute instances update-container nginx-vm \
    --container-image gcr.io/cloud-marketplace/google/nginx1:latest

This command sets the container image to gcr.io/cloud-marketplace/google/nginx1:latest and restarts the instance to actuate the changes. You can also update any of the properties described in Configuring options to run your container by adding corresponding flags.

After the instance restarts, it downloads the new container image and starts the container with the new configuration.

Deploying a container on a managed instance group

You can deploy a container to a new managed instance group (MIG) using Google Cloud console or the Google Cloud CLI by following these steps:

1. Create an instance template that is based on a Docker image.

   Note: To maintain identical instances in your group, include a specific Docker image version in your instance template, such as nginx1:15. For more information, see Deterministic instance templates.

2. Create a MIG from the new instance template.

Console

The following example creates an instance template that deploys a container from a Google-provided Nginx (gcr.io/cloud-marketplace/google/nginx1:15) Docker image to a MIG. To use other Docker images, specify your desired image instead of gcr.io/cloud-marketplace/google/nginx1:15 in the following example.

1. Go to the Instance templates page.

   Go to Instance templates

2. To create an instance template, click Create instance template.

3. Under Container, select Deploy container image.

4. Under Container image, specify the Docker image name and configure options to run the container if desired. For example, you can specify gcr.io/cloud-marketplace/google/nginx1:15 for the container image.

5. Click Create.

Next, create a MIG that uses the new instance template.

gcloud

Create an instance template for running Docker images using the gcloud compute instance-templates create-with-container command:

gcloud compute instance-templates create-with-container TEMPLATE_NAME \
  --container-image DOCKER_IMAGE

You can also configure options to run your container if desired.

For example, the following command creates a new instance template with name nginx-template, which includes information about the Docker image. A VM instance created from this template launches and runs the Docker image gcr.io/cloud-marketplace/google/nginx1:15 when the VM starts.

gcloud compute instance-templates create-with-container nginx-template \
    --container-image gcr.io/cloud-marketplace/google/nginx1:15

Next, create a MIG using the new instance template.

Now that you have an instance template, you can create a MIG that uses the instance template. For example, to create a MIG by using the gcloud CLI and the nginx-template that you just created, run the following command:

gcloud compute instance-groups managed create example-group \
    --base-instance-name nginx-vm \
    --size 3 \
    --template nginx-template

Updating a managed instance group running a container

Beta

This feature is subject to the "Pre-GA Offerings Terms" in the General Service Terms section of the Service Specific Terms. Pre-GA features are available "as is" and might have limited support. For more information, see the launch stage descriptions.

You can update a managed instance group (MIG) to deploy a new version of a Docker image or a new version of the Container-Optimized OS image.

Updating a MIG to a new version of a container image

You can deploy a new version of a Docker image to a MIG by using the Managed Instance Group Updater, in three steps:

1. Prepare a new Docker image for deployment.
2. Create an instance template based on the new Docker image in the same way you create a container-based template.
3. Update a MIG to the new instance template by using the Managed Instance Group Updater.

Updating a managed instance group to a new version of Container-Optimized OS image

Google updates Container-Optimized OS images regularly, and you might want to apply those updates to your containerized MIGs without changing your Docker image. You can update a MIG to a new version of a Container-Optimized OS image by using Google Cloud console or the Google Cloud CLI in two steps:

1. Create an instance template based on the current version of your Docker image, the same way you create a container-based template for a new MIG. The latest supported version of a Container-Optimized OS image is used by default.
2. Update a MIG with the new instance template by using Managed Instance Group Updater.

Connecting to a container using SSH

You can connect to a container on a VM by using SSH. Use the gcloud CLI to run gcloud compute ssh with the --container flag:

gcloud compute ssh VM_NAME --container CONTAINER_NAME

Replace the following:

• VM_NAME: the name of the VM instance
• CONTAINER_NAME: the name of the container

Learn more about the gcloud compute ssh command and its arguments.

Monitoring containers on Compute Engine

To monitor your instances running a Container-Optimized OS image, use the Node Problem Detector agent, which communicates with Cloud Monitoring and reports health-related metrics. The agent is built into Container-Optimized OS images starting with Milestone 77.

To enable the agent, in containers using images with Milestone 88 or later, edit the custom metadata section and set google-monitoring-enabled to true.

Note: For more information about setting custom metadata for an instance, refer to the VM metadata documentation.

To find other ways of enabling the Node Problem Detector, visit Enabling health monitoring.

The Node Problem Detector agent supports the metrics in the metrics list that begin with guest/.

To interact with the metrics collected by the agent, visit the Metrics Explorer.

Viewing logs

You can view three types of logs related to containers:

1. Startup agent logs, also known as konlet logs. The startup agent parses the container's configuration and runs tasks to start the container on a Compute Engine VM instance.

2. Docker event logs report container events, including container start and stop events.

3. Logs from your container include the STDOUT from apps that run in your container.

Note: Depending on the container that you're using, you might need to allow incoming traffic to the container before you see any log entries. To enable access to your containers, see Publishing container ports.".

Viewing startup agent logs

Startup agent logs are available in the serial console, through the journald system service included in the OS image, and through Cloud Logging.

Viewing startup agent logs in the serial console

Console

1. Go to the VM instances page.

   Go to VM instances

2. Select the VM instance for which you want to view startup agent logs.

3. Under Logs, click Serial port 1 (console) to view serial console logs.

   

gcloud

Use the get-serial-port-output command to view logs on the instance's serial port.

gcloud compute instances get-serial-port-output VM_NAME

Replace VM_NAME with the name of the VM instance.

For example, use the following command to view the serial port output of a VM instance named nginx-vm:

gcloud compute instances get-serial-port-output nginx-vm

Viewing startup agent logs in journald

1. Connect to your instance with a container by using SSH.

2. Execute the sudo journalctl command to see the VM startup and container startup logs. Use the following command to filter for container startup agent logs (konlet).

   sudo journalctl -u konlet*
   

Viewing startup agent logs in Logging

Console

1. Go to the VM instances page.

   Go to VM instances

2. Select the VM instance for which you want to view startup agent logs.

3. Under Logs, click Cloud Logging to view Cloud Logging logs.

   

4. Enter a search filter to retrieve startup agent logs.

   resource.type="gce_instance"
   logName="projects/PROJECT_ID/logs/cos_system"
   jsonPayload.SYSLOG_IDENTIFIER="konlet-startup"
   jsonPayload._HOSTNAME="VM_NAME"
   

   Replace the following:

   • PROJECT_ID: the project ID that contains the instance
   • VM_NAME: the name of the instance you want to get logs for

gcloud

Use the gcloud logging read command with an appropriate filter to view container startup agent logs.

gcloud logging read "resource.type=gce_instance AND \
    logName=projects/PROJECT_ID/logs/cos_system AND \
    jsonPayload.SYSLOG_IDENTIFIER=konlet-startup AND \
    jsonPayload._HOSTNAME=VM_NAME"

Replace the following:

• PROJECT_ID: the project ID that contains the instance
• VM_NAME: the name of the instance that you want to get logs for

For example, use the following command to view the last 10 startup agent logs in Logging for a VM instance named nginx-vm that's running COS 70 and that exists in my-project.

gcloud logging read "resource.type=gce_instance AND \
    logName=projects/my-project/logs/cos_system AND \
    jsonPayload.SYSLOG_IDENTIFIER=konlet-startup AND \
    jsonPayload._HOSTNAME=nginx-vm" \
    --limit 10

Viewing Docker event logs

You can view Docker event logs in journald and in Cloud Logging.

Viewing Docker event logs in journald

1. Connect to your instance with a container using SSH.

2. Execute the sudo journalctl command with the following filter to see Docker event logs.

   sudo journalctl -u docker-events-collector
   

Viewing Docker event logs in Logging

Console

1. Go to the VM instances page.

   Go to VM instances

2. Select the VM instance for which you want to view startup agent logs.

3. Under Logs, click Cloud Logging to view Cloud Logging logs.

   

4. Enter the following search filter to retrieve Docker event logs.

   resource.type="gce_instance"
   logName="projects/PROJECT_ID/logs/cos_system"
   jsonPayload._HOSTNAME="VM_NAME"
   jsonPayload.SYSLOG_IDENTIFIER="docker"
   

   Replace the following:

   • PROJECT_ID: the project ID that contains the instance
   • VM_NAME: the name of the instance you want to get logs for

gcloud

Use the gcloud logging read command with an appropriate filter to view Docker event logs.

gcloud logging read "resource.type=gce_instance AND \
    logName=projects/PROJECT_ID/logs/cos_system AND \
    jsonPayload._HOSTNAME=VM_NAME AND \
    jsonPayload.SYSLOG_IDENTIFIER=docker"

Replace the following:

• PROJECT_ID: the project ID that contains the instance
• VM_NAME: the name of the instance you want to get logs for

For example, use the following command to view the last 10 Docker event logs in Logging for a VM instance named nginx-vm that's running COS 70 and that exists in my-project.

gcloud logging read "resource.type=gce_instance AND \
    logName=projects/my-project/logs/cos_system AND \
    jsonPayload._HOSTNAME=nginx-vm AND \
    jsonPayload.SYSLOG_IDENTIFIER=docker" \
    --limit 10

Viewing container logs

Console

1. Go to the VM instances page.

   Go to VM instances

2. Select the VM instance for which you want to view startup agent logs.

3. Under Logs, click Cloud Logging to view Cloud Logging logs.

   

4. The Cloud Logging page loads with a default search filter. Copy the value for resource.labels.instance_id. You will use it later.

5. Update the search filter to retrieve container logs.

   resource.type="gce_instance"
   logName="projects/PROJECT_ID/logs/cos_containers"
   resource.labels.instance_id="INSTANCE_ID"
   

   Replace the following:

   • PROJECT_ID: the project ID that contains the instance
   • INSTANCE_ID: the ID of the instance that you want to get logs for

gcloud

Use the gcloud logging read command to view container logs.

1. Determine the ID for the instance that you want to get logs for:

   gcloud compute instances describe VM_NAME \
      --zone ZONE \
      --format="value(id)"
   

   Replace the following:

   • VM_NAME: the name of the instance that you want to get logs for
   • ZONE: the zone where the instance is located

2. Use the following command and filter to view the instance's container logs.

   gcloud logging read "resource.type=gce_instance AND \
       logName=projects/PROJECT_ID/logs/cos_containers AND \
       resource.labels.instance_id=INSTANCE_ID"
   

   Replace the following:

   • PROJECT_ID: the project ID that contains the instance.
   • INSTANCE_ID: the ID of the instance.

   For example, use the following command to view the last 10 container logs in Cloud Logging for a VM instance that is running COS 70, that exists in my-project, and that has an instance ID of 555123456789012345.

   gcloud logging read "resource.type=gce_instance AND \
       logName=projects/my-project/logs/cos_containers AND \
       resource.labels.instance_id=555123456789012345" \
       --limit 10
   

Specifying container-optimized images or image families

Containerized VM instances or instance templates are created to use the latest supported container-optimized image by default. The image belongs to the cos-cloud project.

You can override this default with another image from the cos-cloud project. For information about available image families and their attributes, see Choosing the right Container-Optimized OS version.

For example, after you know which image you want to use, in the gcloud CLI, either provide the --image flag to override the default container-optimized image or provide the --image-family flag to pick the latest image from the specified family at VM creation time.

The following example creates a containerized VM instance that uses the latest image from the cos-dev image family:

gcloud compute instances create-with-container nginx-vm \
    --image-family cos-dev \
    --image-project cos-cloud \
    --container-image gcr.io/cloud-marketplace/google/nginx1:1.15

Configuring firewall rules

Containerized VMs launch containers whose network is set to host mode. A container shares the host network stack, and all interfaces from the host are available to the container.

Note: A VM instance starts a container with the --network="host" flag of the docker run command. Learn about container network settings and host mode.

By default, Google Cloud firewall rules block all incoming connections to a VM instance and allow all outgoing connections from a VM instance.

Create firewall rules to allow incoming connections to your instance and therefore to the container.

Note: This method of VM creation overrides the default host firewall configuration of container-optimized images and opens all ports for incoming TCP connections. The ports are still not accessible from outside the VM because of the default firewall settings of the Virtual Private Cloud (VPC) network.

Configuring options to run a container

You can configure the following options to run your container:

• Specify a container restart policy.
• Override container ENTRYPOINT (default command to be executed on container start).
• Pass arguments to container ENTRYPOINT command.
• Run a container in a privileged mode.
• Mount a host directory or tmpfs as a data volume inside the container.
• Set environment variables.
• Allocate a buffer for STDIN in the container runtime.
• Allocate a pseudo-TTY.

Learn more about configuring options to run your container.

What's next

• Learn about configuring options to run your container.
• Learn more about managed instance groups.
• Learn about Container-Optimized OS.