Tutorial: Deploy an existing VM in an GKE Enterprise cluster using VM Runtime on Google Distributed Cloud

---

Version 1.14 

• Version 1.28 (latest)
• Version 1.16
• Version 1.15
• Earlier versions (unsupported)

This document provides a step-by-step guide to deploy a virtual machine (VM) based workload into GKE on Bare Metal using VM Runtime on Google Distributed Cloud. The workload used in this guide is the sample point of sale application. This application represents a typical point of sale terminal that runs on on-premises hardware at a retail store.

In this document you migrate this application from a VM into an GKE on Bare Metal cluster and access the application's web frontend. To migrate an existing VM into the cluster, first a disk image of that VM must be created. Then, the image must be hosted in a repository that the cluster can access. Finally, the URL of that image can be used to create the VM. VM Runtime on Google Distributed Cloud expects the images to be in qcow2 format. If you provide a different image type, it's automatically converted into the qcow2 format. To avoid repetitive conversion and to enable reuse, you can convert a virtual disk image and host the qcow2 image.

This document uses a pre-prepared image of a Compute Engine VM instance where the workload runs as a systemd service. You may follow these same steps to deploy your own application.

Objectives

• Enable VM Runtime on Google Distributed Cloud and install the virtctl plugin
• Deploy the VM-based workload
• Log in to the VM and check the application status
• Access the VM-based workload

Before you begin

To complete this document you need the following resources:

• Access to an GKE on Bare Metal version 1.12.0 or higher cluster that was created by following the Running GKE on Bare Metal on Compute Engine VMs with Manual Load Balancer guide. This document sets up networking resources so that you can access the workload running inside the VM through a browser. If you don't need that behavior, you can follow this document using any GKE on Bare Metal.
• A workstation that meets the following requirements:
  • Has access to your cluster using the bmctl CLI.
  • Has access to your cluster using the kubectl CLI.

Enable VM Runtime on Google Distributed Cloud and install the virtctl plugin

The VM Runtime on Google Distributed Cloud custom resource definition (CRD) is part of all GKE on Bare Metal clusters since version 1.10. An instance of the VMRuntime custom resource is already created upon installation. However, it is disabled by default.

1. Enable VM Runtime on Google Distributed Cloud:

   sudo bmctl enable vmruntime --kubeconfig KUBECONFIG_PATH
   

   • KUBECONFIG_PATH: Path to the Kubernetes configuration file of the GKE Enterprise user cluster

2. Validate that the VMRuntime is enabled:

   kubectl wait --for=jsonpath='{.status.ready}'=true vmruntime vmruntime
   

   It can take a few minutes for the VMRuntime to be ready. If it is not ready, then check a few times with short delays. The following example output shows the VMRuntime is ready:

   vmruntime.vm.cluster.gke.io/vmruntime condition met
   

3. Install the virtctl plugin for kubectl:

   sudo -E bmctl install virtctl
   

   The following example output shows the virtctl plugin installation process is complete:

   Please check the logs at bmctl-workspace/log/install-virtctl-20220831-182135/install-virtctl.log
   [2022-08-31 18:21:35+0000] Install virtctl succeeded
   

4. Verify installation of the virtctl plugin:

   kubectl virt
   

   The following example output shows that the virtctl plugin is available for use with kubectl:

   Available Commands:
     addvolume         add a volume to a running VM
     completion        generate the autocompletion script for the specified shell
     config            Config subcommands.
     console           Connect to a console of a virtual machine instance.
     create            Create subcommands.
     delete            Delete  subcommands.
   ...
   

Deploy the VM-based workload

When you deploy a VM into GKE on Bare Metal, VM Runtime on Google Distributed Cloud expects a VM image. This image serves as the boot disk for the deployed VM.

In this tutorial, you migrate a Compute Engine VM-based workload into an GKE on Bare Metal cluster. This Compute Engine VM was created, and the sample point of sale (PoS) application was configured to run as a systemd service. A disk image of this VM along with the PoS application workload was created in Google Cloud. This image was then exported into a Cloud Storage bucket as a qcow2 image. You use this pre-prepared qcow2 image in the following steps.

Note: Although we use a Compute Engine VM in this document, the steps to create a disk image of a VM vary depending on the environment or platform in which your existing VM is currently running. You can create a disk image of your existing VMs if you want to follow the steps in this document with your own custom image.

The source code in this document is available in the anthos-samples GitHub repository. You use resources from this repository to complete the steps that follow.

1. Deploy a MySQL StatefulSet. The point of sale application expects to connect to a MySQL database to store inventory and payment information. The point of sale repository has a sample manifest that deploys a MySQL StatefulSet, configures an associated ConfigMap, and a Kubernetes Service. The ConfigMap defines the credentials for the MySQL instance, which are the same credentials passed into the point of sale application.

   kubectl apply -f https://raw.githubusercontent.com/GoogleCloudPlatform/point-of-sale/main/k8-manifests/common/mysql-db.yaml
   

2. Deploy the VM workload using the pre-prepared qcow2 image:

   kubectl virt create vm pos-vm \
       --boot-disk-size=80Gi \
       --memory=4Gi \
       --vcpu=2 \
       --image=https://storage.googleapis.com/pos-vm-images/pos-vm.qcow2
   

   This command creates a YAML file named after the VM (google-virtctl/pos-vm.yaml). You can inspect the file to see the definition of the VirtualMachine and VirtualMachineDisk. Instead of using the virtctl plugin, you could have deployed the VM workload using Kubernetes Resource Model (KRM) definitions, as seen in the created YAML file.

   When the command runs successfully, it produces an output like the following example that explains the different resources that were created:

   Constructing manifest for vm "pos-vm":
   Manifest for vm "pos-vm" is saved to /home/tfadmin/google-virtctl/pos-vm.yaml
   Applying manifest for vm "pos-vm"
   Created gvm "pos-vm"
   

3. Check the VM creation status.

   The VirtualMachine resource is identified by the vm.cluster.gke.io/v1.VirtualMachine resource in VM Runtime on Google Distributed Cloud. The short form for it is gvm.

   When you create a VM, the following two resources are created:

   • A VirtualMachineDisk is the persistent disk where the contents of the VM image is imported into.
   • A VirtualMachine is the VM instance itself. The DataVolume is mounted into the VirtualMachine before the VM is booted up.

   Check the status of the VirtualMachineDisk. VirtualMachineDisk internally creates a DataVolume resource. The VM image is imported into the DataVolume which is mounted into the VM:

   kubectl get datavolume
   

   The following example output shows the start of the image import:

   NAME              PHASE             PROGRESS   RESTARTS   AGE
   pos-vm-boot-dv    ImportScheduled   N/A                   8s
   

4. Check the status of the VirtualMachine. The VirtualMachine is in the Provisioning state until the DataVolume is imported completely:

   kubectl get gvm
   

   The following example output shows the VirtualMachine being provisioned:

   NAME      STATUS         AGE     IP
   pos-vm    Provisioning   1m
   

5. Wait for the VM image to be fully imported into the DataVolume. Continue to watch the progress while the image is imported:

   kubectl get datavolume -w
   

   The following example output shows the disk image being imported:

   NAME              PHASE              PROGRESS   RESTARTS   AGE
   pos-vm-boot-dv   ImportInProgress   0.00%                 14s
   ...
   ...
   pos-vm-boot-dv   ImportInProgress   0.00%                 31s
   pos-vm-boot-dv   ImportInProgress   1.02%                 33s
   pos-vm-boot-dv   ImportInProgress   1.02%                 35s
   ...
   

   When the import is complete and the DataVolume is created, the following example output shows the PHASE of Succeeded :

   kubectl get datavolume
   

   NAME              PHASE             PROGRESS   RESTARTS   AGE
   pos-vm-boot-dv    Succeeded         100.0%                14m18s
   

   Note: The import time depends on the size of the VM image used. The Compute Engine VM image in this document is about 5 GB and takes up to 15 minutes to import.

6. Confirm that the VirtualMachine has been created successfully:

   kubectl get gvm
   

   If the creation was successful the STATUS shows RUNNING, as shown in the following example, along with the VM's IP address:

   NAME      STATUS    AGE     IP
   pos-vm    Running   40m     192.168.3.250
   

Connect to the VM and check the application status

The image used for the VM includes the point of sale sample application. The application is configured to automatically start on boot as a systemd service. You can see the systemd services' configuration files in the pos-systemd-services directory.

1. Connect to the VM console. Run the following command and press Enter⏎ after you see the Successfully connected to pos-vm… message:

   kubectl virt console pos-vm
   

   This command produces the following example output that prompts you to input the login details:

   Successfully connected to pos-vm console. The escape sequence is ^]
   
   pos-from-public-image login:
   

   Use the following user account and password. This account was set up inside the original VM from which the image for the VM Runtime on Google Distributed Cloud VirtualMachine was created.

   • Login Username: abmuser
   • Password: abmworks

2. Check the status of the point of sale application services. The point of sale application includes three services: API, Inventory, and Payments. These services all run as system services.

   The three services all connect to each other through localhost. However, the application connects to the MySQL database using a mysql-db Kubernetes Service that was created in the earlier step. This behavior means that the VM is automatically connected to the same network as the Pods and Services, enabling seamless communication between VM workloads and other containerized applications. You don't have to do anything extra to make the Kubernetes Services reachable from the VMs deployed using Anthos VM Runtime.

   sudo systemctl status pos*
   

   The following example output shows the status of the three services and root system service, pos.service:

   ● pos_payments.service - Payments service of the Point of Sale Application
       Loaded: loaded (/etc/systemd/system/pos_payments.service; enabled; vendor >
       Active: active (running) since Tue 2022-06-21 18:55:30 UTC; 1h 10min ago
     Main PID: 750 (payments.sh)
         Tasks: 27 (limit: 4664)
       Memory: 295.1M
       CGroup: /system.slice/pos_payments.service
               ├─750 /bin/sh /pos/scripts/payments.sh
               └─760 java -jar /pos/jars/payments.jar --server.port=8083
   
   ● pos_inventory.service - Inventory service of the Point of Sale Application
       Loaded: loaded (/etc/systemd/system/pos_inventory.service; enabled; vendor>
       Active: active (running) since Tue 2022-06-21 18:55:30 UTC; 1h 10min ago
     Main PID: 749 (inventory.sh)
         Tasks: 27 (limit: 4664)
       Memory: 272.6M
       CGroup: /system.slice/pos_inventory.service
               ├─749 /bin/sh /pos/scripts/inventory.sh
               └─759 java -jar /pos/jars/inventory.jar --server.port=8082
   
   ● pos.service - Point of Sale Application
       Loaded: loaded (/etc/systemd/system/pos.service; enabled; vendor preset: e>
       Active: active (exited) since Tue 2022-06-21 18:55:30 UTC; 1h 10min ago
     Main PID: 743 (code=exited, status=0/SUCCESS)
         Tasks: 0 (limit: 4664)
       Memory: 0B
       CGroup: /system.slice/pos.service
   
   Jun 21 18:55:30 pos-vm systemd[1]: Starting Point of Sale Application...
   Jun 21 18:55:30 pos-vm systemd[1]: Finished Point of Sale Application.
   
   ● pos_apiserver.service - API Server of the Point of Sale Application
       Loaded: loaded (/etc/systemd/system/pos_apiserver.service; enabled; vendor>
       Active: active (running) since Tue 2022-06-21 18:55:31 UTC; 1h 10min ago
     Main PID: 751 (api-server.sh)
         Tasks: 26 (limit: 4664)
       Memory: 203.1M
       CGroup: /system.slice/pos_apiserver.service
               ├─751 /bin/sh /pos/scripts/api-server.sh
               └─755 java -jar /pos/jars/api-server.jar --server.port=8081
   

3. Exit the VM. To exit the console connection, use escape sequence ^] by pressing Ctrl + ].

Access the VM-based workload

If your cluster was set up by following the Running GKE on Bare Metal on Compute Engine VMs with Manual Load Balancer guide, it has an Ingress resource called pos-ingress already created. This resource routes the traffic from the public IP address of the Ingress Load Balancer to the API server service of the point of sale sample application.

1. If your cluster doesn't have this Ingress resource, create it by applying the following manifest:

   kubectl apply -f https://raw.githubusercontent.com/GoogleCloudPlatform/anthos-samples/main/anthos-bm-gcp-terraform/resources/manifests/pos-ingress.yaml
   

   apiVersion: networking.k8s.io/v1
   kind: Ingress
   metadata:
     name: pos-ingress
   spec:
     rules:
     - http:
         paths:
         - path: /
           pathType: Prefix
           backend:
             service:
               name: api-server-svc
               port:
                 number: 8080

2. Create a Kubernetes Service that routes traffic to the VM. The Ingress resource routes traffic to this Service:

   kubectl apply -f https://raw.githubusercontent.com/GoogleCloudPlatform/anthos-samples/main/anthos-vmruntime/pos-service.yaml
   

   The following example output confirms the creation of a Service:

   service/api-server-svc created
   

   apiVersion: v1
   kind: Service
   metadata:
     name: api-server-svc
   spec:
     selector:
       kubevirt/vm: pos-vm
     ports:
     - protocol: TCP
       port: 8080
       targetPort: 8081

3. Get the public IP address of the Ingress load balancer. The Ingress Loadbalancer routes traffic based on the Ingress resource rules. You already have a pos-ingress rule to forward requests to the API server Service. This Service forwards the requests to the VM:

   INGRESS_IP=$(kubectl get ingress/pos-ingress -o jsonpath='{.status.loadBalancer.ingress[0].ip}')
   echo $INGRESS_IP
   

   The following example output shows the IP address of the Ingress load balancer:

   172.29.249.159 # you might have a different IP address
   

4. Access the application by using the Ingress Loadbalancer IP address in a browser. The following example screenshots shows the simple point of sale kiosk with two items. You can click on the items, more than once if you want to order multiple of them, and place an order with the Pay button. This experience shows that you have successfully deployed a traditional VM-based workload into an Anthos cluster using VM Runtime on Google Distributed Cloud.

Point of sale application UI (click image to enlarge)

Clean up

You may delete all the resources created in this tutorial or delete only the VM and keep reusable resources. Delete a VM in GKE on Bare Metal explains the available options in detail.

Delete All

• Delete the VM Runtime on Google Distributed Cloud VirtualMachine along with all the resources:

  kubectl virt delete vm pos-vm --all
  

  The following example output confirms the deletion:

  vm "pos-vm" used the following resources: 
      gvm: pos-vm
      VirtualMachineDisk: pos-vm-boot-dv
  Start deleting the resources:
      Deleted gvm "pos-vm".
      Deleted VirtualMachineDisk "pos-vm-boot-dv".
  

Delete only VM

• Deleting only the VM preserves the VirtualMachineDisk that gets created. This enables reuse of this VM image and saves time spent on importing the image when creating a new VM.

  kubectl virt delete vm pos-vm
  

  The following example output confirms the deletion:

  vm "pos-vm" used the following resources: 
      gvm: pos-vm
      VirtualMachineDisk: pos-vm-boot-dv
  Start deleting the resources:
      Deleted gvm "pos-vm".
  

What's next

• The original VM used in this guide is a Compute Engine instance that runs Ubuntu 20.04 LTS. The image of this VM is publicly accessible through the pos-vm-images Cloud Storage bucket. For more information on how the VM was configured and its image was created, see the instructions in the point-of-sale repository.
• When you create a VM in an Anthos cluster using the kubectl virt create vm pos-vm command, a YAML file named after the VM (google-virtctl/pos-vm.yaml) is created. You can inspect the file to see the definition of the VirtualMachine and VirtualMachineDisk. Instead of using the virtctl plugin, you can deploy a VM using KRM definitions as seen in the created YAML file.