This page describes how to manage virtual machines on Google Distributed Cloud connected servers running VM Runtime on Google Distributed Cloud. You must be familiar with VM Runtime on GDC before completing the steps on this page. For a list of supported guest operating systems, see Verified guest operating systems for VM Runtime on GDC.
To learn how virtual machines serve as an essential component of the Distributed Cloud connected platform, see Extending GKE Enterprise to manage on-premises edge VMs.
Distributed Cloud connected clusters support virtual machine webhooks. This allows Distributed Cloud connected to validate user requests made to the local Kubernetes API server. Rejected requests generate detailed information on the reason for rejection.
Configure Symcloud Storage
Google Distributed Cloud connected servers use Rakuten Symcloud Storage, as their storage solution. Symcloud Storage is a third-party solution that acts as a local storage abstraction layer on each Distributed Cloud connected node and makes its local storage available to workloads running on other Distributed Cloud connected nodes.
Symcloud Storage is deployed from Google Cloud Marketplace and is subject to the terms stated therein. Google provides limited support for using Symcloud Storage with Distributed Cloud connected and might engage the third-party provider for assistance. Software updates for Symcloud Storage are included in the Distributed Cloud connected software updates.
Configure your Google Distributed Cloud connected cluster as follows to enable Symcloud Storage for virtual machines:
Create the
robinio
namespace with the following command:kubectl create ns robinio
Obtain the Symcloud Storage license file and apply it to the cluster with the following command:
kubectl apply LICENSE_FILE
Verify that Symcloud Storage is up and running with the following command:
kubectl apply LICENSE_FILE
The command returns output similar to the following:
Name: robin Namespace: Labels: app.kubernetes.io/instance=robin app.kubernetes.io/managed-by=robin.io app.kubernetes.io/name=robin Annotations: <none> API Version: manage.robin.io/v1 Kind: RobinCluster Metadata: … Spec: … Status: … Phase: Ready …
Create the
robin-block-immediate
storage class by applying the following configuration to the cluster:apiVersion: storage.k8s.io/v1 kind: StorageClass metadata: name: robin-block-immediate parameters: faultdomain: host replication: "3" blocksize: "512" provisioner: robin reclaimPolicy: Delete volumeBindingMode: Immediate allowVolumeExpansion: true
Create the
robin-snapshotclass
volume snapshot class by applying the following configuration to the cluster:apiVersion: snapshot.storage.k8s.io/v1 kind: VolumeSnapshotClass metadata: name: robin-snapshotclass labels: app.kubernetes.io/instance: robin app.kubernetes.io/managed-by: robin.io app.kubernetes.io/name: robin annotations: snapshot.storage.kubernetes.io/is-default-class: "true" driver: robin deletionPolicy: Delete
Upgrade existing virtual machines to Distributed Cloud connected version 1.7.0
You must add the following to the spec
section of the VMRuntime
resource on your cluster
and then apply the modified resource to the cluster before upgrading the cluster to
Distributed Cloud connected software to version 1.7.0:
spec: haPolicy: defaultRecoveryStrategy: Reschedule nodeHeartbeatInterval: 15s nodeMonitorGracePeriod: 55s
Enable VM Runtime on GDC support on Distributed Cloud connected
By default, VM Runtime on GDC virtual machine support is disabled on Distributed Cloud connected. To enable it, complete the steps in this section. The instructions in this section assume that you have a fully functioning Distributed Cloud connected cluster.
To enable the VM Runtime on GDC virtual machine subsystem, complete the following steps:
Modify the
VMRuntime
custom resource with the following contents and apply it to your cluster:apiVersion: vm.cluster.gke.io/v1 kind: VMRuntime metadata: annotations: baremetal.cluster.gke.io/vmrumtime-force-disable: "false" vm.cluster.gke.io/enable-vm-backup: "true" spec: enabled: true storage: defaultStorageClass: robin-block-immediate haPolicy: defaultRecoveryStrategy: Reschedule nodeHeartbeatInterval: 15s nodeMonitorGracePeriod: 55s
This process typically takes several minutes to complete.
Use the following command to verify that the
VMRuntime
custom resource has been applied to your cluster:kubectl get vmruntime
The command returns output similar to the following example:
NAME AGE ENABLED READY PREFLIGHTCHECK vmruntime 5m true true true
Modify the
storageprofile
for therobin-block-immediate
storage class with the following contents and apply it to your cluster:apiVersion: cdi.kubevirt.io/v1beta1 kind: StorageProfile metadata: name: robin-block-immediate spec: claimPropertySets: accessModes: ReadWriteMany volumeMode: Block
Install the virtctl
management tool
You need the virtctl
client tool to manage virtual machines on your
Distributed Cloud connected cluster. To install the tool, complete the
following steps:
Install the
virtctl
client tool as akubectl
plugin:export VERSION=v0.59.0-anthos1.28-gke.8 gcloud storage cp gs://anthos-baremetal-release/virtctl/${VERSION}/linux-amd64/virtctl/usr/local/bin/virtctl cd /usr/local/bin sudo ln -s virtctl kubectl-virt sudo chmod a+x virtctl cd -
Verify that the
virt
plugin is installed:kubectl plugin list
If the plugin has been successfully installed, the command's output lists
kubectl-virt
as one of the plugins.
Create a Cloud Storage bucket for virtual machine images
Complete the steps in this section to create a Cloud Storage bucket for your virtual machine images. If you already have an established image repository, skip this section.
Complete the steps in Create buckets to create a bucket.
Configure a service account and a key to for accessing the bucket with the following commands:
export PROJECT_ID=$(gcloud config get-value project) gcloud iam service-accounts create image-access gcloud projects add-iam-policy-binding $PROJECT_ID \ --member="serviceAccount:image-access@${PROJECT_ID}.iam.gserviceaccount.com" \ --role="roles/storage.objectViewer" \ gcloud iam service-accounts keys create ./image-access-gcr.json \ --iam-account="image-access@${PROJECT_ID}.iam.gserviceaccount.com"
Create a secret in the cluster for accessing the bucket. If your bucket is public, skip this step. This secret must exist in the same namespace as your virtual machine disks. You must create a secret in each affected namespace.
kubectl create secret generic gcs-image-sa --from-file=creds-gcp.json=./image-access-gcr.json -n NAMESPACE
Replace
CLUSTER_ID
with the name of the target namespace.Store your images in the bucket.
Create a virtual machine disk from a virtual machine image
Complete the steps in this section to create a virtual machine disk from a virtual machine image.
Create a disk from an image stored in Cloud Storage
Create a virtual machine disk from a virtual machine stored in your Cloud Storage bucket by applying the following configuration to your cluster:
apiVersion: vm.cluster.gke.io/v1 kind: VirtualMachineDisk metadata: name: DISK_NAME namespace: NAMESPACE spec: source: gcs: url: gs://{PROJECT_ID}-vm-images/IMAGE_FILE secretRef: gcs-image-sa size: DISK_SIZE storageClassName: robin-block-immediate
Replace the following:
DISK_NAME
: the name of this virtual machine disk.NAMESPACE
: the target namespace.IMAGE_FILE
: the name of virtual machine image file.DISK_SIZE
: the desired disk size. This must be larger than thevirtual-size
value of the virtual machine image file. You can find this value with the commandqemu-img info DISK_SIZE
.
If you don't specify a storageClassName
value, the default value specified in
the VMRuntime
resource is used.
Create a disk from an existing disk image
Create a virtual machine disk from an existing virtual machine disk or image file on your cluster as follows.
Create the target disk by applying the following configuration to your cluster:
apiVersion: vm.cluster.gke.io/v1 kind: VirtualMachineDisk metadata: name: IMAGE_DISK_NAME namespace: NAMESPACE spec: source: virtualMachineDisk: name: EXISTING_DISK_NAME size: DISK_SIZE storageClassName: robin-block-immediate
Replace the following:
IMAGE_DISK_NAME
: the name of this virtual machine disk.NAMESPACE
: the target namespace.EXISTING_DISK_NAME
: the name of the existing virtual machine disk.DISK_SIZE
: the desired disk size. This must be equal to or larger than the size of the existing disk or image file.
You can obtain the size of your existing disk or image file using the
qemu-img
tool as follows:qemu-img info EXISTING_DISK_NAME
If you don't specify a
storageClassName
value, the default value specified in theVMRuntime
resource is used.Create a new disk from your existing image file by applying the following configuration to your cluster:
apiVersion: vm.cluster.gke.io/v1 kind: VirtualMachineDisk metadata: name: DISK_NAME namespace: NAMESPACE spec: source: virtualMachineDisk: name: IMAGE_DISK_NAME size: DISK_SIZE storageClassName: robin-block-immediate
Replace the following:
IMAGE_DISK_NAME
: the name of the virtual machine disk you created in the previous step.NAMESPACE
: the target namespace.DISK_SIZE
: the desired disk size. Must be equal to the size of the virtual machine disk you created in the previous step.
Resize the target disk to the desired size using the following command:
kubectl edit gdisk DISK_NAME -n NAMESPACE
Replace the following:
DISK_NAME
: the name of the virtual machine disk you created in the previous step.NAMESPACE
: the target namespace.
Modify the
spec.size
value in the configuration of the disk and apply it to your cluster.
Create an empty disk
Create an empty virtual machine disk by applying the following configuration to your cluster:
apiVersion: vm.cluster.gke.io/v1 kind: VirtualMachineDisk metadata: name: DISK_NAME namespace: NAMESPACE spec: size: DISK_SIZE storageClassName: robin-block-immediate
Replace the following:
DISK_NAME
: the name of this virtual machine disk.NAMESPACE
: the target namespace.DISK_SIZE
: the desired disk size in gibibytes. This must be larger than thevirtual-size
value of the virtual machine image file. You can find this value with the commandqemu-img info DISK_SIZE
.
If you don't specify a storageClassName
value, the default value specified in
the VMRuntime
resource is used.
Configure virtual networking
Follow the steps in Networking to configure the virtual networking for your virtual machines.
Create a virtual machine
Complete the steps in this section to create a virtual machine on your Distributed Cloud connected server deployment. The instructions in this section are examples meant to illustrate configurations for different scenarios. For detailed information on configuring virtual machines, see Create a VM with specific CPU and memory resources using VM Runtime on GDC.
Create a virtual machine from a bootable disk image
To create a virtual machine from a bootable disk image, apply the following configuration to your cluster:
kind: VirtualMachine metadata: name: my-virtual-machine namespace: my-vm-namespace spec: osType: Linux/Windows guestEnvironment: {} // comment out this line to enable guest environment for access management autoRestartOnConfigurationChange: true compute: cpu: vcpus: 6 memory: capacity: 8Gi interfaces: - name: eth0 networkName: network-410 ipAddresses: - 10.223.237.10/25 disks: - virtualMachineDiskName: my-boot-disk boot: true - virtualMachineDiskName: my-data-disk
Replace the following:
DISK_NAME
: the name of this virtual machine disk.NAMESPACE
: the target namespace.
Create a virtual machine from an ISO optical disc image
To create a virtual machine from an ISO optical disc image, complete the steps in Create a Windows VM from ISO image in Google Distributed Cloud.
Create a virtual machine with GPU support
Complete the steps
Access a virtual machine
Complete the steps in this section to access a virtual machine running on your Distributed Cloud connected server deployment.
Obtain access credentials
Complete the steps in this section to obtain the credentials necessary to access your virtual machine using the Linux guest environment feature.
Enable the Linux guest environment for the target virtual machine by applying the following configuration to your cluster:
kind: VirtualMachine metadata: name: my-virtual-machine namespace: my-vm-namespace spec: osType: Linux guestEnvironment: {} autoRestartOnConfigurationChange: true
Generate an
id_rsa.pub
file containing an SSH key pair with the following command:ssh-keygen -t rsa
Create a
VirtualMachineAccessRequest
resource by applying the following configuration to your cluster:apiVersion: vm.cluster.gke.io/v1alpha1 kind: VirtualMachineAccessRequest metadata: name: RESOURCE_NAME namespace: NAMESPACE spec: vm: VM_NAME user: USER_NAME ssh: key: RSA_KEY ttl: 2h
Replace the following:
RESOURCE_NAME
: a descriptive name for this virtual machine access request resource.NAMESPACE
: the target namespace.VM_NAME
: the name of the target virtual machine.USER_NAME
: the name of the user to whom access is being granted.RSA_KEY
: the contents of theid_rsa.pub
file you generated in the previous step.
Check the status of the access request with the following command:
kubectl get vmar
When the command returns a
Configured
status, proceed to the next step.Access the virtual machine with SSH or Remote Desktop:
- If the virtual machine is connected to your local network, you can access it directly.
- If the virtual machine is connected to the Pod network, you must create a load balancer service to access the required ports.
Start, restart, or stop a virtual machine
Use the following commands to start, restart, or stop a virtual machine:
- Start a virtual machine:
kubectl virt start vm
VM_NAME
-n
NAMESPACE
- Restart a virtual machine:
kubectl virt restart vm
VM_NAME
-n
NAMESPACE
- Stop a virtual machine:
kubectl virt stop vm
VM_NAME
-n
NAMESPACE
Replace the following:
VM_NAME
: the name of the target virtual machine.NAMESPACE
: the target namespace.
Back up a virtual machine
This section describes how to create a backup repository and back up your virtual machines.
Create a backup repository
Complete the following steps to create a backup repository for your virtual machines:
Create a Cloud Storage bucket as described in Create buckets.
Create a service account and key to access the backup bucket using the following commands:
export PROJECT_ID=$(gcloud config get-value project) export SVC_ACCOUNT=backup-access gcloud iam service-accounts create $SVC_ACCOUNT gcloud projects add-iam-policy-binding $PROJECT_ID \ --member="serviceAccount:${SVC_ACCOUNT}@${PROJECT_ID}.iam.gserviceaccount.com" \ --role="roles/backupdr.cloudStorageOperator" gcloud storage hmac create ${SVC_ACCOUNT}@${PROJECT_ID}.iam.gserviceaccount.com > hmac_temp_key_file awk 'NR==1{print "ACCESS_ID=" $NF} NR==2{print "SECRET=" $NF}' < hmac_temp_key_file > hmac_key_file
Configure the secret for accessing the backup bucket using the following commands:
source ./hmac_key_file kubectl create secret generic gcs-hmac-secret \ --from-literal=access-key=$SECRET \ --from-literal=access-key-id=$ACCESS_ID -n NAMESPACE
Replace
NAMESPACE
with the name of the target namespace.Configure the cluster to access the repository by applying the following configuration to your cluster:
apiVersion: backup.gdc.goog/v1 kind: BackupRepository metadata: name: "REPOSITORY_NAME" spec: secretReference: namespace: NAMESPACE name: gcs-hmac-secret endpoint: "https://storage.googleapis.com" type: "S3" s3Options: bucket: $BUCKET_NAME region: "REGION" forcePathStyle: true importPolicy: "ReadWrite" # Force attachment for convenience. force: true
Verify that the cluster can access the backup repository using the following command:
kubectl get BackupRepository
Replace the following:
REPOSITORY_NAME
: a descriptive name for the repository.BUCKET_NAME
: the name of the backup bucket.NAMESPACE
: the target namespace.REGION
: the Google Cloud region in which the target Distributed Cloud cluster has been created.
Back up virtual machines
Complete the steps in this section to back up your virtual machines.
Create a
VirtualMachineBackupPlanTemplate
resource in the namespace in which the target virtual machine resides by applying the following configuration to the cluster:apiVersion: vm.cluster.gke.io/v1 kind: VirtualMachineBackupPlanTemplate metadata: name: default-backup-plan-template namespace: NAMESPACE spec: backupRepository: REPOSITORY_NAME
Replace the following:
REPOSITORY_NAME
: the name of the backup repository.NAMESPACE
: the target namespace.REGION
: the Google Cloud region in which the target Distributed Cloud cluster has been created.
Trigger the backup by creating a
VMBackupRequest
resource with the following configuration and applying it to the cluster:apiVersion: vm.cluster.gke.io/v1 kind: VirtualMachineBackupRequest metadata: name: myVmBackup namespace: NAMESPACE spec: vmBackupPlanTemplate: default-backup-plan-template virtualMachine: VM_NAME vmBackupName: BACKUP_NAME
Replace the following:
NAMESPACE
: the target namespace.VM_NAME
: the name of the target virtual machine.BACKUP_NAME
: a descriptive name for this backup.
Verify the integrity of the backup with the following command:
kubectl get vmbackup BACKUP_NAME -n NAMESPACE
Replace the following:
BACKUP_NAME
: the name of the target backup.NAMESPACE
: the target namespace.
The command returns output similar to the following:
apiVersion: vm.cluster.gke.io/v1 kind: VirtualMachineBackup metadata: creationTimestamp: "2024-04-09T17:57:44Z" finalizers: - vm.cluster.gke.io/virtual-machine-backup-finalizer generation: 1 name: vmt13-backup-0409-2 namespace: default ownerReferences: - apiVersion: backup.gdc.goog/v1 kind: Backup name: vmt13-backup-0409-2 uid: 0ee0b92c-1e27-48cc-8f8f-5606ea925e88 resourceVersion: "36192759" uid: e471f8c7-637c-485f-acda-108017a5638f spec: backupConfig: backupRepository: default backupScope: selectedVirtualMachines: - resourceName: vm-t13 volumeStrategy: Portable vmBackupPlan: my-backup-plan-template-vm-vm-t13-portable status: backedUpVirtualMachineDisks: - vm-t13-boot-disk - vm-t13-data-disk backedUpVirtualMachines: - vm-t13 backup: vmt13-backup-0409-2 backupStatus: clusterMetadata: k8sVersion: "1.28" completeTime: "2024-04-09T18:07:36Z" createTime: "2024-04-09T17:57:44Z" jobCreated: true resourceCount: 849 sizeBytes: 1948672 state: Succeeded
Create a backup plan for a virtual machine
You can create a scheduled backup plan for a virtual machine. For example:
apiVersion: vm.cluster.gke.io/v1 kind: VirtualMachineBackupPlan metadata: name: BACKUP_PLAN_NAME namespace: NAMESPACE spec: backupConfig: backupRepository: default backupScope: selectedVirtualMachines: - resourceName: VM_NAME volumeStrategy: Portable backupSchedule: cronSchedule: "*/60 * * * *" // Meaning one backup per 60 minutes paused: false retentionPolicy: backupDeleteLockDays: 0 backupRetainDays: 1 locked: false
Replace the following:
BACKUP_PLAN_NAME
: a descriptive name for this backup plan.NAMESPACE
: the target namespace.VM_NAME
: the name of the target virtual machine.
Restore a virtual machine from a backup
Complete the steps in this section to restore a virtual machine from a backup.
Trigger the restore process by creating a
VirtualMachineRestoreRequest
resource with the following configuration and applying it to your cluster:apiVersion: vm.cluster.gke.io/v1 kind: VirtualMachineRestoreRequest metadata: name: restore-req namespace: NAMESPACE spec: vmBackup: BACKUP_NAME restoreName: RESTORE_NAME restoredResourceName: RESTORED_VM_NAME restoredResourceDescription: "any comment"
Replace the following:
BACKUP_NAME
: the name of the target backup.RESTORE_NAME
: a descriptive name for this restore operation.NAMESPACE
: the target namespace.RESTORED_VM_NAME
: the name given to the virtual machine during restoration. This name must not conflict with any virtual machines already present on the cluster.
Use the following command to check the progress of the restore operation:
kubectl get virtualmachinerestores.vm.cluster.gke.io RESTORE_NAME -n NAMESPACE
Replace the following:
RESTORE_NAME
: the name of the target restore operation.NAMESPACE
: the target namespace.
The command returns output similar to the following:
apiVersion: vm.cluster.gke.io/v1 kind: VirtualMachineRestore metadata: creationTimestamp: "2024-04-09T18:09:51Z" finalizers: - vm.cluster.gke.io/virtual-machine-restore-finalizer generation: 1 name: vmt13-restore-0409-2-1 namespace: default ownerReferences: - apiVersion: backup.gdc.goog/v1 kind: Restore name: vmt13-restore-0409-2-1 uid: 4ce1ca83-eba0-4cc3-bad3-af6cf9185d7d resourceVersion: "36194596" uid: aba50b59-e18d-4687-ad11-47baa45478b4 spec: targetVirtualMachineDisks: - vm-t13-boot-disk - vm-t13-data-disk targetVirtualMachines: - vm-t13 vmBackup: vmt13-backup-0409-2 status: restore: vmt13-restore-0409-2-1 restoreStatus: completeTime: "2024-04-09T18:10:00Z" jobCreated: true resourcesRestoredCount: 5 restoredVolumesCount: 2 startTime: "2024-04-09T18:09:51Z" state: Succeeded stateReason: restore is successful
Delete a virtual machine backup
To delete a virtual machine backup, create a VirtualMachineDeleteBackupRequest
resource with the following configuration and apply it to your cluster:
apiVersion: vm.cluster.gke.io/v1 kind: VirtualMachineDeleteBackupRequest metadata: name: vmdbr namespace: BACKUP_NAME spec: vmBackup: NAMESPACE
Replace the following:
NAMESPACE
: the target namespace.BACKUP_NAME
: the name of the target backup.
What's next
- Manage virtual machines on Distributed Cloud connected racks
- Deploy workloads on Distributed Cloud connected
- Manage GPU workloads
- Manage zones
- Manage machines
- Manage clusters
- Manage node pools