Schedule VMs that use VM Runtime on Google Distributed Cloud

Version 1.13 

• Version 1.28 (latest)
• Version 1.16
• Version 1.15
• Earlier versions (unsupported)

This document is intended for application owners and platform administrators that run GKE on Bare Metal. This document shows you how to use schedule configurations such as affinity and anti-affinity for VMs that use VM Runtime on Google Distributed Cloud.

Before you begin

To complete this document, you need access to the following resources:

• Access to GKE on Bare Metal version 1.12.0 (anthosBareMetalVersion: 1.12.0) or higher cluster. You can use any cluster type capable of running workloads. If needed, try GKE on Bare Metal on Compute Engine or see the cluster creation overview.
• The virtctl client tool installed as a plugin for kubectl. If needed, install the virtctl client tool.

Note: The VirtualMachine manifests in this document use the default networking configuration. For more information about VM networking, see Create and use virtual networks for VM Runtime on Google Distributed Cloud.

Schedule configurations overview

Schedule configurations are optional values in VM Runtime on Google Distributed Cloud. If no scheduling configuration is specified, the VM defaults to Kubernetes default scheduling behavior.

With the default scheduling behavior, VMs are spread across your cluster. The scheduler looks at the current node resource availability, such as CPU and memory, and schedules VMs on nodes to distribute the compute demands. If you have no specific requirements, you don't need to define any schedule configurations.

The following three fields are available to schedule VMs:

• nodeSelector: specifies node labels that the host node of a VM must have. VM Runtime on Google Distributed Cloud schedules the VM only on those nodes that have a specified label.
• Affinity: specifies the affinity rules of the VM. It includes node affinity and inter-VM affinity or anti-affinity. You define a soft or hard requirement for the scheduler:
  • preferredDuringSchedulingIgnoredDuringExecution: is a soft requirement. The scheduler tries to honor your request. If the scheduler can't honor the request, the VM may be scheduled on an unpreferred node.
  • requiredDuringSchedulingIgnoredDuringExecution: is a hard requirement. The scheduler tries to honor your request. If no nodes are available that match your requirement, the VM is not scheduled.
• Tolerations: allows the VM to be scheduled onto nodes with matching taints.

You can define any of these scheduling configurations to support your compute workloads and scheduling needs. In addition to scheduling configurations, VM scheduling is contingent upon available resources.

Note: To update scheduling configurations on existing VMs, the VM must first be in a Stopped state. After you make changes, start the VM. The scheduler uses the new configuration to run the VM on an appropriate nodes. For more information, see manage the powers state of a VM.

VM Runtime on Google Distributed Cloud uses the same VM scheduling logic and manifest structure as Kubernetes to assign Pods to Nodes. For more information on these scheduling configurations, see the following links:

• nodeSelector
• Affinity and anti-affinity
• Taints and tolerations

Put VMs on a specific node

If you have nodes with specific hardware configurations, you can schedule VMs to only run on these nodes. For example, your VM might want a particular CPU chipset, or need GPU support. You can use a basic nodeSelector, or more flexible affinity rules, to schedule VMs to run on these nodes.

nodeSelector

The following VirtualMachine manifest uses a nodeSelector for a hard scheduling requirement. If no node is available that meets the scheduling configuration, the VM can't be scheduled.

1. Create a VirtualMachine manifest, such as my-scheduled-vm.yaml, in the editor of your choice:

   nano my-scheduled-vm.yaml
   

2. Copy and paste the following YAML manifest:

   apiVersion: vm.cluster.gke.io/v1
   kind: VirtualMachine
   metadata:
     name: VM_NAME
   spec:
     interfaces:
       - name: eth0
         networkName: pod-network
         default: true
     disks:
       - virtualMachineDiskName: VM_NAME-boot-dv
         boot: true
     scheduling:
       nodeSelector:
         kubernetes.io/hostname: NODE_NAME
   

   Replace the following values:

   • VM_NAME: the name of your VM.
   • NODE_NAME: the node(s) that you want to schedule your VM on.

   The boot disk named VM_NAME-boot-dv must already exist. For more information, see Create a VM boot disk.

3. Save and close the VM manifest in your editor.

4. Create the VM and schedule configuration using kubectl:

   kubectl apply -f my-scheduled-vm.yaml
   

Affinity

The following VirtualMachine manifest uses affinity for a soft scheduling requirement. The scheduler tries to honor your request. If the scheduler can't honor the request, the VM is scheduled on an unpreferred node.

1. Create a VirtualMachine manifest, such as my-scheduled-vm.yaml, in the editor of your choice:

   nano my-scheduled-vm.yaml
   

2. Copy and paste the following YAML manifest:

   apiVersion: vm.cluster.gke.io/v1
   kind: VirtualMachine
   metadata:
     name: VM_NAME
   spec:
     interfaces:
       - name: eth0
         networkName: pod-network
         default: true
     disks:
       - virtualMachineDiskName: VM_NAME-boot-dv
         boot: true
     scheduling:
         affinity:
           nodeAffinity:
             preferredDuringSchedulingIgnoredDuringExecution:
             - weight: 100
               preference:
                 matchExpressions:
                 - key: kubernetes.io/hostname
                   operator: In
                   values:
                   - NODE_NAME
   

   Replace the following values:

   • VM_NAME: the name of your VM.
   • NODE_NAME: the node(s) that you want to schedule your VM on.

   The boot disk named VM_NAME-boot-dv must already exist. For more information, see Create a VM boot disk.

   Note: For a hard scheduling requirement, set requiredDuringSchedulingIgnoredDuringExecution rather than preferredDuringSchedulingIgnoredDuringExecution.

3. Save and close the VM manifest in your editor.

4. Create the VM and schedule configuration using kubectl:

   kubectl apply -f my-scheduled-vm.yaml
   

Don't put VMs on a specific node

Certain VMs might have workloads that don't run on a particular node. You can use anti-affinity rules to avoid scheduling VMs on these nodes.

The following VirtualMachine manifest uses affinity for a soft scheduling requirement. The scheduler tries to honor your request. If the scheduler can't honor the request, the VM may be scheduled on an inappropriate node.

1. Create a VirtualMachine manifest, such as my-scheduled-vm.yaml, in the editor of your choice:

   nano my-scheduled-vm.yaml
   

2. Copy and paste the following YAML manifest:

   apiVersion: vm.cluster.gke.io/v1
   kind: VirtualMachine
   metadata:
     name: VVM_NAME
   spec:
     interfaces:
       - name: eth0
         networkName: pod-network
         default: true
     disks:
       - virtualMachineDiskName: VM_NAME-boot-dv
         boot: true
     scheduling:
       affinity:
         nodeAffinity:
           preferredDuringSchedulingIgnoredDuringExecution:
           - weight: 100
             preference:
               matchExpressions:
               - key: kubernetes.io/hostname
                 operator: NotIn
                 values:
                 - NODE_NAME
   

   Replace the following values:

   • VM_NAME: the name of your VM.
   • NODE_NAME: the node that you want to schedule your VM on.

   The boot disk named VM_NAME-boot-dv must already exist. For more information, see Create a VM boot disk.

   Note: For a hard scheduling requirement, set requiredDuringSchedulingIgnoredDuringExecution rather than preferredDuringSchedulingIgnoredDuringExecution.

3. Save and close the VM manifest in your editor.

4. Create the VM and schedule configuration using kubectl:

   kubectl apply -f my-scheduled-vm.yaml
   

Keep VMs apart

Your compute workloads might have VMs that should be spread across nodes for high availability, such as a pool of frontend VMs. You can use inter-VM anti-affinity rules to avoid scheduling VMs together on nodes.

The following VirtualMachine manifest uses affinity for a soft scheduling requirement. The scheduler tries to honor your request. If the scheduler can't honor the request, the VM may be scheduled on an inappropriate node.

1. Create a VirtualMachine manifest, such as my-scheduled-vm.yaml, in the editor of your choice:

   nano my-scheduled-vm.yaml
   

2. Copy and paste the following YAML manifest:

   apiVersion: vm.cluster.gke.io/v1
   kind: VirtualMachine
   metadata:
     name: VM_NAME
     labels:
       KEY:VALUE
   spec:
     interfaces:
       - name: eth0
         networkName: pod-network
         default: true
     disks:
       - virtualMachineDiskName: VM_NAME-boot-dv
         boot: true
     scheduling:
       affinity:
         podAntiAffinity:
           preferredDuringSchedulingIgnoredDuringExecution:
           - weight: 100
             podAffinityTerm:
               topologyKey: kubernetes.io/hostname
               labelSelector:
                 matchLabels:
                   KEY:VALUE
   

   Replace the following values:

   • VM_NAME: the name of your VM.
   • KEY:VALUE: the key:value label to apply to your VMs that you want to schedule across different nodes. For more information, see Labels and selectors.

   The boot disk named VM_NAME-boot-dv must already exist. For more information, see Create a VM boot disk.

   Note: For a hard scheduling requirement, set requiredDuringSchedulingIgnoredDuringExecution rather than preferredDuringSchedulingIgnoredDuringExecution.

3. Save and close the VM manifest in your editor.

4. Create the VM and schedule configuration using kubectl:

   kubectl apply -f my-scheduled-vm.yaml
   

Keep VMs together

Your compute workloads might have VMs that should be kept together on nodes to reduce latency, such as a middleware and database tier. You can use inter-VM affinity rules to schedule VMs together on nodes.

The following VirtualMachine manifest uses affinity for a soft scheduling requirement. The scheduler tries to honor your request. If the scheduler can't honor the request, the VM may be scheduled on an inappropriate node.

1. Create a VirtualMachine manifest, such as my-scheduled-vm.yaml, in the editor of your choice:

   nano my-scheduled-vm.yaml
   

2. Copy and paste the following YAML manifest:

   apiVersion: vm.cluster.gke.io/v1
   kind: VirtualMachine
   metadata:
     name: VM_NAME
     labels:
       KEY:VALUE
   spec:
     interfaces:
       - name: eth0
         networkName: pod-network
         default: true
     disks:
       - virtualMachineDiskName: VM_NAME-boot-dv
         boot: true
     scheduling:
       affinity:
         podAffinity:
           preferredDuringSchedulingIgnoredDuringExecution
           - podAffinityTerm:
               topologyKey: kubernetes.io/hostname
               labelSelector:
                 matchLabels:
                   KEY:VALUE
             weight: 100
   

   Replace the following values:

   • VM_NAME: the name of your VM.
   • KEY:VALUE: the key:value label pair to apply to your VMs that you want to schedule across different nodes. For more information, see Labels and selectors.

   The boot disk named VM_NAME-boot-dv must already exist. For more information, see Create a VM boot disk.

   Note: For a hard scheduling requirement, set requiredDuringSchedulingIgnoredDuringExecution rather than preferredDuringSchedulingIgnoredDuringExecution.

3. Save and close the VM manifest in your editor.

4. Create the VM and schedule configuration using kubectl:

   kubectl apply -f my-scheduled-vm.yaml
   

Schedule VMs on nodes with taints

Taints are a scheduling property that lets nodes only allow VMs with specified tolerations to be scheduled to run on them. You can apply a taint to a node, then in the VirtualMachine manifest define a toleration to let the VM run on the node. For more information, see Taints and tolerations.

1. Create a VirtualMachine manifest, such as my-scheduled-vm.yaml, in the editor of your choice:

   nano my-scheduled-vm.yaml
   

2. Copy and paste the following YAML manifest:

   apiVersion: vm.cluster.gke.io/v1
   kind: VirtualMachine
   metadata:
     name: VM_NAME
   spec:
     interfaces:
       - name: eth0
         networkName: pod-network
         default: true
     disks:
       - virtualMachineDiskName: VM_NAME-boot-dv
         boot: true
     scheduling:
       tolerations:
       - key: KEY_NAME
         operator: "Equal"
         value: KEY_VALUE
         effect: "NoSchedule"
   

   Replace the following values:

   • VM_NAME: the name of your VM.
   • KEY_NAME: the key name of your toleration that matches the taint on the node.
   • KEY_VALUE: the value of the key for your toleration that matches the taint on the node.

   The boot disk named VM_NAME-boot-dv must already exist. For more information, see Create a VM boot disk.

3. Save and close the VM manifest in your editor.

4. Create the VM and schedule configuration using kubectl:

   kubectl apply -f my-scheduled-vm.yaml
   

What's next

• Edit a VM in GKE on Bare Metal.