Enable nested virtualization

Linux

---

This document describes how to enable nested virtualization on a virtual machine (VM) instance and how to confirm that you can create a nested VM. Enable nested virtualization on a VM by using one of the following methods:

• Recommended. Enable nested virtualization directly on a new or existing VM by setting the enableNestedVirtualization field to true while creating the VM or by updating the VM. This is the recommended method because it doesn't require that you create a custom image or use the special license key.

• Enable nested virtualization by using the special license key by creating a boot disk, creating a custom image with the special nested virtualization license key, and then creating a VM that uses the custom image.

Before you begin

• If you haven't already, then set up authentication. Authentication is the process by which your identity is verified for access to Google Cloud services and APIs. To run code or samples from a local development environment, you can authenticate to Compute Engine by selecting one of the following options:
  

  Select the tab for how you plan to use the samples on this page:

  gcloud

  1. Install the Google Cloud CLI, then initialize it by running the following command:

     gcloud init

     Note: If you installed the gcloud CLI previously, make sure you have the latest version by running gcloud components update.
  2. Set a default region and zone.

  REST

  To use the REST API samples on this page in a local development environment, you use the credentials you provide to the gcloud CLI.

  Install the Google Cloud CLI, then initialize it by running the following command:

  gcloud init

  For more information, see Authenticate for using REST in the Google Cloud authentication documentation.

Enable nested virtualization directly on a new VM

Permissions required for this task

To perform this task, you must have the following permissions:

• For enabling nested virtualization directly on a new VM, you need the compute.instances.create permissions on the project, folder, or organization.

Enable nested virtualization directly on a VM by using the following procedure.

gcloud

Create an L1 VM with nested virtualization enabled by using the following gcloud compute instances create command:

gcloud compute instances create VM_NAME \
  --enable-nested-virtualization \
  --zone=ZONE \
  --min-cpu-platform="Intel Haswell"

Replace the following:

• VM_NAME: the name of the new L1 VM with nested virtualization enabled

• ZONE: the zone for the new L1 VM with nested virtualization enabled

REST

Create an L1 VM with nested virtualization enabled by using the following instances.insert method:

POST https://compute.googleapis.com/compute/v1/projects/PROJECT_ID/zones/ZONE/instances
{
  ...
  "name": "VM_NAME",
  ...
  "minCpuPlatform": "Intel Haswell",
  "advancedMachineFeatures": {
    "enableNestedVirtualization": true
  },
  ...
}

Replace the following:

• PROJECT_ID: the project ID

• ZONE: the zone for the new L1 VM with nested virtualization enabled

• VM_NAME: the name of the new L1 VM with nested virtualization enabled

Enable nested virtualization directly on an existing VM

Permissions required for this task

To perform this task, you must have the following permissions:

• For enabling nested virtualization directly on a new VM, you need the compute.instances.create permissions on the project, folder, or organization.

Enable nested virtualization on an existing VM by using the following procedure.

gcloud

1. Export the properties of the VM by using the following gcloud compute instances export command:

   gcloud compute instances export VM_NAME \
     --destination=YAML_FILE_PATH \
     --zone=ZONE
   

   Replace the following:

   • VM_NAME: the name of the VM from which to export properties

   • YAML_FILE_PATH: the path and file name of a .yaml file in which to save the exported configuration data

   • ZONE: the zone that contains the VM

2. In the VM configuration file that was saved in FILE_PATH, update the value for enableNestedVirtualization. If the value is not in the file, add the following:

   advancedMachineFeatures:
     enableNestedVirtualization: true
   

3. Update the VM with the value for enableNestedVirtualization by using the following gcloud compute instances update-from-file command:

   gcloud compute instances update-from-file VM_NAME \
     --source=FILE_PATH \
     --most-disruptive-allowed-action=RESTART \
     --zone=ZONE
   

   Replace the following:

   • VM_NAME: the name of the VM to update

   • FILE_PATH: the path to the updated VM configuration file

   • ZONE: the zone that contains the VM to update

REST

Update the value for enableNestedVirtualization by using the following instances.update method:

PUT https://compute.googleapis.com/compute/v1/projects/PROJECT_ID/zones/ZONE/instances/VM_NAME?most_disruptive_allowed_action=RESTART

{
  ⋮
  "advanced_machine_features": {
    ⋮
    "enableNestedVirtualization": "true"
  },
  ⋮
}

Replace the following:

• PROJECT_ID: the ID of the project

• ZONE: the zone that contains the VM

• VM_NAME: the name of the VM from which to export properties

Enable nested virtualization by using the special license key

Permissions required for this task

To perform this task, you must have the following permissions:

• compute.disks.create on the project, folder, or organization
• compute.images.create on the project, folder, or organization
• compute.disks.delete on the project, folder, or organization
• compute.instances.create on the project, folder, or organization

You can enable nested virtualization on VM by creating a custom image with a special license key that enables VMX on the L1 VM. The license key does not incur additional charges.

1. Create a boot disk from a public image or from a custom image. The following example uses debian-cloud for the image project and debian-10 for the image family. If you already have a VM instance with an existing disk, you can skip this step.

   gcloud

   gcloud compute disks create DISK_NAME \
     --zone=ZONE \
     --image-project=debian-cloud \
     --image-family=debian-10
   

   Replace the following:

   • DISK_NAME: the name of the new disk

   • ZONE: the zone to create the disk in

   REST

   POST https://compute.googleapis.com/compute/v1/projects/PROJECT_ID/zones/ZONE/disks
   
   {
     ...
     "name": "DISK_NAME",
     "sourceImage": "projects/debian-cloud/global/images/family/debian-10",
     ...
   }
   

   Replace the following:

   • PROJECT_ID: the project ID

   • ZONE: the zone to create the disk in

   • DISK_NAME: the name of the new disk

2. Create a custom image with the special license key that is required for nested virtualization.

   gcloud

   gcloud compute images create IMAGE_NAME \
     --source-disk DISK_NAME \
     --source-disk-zone ZONE \
     --licenses "https://www.googleapis.com/compute/v1/projects/vm-options/global/licenses/enable-vmx"
   

   Replace the following:

   • IMAGE_NAME: the name of the new image

   • DISK_NAME: the name of the previously created disk

   • ZONE: the zone to create the image in

   REST

   POST https://compute.googleapis.com/compute/v1/projects/PROJECT_ID/global/images
   
   {
     ...
     "licenses": ["projects/vm-options/global/licenses/enable-vmx"],
     "name": "IMAGE_NAME",
     "sourceDisk": "zones/ZONE/disks/DISK_NAME",
     ...
   }
   

   Replace the following:

   • PROJECT_ID: the project ID

   • IMAGE_NAME: the name of the new image

   • ZONE: the zone to create the image in

   • DISK_NAME: the name of the previously created disk

3. Optionally delete the source disk after creating the image with the special license.

   gcloud

   gcloud compute disks delete DISK_NAME --zone=ZONE
   

   Replace the following:

   • DISK_NAME: the name of the disk to delete

   • ZONE: the zone containing the disk to delete

   REST

   DELETE https://compute.googleapis.com/compute/v1/projects/PROJECT_ID/zones/ZONE/disks/DISK_NAME
   

   Replace the following:

   • PROJECT_ID: the project ID

   • ZONE: the zone containing the disk to delete

   • DISK_NAME: the name of the disk to delete

4. Create a VM that uses the new image with the special license. The minimum CPU platform must be "Intel Haswell".

   gcloud

   gcloud compute instances create VM_NAME \
       --zone=ZONE \
       --min-cpu-platform "Intel Haswell" \
       --image IMAGE_NAME
   

   Replace the following:

   • VM_NAME: the name of the VM

   • ZONE: the zone to create the VM in

   • IMAGE_NAME: the name of the previously created image

   REST

   POST https://compute.googleapis.com/compute/v1/projects/PROJECT_ID/zones/ZONE/instances
   
   {
     ...
     "name": "VM_NAME",
     "minCpuPlatform": "Intel Haswell",
     "disks": [
       {
         "initializeParams": {
           "sourceImage": "IMAGE_NAME"
         }
       }
     ]
     ...
   }
   
   

   Replace the following:

   • PROJECT_ID: the project ID

   • VM_NAME: the name of the VM

   • ZONE: the zone to create the VM in

   • IMAGE_NAME: the name of the previously created image

Confirm that nested virtualization is enabled on the VM

1. Connect to the VM instance.

   gcloud compute ssh VM_NAME
   

   Replace VM_NAME with the name of the VM to connect to.

2. Confirm that nested virtualization is enabled. Any response other than 0 confirms that nested virtualization is enabled.

   grep -cw vmx /proc/cpuinfo
   

What's next

• Create nested VMs
• Troubleshoot nested virtualization