About nested virtualization

Linux

---

This document describes Compute Engine support for nested virtualization. Nested virtualization lets you run virtual machine (VM) instances inside of other VMs so you can create your own virtualization environments. To support nested virtualization, Compute Engine adds Intel VT-x instructions to VMs, so when you create a VM, the hypervisor that is already on that VM can run additional VMs.

Compute Engine VMs run on a physical host that has Google's security-hardened, KVM-based hypervisor. With nested virtualization, the physical host and its hypervisor are the level 0 (L0) environment. The L0 environment can host multiple level 1 (L1) VMs. On each L1 VM is another hypervisor, which is used to install the level 2 (L2) VMs. Figure 1 shows the relationship between the physical host, the L1 VMs, and the L2 VMs:

Figure 1. L0 physical host with L1 VMs and L2 VMs.

Use cases

Scenarios where you might consider using nested virtualization include the following:

• You have VMs that you can't run on Compute Engine: For example, you might have a disaster recovery solution for an on-premises workload that is running on VMs that fail over to Compute Engine VMs. Running nested virtualization might save you time that you would use to port your VMs to Compute Engine.

• You have a software-validation framework that you use to test and validate new versions of a software package on numerous versions of different OSes: Using nested virtualization lets you avoid converting and managing a library of Compute Engine images.

Performance considerations

Even with hardware-assisted nested virtualization, nested VMs might experience a 10% or greater decrease in performance for workloads that are CPU-bound and possibly greater than a 10% decrease for workloads that are input/output bound.

Restrictions

L1 VMs have the following restrictions:

• The only hypervisor supported in an L1 VM is Linux KVM; Microsoft Hyper-V isn't supported.

• You can't use the following VMs:

  • E2 VMs

  • Memory-optimized VMs

  • VMs powered by AMD and Arm processors

  For information about the processors supported in each zone, see Available regions and zones.

L2 VMs have the following restrictions:

• For licensed operating systems, you must bring your own licenses.

Using nested virtualization

To use nested virtualization, complete the following steps:

1. Check whether the nested virtualization constraint is disabled

2. Create an L1 VM that has nested virtualization enabled

3. Create a nested L2 VM

If you run into any issues while creating a VM that has nested virtualization enabled or creating nested VMs, see Troubleshoot nested virtualization.