About nested virtualization

Stay organized with collections Save and categorize content based on your preferences.

This document describes Compute Engine support for nested virtualization. Nested virtualization lets you run virtual machine (VM) instances inside of other VMs so you can create your own virtualization environments. To support nested virtualization, Compute Engine adds Intel VT-x instructions to VMs, so when you create a VM, the hypervisor that is already on that VM can run additional VMs.

Compute Engine VMs run on a physical host that has Google's security-hardened, KVM-based hypervisor. With nested virtualization, the physical host and its hypervisor are the level 0 (L0) environment. The L0 environment can host multiple level 1 (L1) VMs. On each L1 VM is another hypervisor, which is used to install the level 2 (L2) VMs. Figure 1 shows the relationship between the physical host, the L1 VMs, and the L2 VMs:

Figure 1. L0 physical host with L1 VMs and L2 VMs.

Use cases

Scenarios where you might consider using nested virtualization include the following:

  • You have VMs that you can't run on Compute Engine: For example, you might have a disaster recovery solution for an on-premises workload that is running on VMs that fail over to Compute Engine VMs. Running nested virtualization might save you time that you would use to port your VMs to Compute Engine.

  • You have a software-validation framework that you use to test and validate new versions of a software package on numerous versions of different OSes: Using nested virtualization lets you avoid converting and managing a library of Compute Engine images.

Performance considerations

Even with hardware-assisted nested virtualization, nested VMs might experience a 10% or greater decrease in performance for workloads that are CPU-bound and possibly greater than a 10% decrease for workloads that are input/output bound.

Restrictions

L1 VMs have the following restrictions:

  • You must run Linux-based OSes; you can't use Windows Server images.

  • You can't use E2, N2D, N1 with attached GPUs, and A2 machine types.

  • You must use Intel Haswell or later processors; AMD processor are not supported. If the default processor for a zone is Sandy Bridge or Ivy Bridge, change the minimum CPU selection for the VMs in that zone to Intel Haswell or later. For information about the processors supported in each zone, see Available regions and zones.

L2 VMs have the following restrictions:

Using nested virtualization

To use nested virtualization, complete the following steps:

  1. Check whether the nested virtualization constraint is disabled.

  2. Create an L1 VM that has nested virtualization enabled.

  3. Create a nested L2 VM.

If you run into any issues while creating a VM that has nested virtualization enabled or creating nested VMs, see troubleshooting nested virtualization.