Nested virtualization overview


This document describes Compute Engine support for nested virtualization. Nested virtualization lets you run virtual machines (VM) instances inside of other VMs. To support nested virtualization, Compute Engine adds Intel VT-x instructions to VMs, so when you create a VM, the hypervisor that is already on that VM can run additional VMs.

Compute Engine VMs run on a physical host that has Google's security-hardened, KVM-based hypervisor. With nested virtualization, the physical host and its hypervisor are the level 0 (L0) environment. The L0 environment can host multiple level 1 (L1) VMs. On each L1 VM is another hypervisor, which is used to install the level 2 (L2) VMs. Figure 1 shows the relationship between the physical host, the L1 VMs, and the L2 VMs.

Figure 1. L0 physical host with L1 VMs and L2 VMs.

Use cases

Scenarios where you might consider using nested virtualization include the following:

  • You have VMs that you can't run on Compute Engine: For example, you might have a disaster recovery solution for an on-premises workload that is running on VMs that fail over to Compute Engine VMs. Running nested virtualization might save you time that you would use to port your VMs to Compute Engine.

  • You have a software-validation framework that you use to test and validate new versions of a software package on numerous versions of different OSes: Using nested virtualization lets you avoid converting and managing a library of Compute Engine images.

Performance considerations

Even with hardware-assisted nested virtualization, nested VMs might experience a 10% or greater decrease in performance for workloads that are CPU-bound and possibly greater than a 10% decrease for workloads that are input/output bound.

Restrictions

L1 VMs have the following restrictions:

  • You must run Linux-based OSes; you can't use Windows Server images.

  • You cannot use E2 and N2D machine types.

  • You must use Intel Haswell or later processors; AMD processor are not supported. If the default processor for a zone is Sandy Bridge or Ivy Bridge, change the minimum CPU selection for the VMs in that zone to Intel Haswell or later. For information about the processors supported in each zone, see Available regions and zones.

L2 VMs have the following restrictions:

  • You must use an OS that can run QEMU.

Tested operating systems

The following table shows the combinations of the L1 and L2 OSes on which Google runs basic boot and integration tests. If you have trouble running a VM combination that is not shown in the table, reproduce the issue using one of the combinations of tested OSes shown in the following table before contacting Cloud Customer Care.

L1 VM OS L2 VM OS
Debian 9, kernel version 4.9 CentOS 6.5, kernel version 2.6
Debian 9, kernel version 4.9
RHEL 5.11, kernel version 2.6
SLES 12 SP3, kernel version 4.4
Ubuntu 16.04 LTS, kernel version 4.15
Windows Server 2016 Datacenter
SLES 12 SP3, kernel version 4.4 SLES 12 SP3, kernel version 4.4
Ubuntu 16.04 LTS, kernel version 4.15 Ubuntu 16.04 LTS, kernel version 4.15

Using nested virtualization

To use nested virtualization, complete the following steps:

  1. Check whether the nested virtualization constraint is disabled.

  2. Create an L1 VM that has nested virtualization enabled.

  3. Create a nested L2 VM.

If you run into any issues while creating a VM that has nested virtualization enabled or creating nested VMs, see troubleshooting nested virtualization.