Version 1.8. This version is supported as outlined in the Anthos version support policy, offering the latest patches and updates for security vulnerabilities, exposures, and issues impacting Anthos clusters on VMware (GKE on-prem). Refer to the release notes for more details. This is not the most recent version.

vSphere requirements

Anthos clusters on VMware (GKE on-prem) runs in your data center in a vSphere environment. This topic describes requirements for your vSphere environment, including storage, CPU, RAM, and virtual networks.

vSphere requirements

The vSphere requirements vary according to which version of Anthos clusters on VMware you are using. For more information, see the version compatibility matrix for fully supported versions and earlier versions.

vSphere is VMware's server virtualization software. Anthos clusters on VMware uses VMware's vCenter Server to manage your clusters. To learn about installing vSphere and vCenter Server, see Overview of the vSphere Installation and Setup Process in the VMware documentation.

License edition and version requirements

Anthos clusters on VMware requires VMware vSphere 6.5 Update 3+, 6.7 Update 3+, or 7.0 Update 1+. You need the following VMware licenses:

  • A vSphere Enterprise Plus or vSphere Standard license.

    The Enterprise Plus license is recommended, because it allows you to enable the VMware Distributed Resource Scheduler (DRS).

    Along with this license, you must purchase a support subscription for at least one year.

  • A vCenter Server Standard license.

    Along with this license, you must purchase a support subscription for at least one year.

Hardware requirements

Anthos clusters on VMware runs on a set of physical hosts that run the VMware ESXi hypervisor. To learn about the hardware requirements for ESXi, see ESXi Hardware Requirements.

By default, Anthos clusters on VMware automatically creates VMware Distributed Resource Scheduler (DRS) anti-affinity rules for your admin cluster and user cluster's nodes, causing them to be spread across at least three physical hosts in your datacenter.

This feature requires that your vSphere environment meets the following conditions:

  • VMware DRS is enabled. VMware DRS requires vSphere Enterprise Plus license edition.

  • Your vSphere user account has the Host.Inventory.Modify cluster privilege.

  • There are at least three physical hosts available.

Recall that if you have a vSphere Standard license, you cannot enable VMware DRS.

If you do not have DRS enabled, or if you do not have at least three hosts where vSphere VMs can be scheduled, set antiAffinityGroups.enabled to false in admin cluster and user cluster configurations.

Minimum hardware requirements for demonstration purposes

If you want to create a proof-of-concept demonstration, the requirements are less than for a standard production implementation. Here are the minimum requirements, and a sample configuration, to set up an Anthos 1.7.0 cluster on a single ESXi host.

  • Vcenter Server Version: 6.7U3
  • Sample ESXi Host Configuration:
    • Manufacturer: Dell Inc.
    • Physical CPUs: 8 CPUs @ 2694MHz
    • Processor type: Intel(R) Xeon(R) Platinum 8168 CPU @ 2.70GHz
    • Processor sockets: 2
    • Version: 6.7U3
    • Hyperthreading: enabled
  • Sample Datastore Configuration:
    • Type: VMFS 6.82
    • Drive type: SSD
    • Vendor: DELL
    • Drive Type: logical
    • RAID Level: RAID1
VM Types (Count)FeaturesPhysical CPUPhysical MemoryStorage
Admin Workstation(1)
Admin master(1)
Admin addon Nodes(2)
User Master(1)
User Worker Node(3)
Gkeadm Loadbalancer: F5/Seesaw
GKE Connect
Non-HA
AntiaffinityGroups: Disabled
8 CPUs @ 2.7Ghz with hyperthreading enabled 40GB 450GB
Admin Workstation(1)
Admin master(1)
Admin addon Nodes(2)
User Master(1)
User Worker Node(3)
Gkeadm Loadbalancer: F5/Seesaw
GKE Connect
Non-HA
AntiaffinityGroups: Disabled
Cloud Logging and Cloud Monitoring (Stackdriver)
8 CPUs @ 2.7Ghz with hyperthreading enabled40GB470GB

vSphere storage requirements

Using the vSphere CSI driver requires the VMware vSphere versions, for both vCenter and the ESXi hosts, to be 6.7 U3 or above.

Anthos clusters on VMware does not support Storage vMotion and Storage DRS.

vCenter user account privileges

You can define custom roles in vCenter or use vCenter system roles for the various users in your organization, including your Anthos cluster administrator and the users who develop on those clusters.

The vCenter user account that you use to install Anthos clusters on VMware must have sufficient privileges. For example, a user account that is assigned the vCenter's Administrator role has privileges for complete access to all vCenter objects and provides an Anthos cluster administrator with full access.

For other vCenter user accounts, you create custom roles to assign the necessary privileges to your cluster's users.

  • Use the following table to understand what the minimum required set of privileges are for your Anthos cluster users.
  • A user account with administrator privileges can use the following commands to create a custom vCenter role, define the minimum required privileges to that role, and then assign that custom role to an existing vCenter user account.

To learn how to manage privileges, refer to Managing Permissions for vCenter Components.

vCenter user account privileges with folders

Starting with Anthos clusters on VMware version 1.4, you can place VM images and templates in a separate VM folder, as opposed to the global datacenter folder. This is only supported in the v1 format of the admin cluster configuration.

In your admin cluster configuration file, create a new key called folder. Set the value of folder to the name of the vCenter folder that you want to use for this Anthos clusters on VMware deployment. All user clusters will inherit the folder automatically. Do not specify a folder in your user cluster configuration files.

If the folder key is not specified, or if the value is left blank, the top level Datacenter VM folder is used. Like other vCenter resources, the folder must be created prior to deployment, with the appropriate permissions.

For example, in the admin-cluster.yaml for your deployment:

apiVersion: v1
kind: AdminCluster
#...
vCenter:
  address: mtv-example-vc01.anthos
  datacenter: mtv-example-vc01
  cluster: admin-permissions
  resourcePool: example-cluster-resourcepool
  datastore: example-cluster-datastore
  # insert the following new line with the path to the vcenter folder here.
  folder: my-vm-folder

Setting permissions when using a folder

In Anthos clusters on VMware versions older than 1.4, we required a set of permissions to be applied to the entire vCenter cluster. While this was simpler to configure, these permissions did not constrain the Anthos clusters on VMware vCenter user sufficiently. While the set of permissions remain the same in Anthos clusters on VMware 1.4, we can now apply them to a much smaller set of objects, provided a folder is specified for the deployment, as described above.

The following is a set of roles, their permissions, and the objects the roles must be applied to. Entries marked "(recursively)" must be applied with the Propagate field set to true, so the permissions are inherited by all child objects.

Role: ClusterEditor

Description: Apply DRS rules to Clusters + Read Only Access

Objects: $VCenter.Cluster(recursively)

Privileges: System.Read System.View System.Anonymous Host.Inventory.EditCluster

Role: SessionValidator

Description: Validate an existing VCenter session + Read Only Access

Objects: $VCenter.Root

Privileges: System.Read System.View System.Anonymous Session.ValidateSessions

Role: ReadOnly

Description: Built in role that permits enumerating objects

Objects: $VCenter.Datacenter(recursively), $VCenter "VM Network"

Privileges: System.Read System.View System.Anonymous

Role: Anthos

Description: Set of permissions required to deploy, manage and monitor clusters. This role represents the set of permissions that was applied to the entire VCenter environment in GKE OnPrem 1.3 and earlier.

Objects: $VCenter.Datastore(recursively), $VCenter.ResourcePool(recursively), $VCenter.Folder(recursively), $VCenter.Network(recursively)

Known issue: installer fails when creating vSphere datadisk

(Issue ID 156233307)

The Anthos clusters on VMware installer can fail if custom roles are bound at the wrong permissions level.

When the role binding is incorrect, creating a vSphere datadisk with govc hangs and the disk is created with a size equal to 0. To fix the issue, you should bind the custom role at the vSphere vcenter level (root).

If you want to bind the custom role at the DC level (or lower than root), you also need to bind the read-only role to the user at the root vCenter level.

For more information on role creation, see vCenter user account privileges.

Resource requirements for admin workstation, admin cluster, and user clusters

The physical ESXi hosts in your data center must provide enough storage, CPU, and RAM resources to fulfill the needs of the virtual machines that you will create during your initial installation of Anthos clusters on VMware. Your data center must also provide enough virtual disk space to fulfill PersistentVolumeClaims (PVCs) created by Prometheus and Google Cloud's operations suite.

The initial installation of Anthos clusters on VMware requires these resources:

  • 36 vCPU
  • 98241 MB RAM
  • 2280 GB virtual disk space

For more detailed information on resource requirements, see CPU, RAM, and Storage requirements.