vSphere requirements

GKE on-prem runs in your data center in a vSphere 6.5 or 6.7 Update 3 environment. This topic describes requirements for your vSphere environment, including storage, CPU, RAM, and virtual networks.

Hardware requirements

GKE on-prem runs on a set of physical hosts that run the VMware ESXi hypervisor. To learn about the hardware requirements for ESXi, see ESXi Hardware Requirements.

vSphere requirements

The vSphere requirements vary according to which version of GKE on-prem you are using. For more information, see the version compatibility matrix for fully supported versions and earlier versions.

vSphere is VMware's server virtualization software. GKE on-prem uses VMware's vCenter Server to manage your clusters. To learn about installing vSphere and vCenter Server, refer to Overview of the vSphere Installation and Setup Process in the VMware documentation.

License edition and version requirements

GKE on-prem requires VMware vSphere 6.5 or 6.7 Update 3. You need the following VMware licenses:

  • A vSphere Enterprise Plus or vSphere Standard license. The Enterprise Plus license is recommended, because it allows you to enable the VMware Distributed Resource Scheduler (DRS). With DRS enabled, VMware automatically distributes your GKE on-prem cluster nodes across physical hosts in your data center.

    Starting with version 1.1.0-gke.6, DRS is enabled by default. So to use a Standard license, you must explicitly disable DRS.

    Along with this license, you must purchase a support subscription for at least one year.

  • A vCenter Server Standard license.

    Along with this license, you must purchase a support subscription for at least one year. See VMware vCenter Server Editions on the VMware pricing page.

vCenter user account privileges

You can define custom roles in vCenter or use a default vCenter role for the various user roles in your organization, including your GKE on-prem cluster administrator and the users who develop on those clusters.

The vCenter user account that you use to install GKE on-prem must have sufficient privileges. For example, a user account that is assigned the vCenter's Administrator role has privileges for complete access to all vCenter objects and provides a GKE on-prem cluster administrator with full access.

For other vCenter user accounts, you create custom roles to assign the necessary privileges to your cluster's users.

  • Use the following table to understand what the minimum required set of privileges are for your GKE on-prem cluster users.
    Entity Privilege
    Datastore
    • Allocate space
    • Browse datastore
    • Low level file operations
    • Remove file
    • Update virtual machine files
    • Update virtual machine metadata
    Folder
    • Create folder
    • Delete folder
    • Move folder
    • Rename folder
    Host Inventory
    • Edit cluster
    vSphere Tagging
    • Create vSphere Tag
    Root vCenter Server
    • Validate session
    Network
    • Assign network
    Resource
    • Apply recommendation
    • Assign virtual machine to resource pool
    • Migrate powered off virtual machine
    • Migrate powered on virtual machine
    • Query vMotion
    Storage views
    • View
    System
    • Anonymous
    • Read
    • View
    Tasks
    • Create task
    • Update task
    vApp
    • Import
    • vApp application configuration
    • vApp instance configuration
    Virtual machines
    • Configuration
      • Add existing disk
      • Add new disk
      • Add or remove device
      • Advanced
      • Change CPU count
      • Change resource
      • Configure managedBy
      • Disk change tracking
      • Disk lease
      • Display connection settings
      • Extend virtual disk
      • Host USB device
      • Memory
      • Modify device settings
      • Query Fault Tolerance compatibility
      • Query unowned files
      • Raw device
      • Reload from path
      • Remove disk
      • Rename
      • Reset guest information
      • Set annotation
      • Settings
      • Swapfile placement
      • Toggle fork parent
      • Upgrade virtual machine compatibility
    • Guest operations
      • Guest operation alias modification
      • Guest operation alias query
      • Guest operation modifications
      • Guest operation program execution
      • Guest operation queries
    • Interaction
      • Answer question
      • Backup operation on virtual machine
      • Configure CD media
      • Configure floppy media
      • Console interaction
      • Create screenshot
      • Defragment all disks
      • Device connection
      • Drag and drop
      • Guest operating system management by VIX API
      • Inject USB HID scan codes
      • Pause or Unpause
      • Perform wipe or shrink operations
      • Power off
      • Power on
      • Record session on virtual machine
      • Replay session on virtual machine
      • Reset
      • Resume Fault Tolerance
      • Suspend
      • Suspend Fault Tolerance
      • Test failover
      • Test restart Secondary VM
      • Turn off Fault Tolerance
      • Turn on Fault Tolerance
      • VMware Tools install
    • Inventory
      • Create from existing
      • Create new
      • Move
      • Register
      • Remove
      • Unregister
    • Provisioning
      • Allow disk access
      • Allow file access
      • Allow read-only disk access
      • Allow virtual machine download
      • Allow virtual machine files upload
      • Clone template
      • Clone virtual machine
      • Create template from virtual machine
      • Customize
      • Deploy template
      • Mark as template
      • Mark as virtual machine
      • Modify customization specification
      • Promote disks
      • Read customization specifications
    • Service configuration
      • Allow notifications
      • Allow polling of global event notifications
      • Manage service configurations
      • Modify service configuration
      • Query service configurations
      • Read service configuration
    • Snapshot management
      • Create snapshot
      • Remove snapshot
      • Rename snapshot
      • Revert to snapshot
    • vSphere Replication
      • Configure replication
      • Manage replication
      • Monitor replication
  • A user account with administrator privileges can use the following commands to create a custom vCenter role, define the minimum required privileges to that role, and then assign that custom role to an existing vCenter user account.
    export GOVC_USERNAME=ADMINISTRATOR_ACCOUNT@vsphere.local
    export GOVC_PASSWORD=ADMINISTRATOR_PASSWORD
    cat <<END |xargs govc role.create anthos
      Datastore.AllocateSpace Datastore.Browse Datastore.Config Datastore.DeleteFile
      Datastore.FileManagement Datastore.UpdateVirtualMachineFiles
      Datastore.UpdateVirtualMachineMetadata Folder.Create Folder.Delete Folder.Move
      Folder.Rename Host.Inventory.EditCluster InventoryService.Tagging.CreateTag
      Network.Assign Resource.ApplyRecommendation Resource.AssignVMToPool
      Resource.ColdMigrate Resource.HotMigrate Resource.QueryVMotion
      Sessions.ValidateSession StorageViews.View System.Anonymous System.Read
      System.View Task.Create Task.Update VApp.ApplicationConfig VApp.Import
      VApp.InstanceConfig VirtualMachine.Config.AddExistingDisk
      VirtualMachine.Config.AddNewDisk VirtualMachine.Config.AddRemoveDevice
      VirtualMachine.Config.AdvancedConfig VirtualMachine.Config.Annotation
      VirtualMachine.Config.CPUCount VirtualMachine.Config.ChangeTracking
      VirtualMachine.Config.DiskExtend VirtualMachine.Config.DiskLease
      VirtualMachine.Config.EditDevice VirtualMachine.Config.HostUSBDevice
      VirtualMachine.Config.ManagedBy VirtualMachine.Config.Memory
      VirtualMachine.Config.MksControl VirtualMachine.Config.QueryFTCompatibility
      VirtualMachine.Config.QueryUnownedFiles VirtualMachine.Config.RawDevice
      VirtualMachine.Config.ReloadFromPath VirtualMachine.Config.RemoveDisk
      VirtualMachine.Config.Rename VirtualMachine.Config.ResetGuestInfo
      VirtualMachine.Config.Resource VirtualMachine.Config.Settings
      VirtualMachine.Config.SwapPlacement VirtualMachine.Config.ToggleForkParent
      VirtualMachine.Config.UpgradeVirtualHardware
      VirtualMachine.GuestOperations.Execute VirtualMachine.GuestOperations.Modify
      VirtualMachine.GuestOperations.ModifyAliases
      VirtualMachine.GuestOperations.Query
      VirtualMachine.GuestOperations.QueryAliases
      VirtualMachine.Hbr.ConfigureReplication VirtualMachine.Hbr.MonitorReplication
      VirtualMachine.Hbr.ReplicaManagement VirtualMachine.Interact.AnswerQuestion
      VirtualMachine.Interact.Backup VirtualMachine.Interact.ConsoleInteract
      VirtualMachine.Interact.CreateScreenshot
      VirtualMachine.Interact.CreateSecondary
      VirtualMachine.Interact.DefragmentAllDisks
      VirtualMachine.Interact.DeviceConnection
      VirtualMachine.Interact.DisableSecondary VirtualMachine.Interact.DnD
      VirtualMachine.Interact.EnableSecondary VirtualMachine.Interact.GuestControl
      VirtualMachine.Interact.MakePrimary VirtualMachine.Interact.Pause
      VirtualMachine.Interact.PowerOff
      VirtualMachine.Interact.PowerOn VirtualMachine.Interact.PutUsbScanCodes
      VirtualMachine.Interact.Record VirtualMachine.Interact.Replay
      VirtualMachine.Interact.Reset
      VirtualMachine.Interact.SESparseMaintenance VirtualMachine.Interact.SetCDMedia
      VirtualMachine.Interact.SetFloppyMedia VirtualMachine.Interact.Suspend
      VirtualMachine.Interact.TerminateFaultTolerantVM
      VirtualMachine.Interact.ToolsInstall
      VirtualMachine.Interact.TurnOffFaultTolerance VirtualMachine.Inventory.Create
      VirtualMachine.Inventory.CreateFromExisting VirtualMachine.Inventory.Delete
      VirtualMachine.Inventory.Move VirtualMachine.Inventory.Register
      VirtualMachine.Inventory.Unregister VirtualMachine.Namespace.Event
      VirtualMachine.Namespace.EventNotify VirtualMachine.Namespace.Management
      VirtualMachine.Namespace.ModifyContent VirtualMachine.Namespace.Query
      VirtualMachine.Namespace.ReadContent VirtualMachine.Provisioning.Clone
      VirtualMachine.Provisioning.CloneTemplate
      VirtualMachine.Provisioning.CreateTemplateFromVM
      VirtualMachine.Provisioning.Customize
      VirtualMachine.Provisioning.DeployTemplate
      VirtualMachine.Provisioning.DiskRandomAccess
      VirtualMachine.Provisioning.DiskRandomRead
      VirtualMachine.Provisioning.FileRandomAccess
      VirtualMachine.Provisioning.GetVmFiles
      VirtualMachine.Provisioning.MarkAsTemplate
      VirtualMachine.Provisioning.MarkAsVM
      VirtualMachine.Provisioning.ModifyCustSpecs
      VirtualMachine.Provisioning.PromoteDisks
      VirtualMachine.Provisioning.PutVmFiles
      VirtualMachine.Provisioning.ReadCustSpecs
      VirtualMachine.State.CreateSnapshot VirtualMachine.State.RemoveSnapshot
      VirtualMachine.State.RenameSnapshot VirtualMachine.State.RevertToSnapshot
    END
    govc permissions.set -principal CLUSTER_USER_ACCOUNT@vsphere.local \
     -role anthos -propagate=true
    

To learn how to manage privileges, refer to Managing Permissions for vCenter Components.

Resource requirements for admin workstation, admin cluster, and user clusters

The physical ESXi hosts in your data center must provide enough storage, CPU, and RAM resources to fulfill the needs of the virtual machines that you will create during your initial installation of GKE on-prem. Your data center must also provide enough virtual disk space to fulfill PersistentVolumeClaims (PVCs) created by Prometheus and Google Cloud Observability.

The initial installation of GKE on-prem requires these resources:

  • 36 vCPU
  • 98241 MB RAM
  • 2280 GB virtual disk space

For more detailed information on resource requirements, see CPU, RAM, and Storage requirements.