GKE on-prem runs in your data center in a vSphere 6.5 or 6.7 Update 3 environment. This topic describes requirements for your vSphere environment, including storage, CPU, RAM, and virtual networks.
Hardware requirements
GKE on-prem runs on a set of physical hosts that run the VMware ESXi hypervisor. To learn about the hardware requirements for ESXi, see ESXi Hardware Requirements.
vSphere requirements
The vSphere requirements vary according to which version of GKE on-prem you are using. For more information, see the version compatibility matrix for fully supported versions and earlier versions.
vSphere is VMware's server virtualization software. GKE on-prem uses VMware's vCenter Server to manage your clusters. To learn about installing vSphere and vCenter Server, refer to Overview of the vSphere Installation and Setup Process in the VMware documentation.
License edition and version requirements
GKE on-prem requires VMware vSphere 6.5 or 6.7 Update 3. You need the following VMware licenses:
-
A vSphere Enterprise Plus or vSphere Standard license. The Enterprise Plus license is recommended, because it allows you to enable the VMware Distributed Resource Scheduler (DRS). With DRS enabled, VMware automatically distributes your GKE on-prem cluster nodes across physical hosts in your data center.
Starting with version 1.1.0-gke.6, DRS is enabled by default. So to use a Standard license, you must explicitly disable DRS.
Along with this license, you must purchase a support subscription for at least one year.
-
A vCenter Server Standard license.
Along with this license, you must purchase a support subscription for at least one year. See VMware vCenter Server Editions on the VMware pricing page.
vCenter user account privileges
You can define custom roles in vCenter or use a default vCenter role for the various user roles in your organization, including your GKE on-prem cluster administrator and the users who develop on those clusters.
The vCenter user account that you use to install GKE on-prem must have sufficient privileges. For example, a user account that is assigned the vCenter's Administrator role has privileges for complete access to all vCenter objects and provides a GKE on-prem cluster administrator with full access.
For other vCenter user accounts, you create custom roles to assign the necessary privileges to your cluster's users.
- Use the following table to understand what the minimum required set of privileges are for your GKE on-prem cluster users.
- A user account with administrator privileges can use the following commands to create a custom vCenter role, define the minimum required privileges to that role, and then assign that custom role to an existing vCenter user account.
To learn how to manage privileges, refer to Managing Permissions for vCenter Components.
vCenter user account privileges with folders
Starting with GKE on-prem version 1.4, you can place VM images and templates in a separate VM folder, as opposed to the global datacenter folder. This is only supported in the v1 format of the admin cluster configuration.
In your
admin cluster configuration file,
create a new key called folder. Set the value of folder to the name of the
vCenter folder that you want to use for this GKE on-prem deployment.
All user clusters will inherit the folder automatically. Do not specify a folder
in your
user cluster configuration files.
If the folder key is not specified, or if the value is left blank, the top
level Datacenter VM folder is used. Like other vCenter resources, the folder
must be created prior to deployment, with the appropriate permissions.
For example, in the admin-cluster.yaml for your deployment:
apiVersion: v1 kind: AdminCluster #... vCenter: address: mtv-example-vc01.anthos datacenter: mtv-example-vc01 cluster: admin-permissions resourcePool: example-cluster-resourcepool datastore: example-cluster-datastore # insert the following new line with the path to the vcenter folder here. folder: my-vm-folder
Setting permissions when using a folder
In GKE on-prem versions older than 1.4, we required a set of permissions to be applied to the entire vCenter cluster. While this was simpler to configure, these permissions did not constrain the GKE on-prem vCenter user sufficiently. While the set of permissions remain the same in GKE on-prem 1.4, we can now apply them to a much smaller set of objects, provided a folder is specified for the deployment, as described above.
The following is a set of roles, their permissions, and the objects the roles must be applied to. Entries marked "(recursively)" must be applied with the Propagate field set to true, so the permissions are inherited by all child objects.
Known issue: installer fails when creating vSphere datadisk
(Issue ID 156233307)
The GKE on-prem installer can fail if custom roles are bound at the wrong permissions level.
When the role binding is incorrect, creating a vSphere datadisk with
govc hangs and the disk is created with a size equal to 0.
To fix the issue, you should bind the custom role at the
vSphere vcenter level (root).
If you want to bind the custom role at the DC level (or lower than root), you also need to to bind the read-only role to the user at the root vCenter level.
For more information on role creation, see vCenter user account privileges.
Resource requirements for admin workstation, admin cluster, and user clusters
The physical ESXi hosts in your data center must provide enough storage, CPU, and RAM resources to fulfill the needs of the virtual machines that you will create during your initial installation of GKE on-prem. Your data center must also provide enough virtual disk space to fulfill PersistentVolumeClaims (PVCs) created by Prometheus and Google Cloud Observability.
The initial installation of GKE on-prem requires these resources:
- 36 vCPU
- 98241 MB RAM
- 2280 GB virtual disk space
For more detailed information on resource requirements, see CPU, RAM, and Storage requirements.