Version 1.9. This version is supported as outlined in the Anthos version support policy, offering the latest patches and updates for security vulnerabilities, exposures, and issues impacting Anthos clusters on VMware (GKE on-prem). Refer to the release notes for more details. This is the most recent version.

User guide for Windows Server OS node pools

With Anthos clusters on VMware (GKE on-prem), you can create a node pool of Windows Server OS nodes. The user cluster that runs the Windows Server OS node pools can also run node pools that contain nodes using Ubuntu or Container-Optimized OS.

Requirements for a Windows Server OS node pool

The nodes in a node pool must all use the same operating system, indicated by the osImageType parameter.

Before you create, in your user cluster, a node pool that has Windows Server OS nodes, make sure you meet these requirements:

  • An admin cluster must already be in place before you can create a Windows node pool, because a Windows node pool is only supported in the user cluster.
  • The user cluster must run at least one Linux node pool, because the Linux node pool is required to create a Windows node pool.
  • A user cluster with Windows node pools must have the enabledataplanev2 field set to true in the user cluster configuration file. This enables Dataplane V2 on the Linux nodes in that cluster.
  • If you want Windows Dataplane v2 to be enabled for the Windows node pools in the user cluster, the user cluster configuration file must have the enableWindowsDataplanev2 field set to true in the user cluster configuration file.
  • You have downloaded a Windows Server 2019 ISO from Microsoft to create a VM template specific for Windows node pools.
  • Your vSphere environment must be vSphere 6.7, Update 3 or later.

Create a Windows node pool in a user cluster

Step 1: Create the Windows VM Template for Anthos clusters on VMware

Before you begin, make sure that you have already created an admin cluster.

  1. Create a base Windows VM template from the Windows Server 2019 ISO.

    • The initial network adapter type for the Windows VM to install Windows Server 2019 ISO should be E1000E.
    • Follow these steps: Create a VMware vSphere template for Windows Server 2019.
    • Take note of the initial password that is set when you run the Windows ISO installer, to use it in the future.
    • Make sure you are using the latest qualified patch version for Windows Server 2019, 10.0.17763.2114. See Security patch process.
    • You can't attach any device that uses the IDE controller to the base VM template.
  2. Install VMware Tools on the base Windows VM template if not already installed, using the VMWare instructions.

  3. Create a Windows VM template:

    gkectl prepare windows --base-vm-template BASE_WINDOWS_VM_TEMPLATE_NAME --bundle-path BUNDLE_PATH --kubeconfig ADMIN_CLUSTER_KUBECONFIG_PATH

  4. In the last line of the command output, you can find the name of the generated Windows VM template. Make a note of the name for future use. The name has the following format:

Successfully created Anthos Windows VM template "gke-on-prem-windows-server-2019-VERSION"

Step 2: Upload Windows container images to a private registry

Omit this step if you are not using a private registry.

You can automate the upload of Windows container images to a private registry using containerd on a Linux admin workstation. But containerd cannot push the Windows container image base layer, which means the base layers must be pulled from the Microsoft registry when pulling the image. To push the base layers, follow the Option 2 steps.

Option 1: If you don't need to manually push the Windows base layer images to the private registry:

gkectl prepare --config <var class="edit">ADMIN_CLUSTER_CONFIG</var> --upload-windows-images

Replace ADMIN_CLUSTER_CONFIG with the path to the admin cluster configuration file.

The flag --upload-windows-images specifies that Windows Container images will be pushed. Only Linux container images will be pushed to the private registry without specifying this flag.

Option 2: If you need to manually push the Windows base layer images to the private registry:

  • Use a Windows machine with Docker installed, and with access to, before attempting these steps. You can only pull Windows container images to a Windows machine.
  • Run docker login to authenticate to your private registry.
  • Upload the Windows Container images together with their base layers to your private registry, following these steps:

    • Go to the Docker daemon.json file on your Windows machine:

      PS C:> cat C:\ProgramData\docker\config\daemon.json

    • Add the following lines to configure your Docker daemon.json file to allow pushing foreign layers to your private registry:

      "allow-nondistributable-artifacts": ["PRIVATE_REGISTRY_NAME"]
    • Download the required Windows Container images to your local Windows machine, and then tag and push them to your private registry. The changes you made to the daemon.json Docker configuration file mean that the base layer can be pushed to the private registry. To complete these tasks, run the following commands:
# Pull the Windows container images
docker pull
docker pull
docker pull

# Tag the images to use private registry
docker tag $PRIVATE_REGISTRY_URL/pause-win:gke_windows_pause_20210302_RC00_2019
docker tag $PRIVATE_REGISTRY_URL/fluent-bit-win:v1.8.3-gke.1_ltsc2019
docker tag $PRIVATE_REGISTRY_URL/gke-metrics-agent-windows:0.3.10-gke.0_2019

# Push to private registry
docker push PRIVATE_REGISTRY_URL/pause-win:gke_windows_pause_20210302_RC00_2019
docker push PRIVATE_REGISTRY_URL/fluent-bit-win:v1.8.3-gke.1_ltsc2019
docker push PRIVATE_REGISTRY_URL/gke-metrics-agent-windows:0.3.10-gke.0_2019

Step 3: (Required if using proxy) Allowlisting URLs for creating Windows node pools

If your cluster is behind a proxy server, add these URLs to your proxy server allowlist in addition to the other addresses that Anthos clusters on VMware requires.

# Microsoft registry URLs, needed by every Windows node if using GCR

# Microsoft WSUS server URLs, needed by `gkectl prepare windows` on the Windows VM

# Cloudbase-Init URL, needed by `gkectl prepare windows` on the Windows VM

# Powershell Gallery URLs, needed by `gkectl prepare windows` on the Windows VM

# Windows Update Service, needed by `gkectl prepare windows` on the Windows VM

# Access for Installing docker, needed by `gkectl prepare windows` on the Windows VM

Step 4: Add a Windows node pool to the user cluster configuration file

  1. Dataplane V2 must be enabled in your user cluster to use Windows node pools. Add the following line to your user cluster configuration file to enable Dataplane V2:

    enableDataplaneV2: true
  2. Add a Windows node pool to your nodePools section in the user cluster configuration file. At least one Linux node pool is required in addition to your Windows node pools. Set the osImage and osImageType fields to create Windows node pools:

  • osImage: Replace WINDOWS_VM_TEMPLATE_NAME with the name of your prepared Windows VM template in step 1, which should be in the same vCenter datastore specified in the user cluster configuration file.
  • osImageType: Specify the OS image type to be windows.
# user-cluster.yaml

- name: windows-nodepool-1
  cpus: 4
  memoryMB: 8192
  replicas: 3
  bootDiskSizeGB: 100
  osImageType: windows

Step 5: Create Windows node pools

Before you create Windows node pools, run a list of preflight validators for Windows. Skip this step if you already have a user cluster. - (Optional) Run either or both the fast and slow preflight checks, which create a test VM for Windows and validate the Windows VM template:

gkectl check-config --config USER_CLUSTER_CONFIG --kubeconfig ADMIN_CLUSTER_KUBECONFIG
  • This command is intended for you to run before creating a user cluster. If you already have a user cluster, certain checks might fail. For example, the IP addresses in the hostconfig.yaml file might already be in use by existing nodes in your user cluster.
  • Although not recommended, you can skip the Windows preflight checks with the --skip-validation-windows flag.
  • Managing Windows node pools is the same as it is for Linux node pools. See Managing node pools. The commands for creating, updating, and upgrading clusters and node pools also remain the same and are listed here.
# Create a new cluster
gkectl create cluster --kubeconfig ADMIN_CLUSTER_KUBECONFIG --config USER_CLUSTER_CONFIG

# Update an existing cluster with the new Windows node pool
gkectl update cluster --kubeconfig ADMIN_CLUSTER_KUBECONFIG --config USER_CLUSTER_CONFIG

# Upgrade an existing cluster with the new Windows node pool
gkectl upgrade cluster --kubeconfig ADMIN_CLUSTER_KUBECONFIG --config USER_CLUSTER_CONFIG

Step 6: Validate Windows nodes are running

  1. Check that your Windows nodes have been created and are Ready.

    kubectl --kubeconfig USER_KUBECONFIG get nodes 
  2. Diagnose the user cluster to check whether it's healthy.

    gkectl diagnose cluster --kubeconfig ADMIN_CLUSTER_KUBECONFIG  --cluster-name CLUSTER_NAME

Deploy a Windows Pod

Windows Server nodes are tainted with this key-value pair:

This taint ensures that the GKE scheduler does not attempt to run Linux containers on Windows Server nodes. To schedule Windows Server containers on Windows Server nodes, your manifest file must include this nodeSelector section:

nodeSelector: windows

With nodeSelector configured, an admission webhook running in the cluster checks new workloads for the presence of this Windows node selector and when found, applies the following toleration to the workload which allows it to run on the tainted Windows Server nodes:

- key: ""
  operator: "Equal"
  value: "windows"
  effect: "NoSchedule"

Step 1: Create an Internet Information Services (IIS) deployment file

Here is a sample configuration, which deploys Microsoft's official IIS image to a single Pod.

Create an IIS file named iis.yaml with the following content:

apiVersion: apps/v1
kind: Deployment
  name: iis
    app: iis
  replicas: 1
      app: iis
        app: iis
      nodeSelector: windows
      - name: iis-server
        - containerPort: 80
apiVersion: v1
kind: Service
    app: iis
  name: iis
  - port: 80
    protocol: TCP
    targetPort: 80
    app: iis
  sessionAffinity: None
  type: LoadBalancer
  loadBalancerIP: [Fill in with an available IP address]

Step 2: Create the deployment and expose it via a service

# Create the deployment
kubectl --kubeconfig USER_CLUSTER_KUBECONFIG create -f iis.yaml

Step 3: Validate the Pod

Check the status of the Pod using kubectl.

kubectl --kubeconfig USER_CLUSTER_KUBECONFIG get pods

Wait until the returned output shows that the Pod has a status of "Running".

NAME                   READY     STATUS    RESTARTS   AGE
iis-5c997657fb-w95dl   1/1       Running   0          28s

Get the status of the service, and wait until the external IP field is populated.

kubectl --kubeconfig USER_CLUSTER_KUBECONFIG  get service iis

Expected output:

iis    LoadBalancer   35.x.x.x     80:32233/TCP   17s

You can use your browser to open http://EXTERNAL_IP to see the IIS web page!

Upgrade user cluster with Windows node pools

The upgrade process for a user cluster with Windows node pools is similar to that for upgrading Linux-only user clusters, except that you must create a Windows VM template from a base VM template before upgrading.

You can update the patch build version of the base VM template during upgrade by downloading a newer patch version of Windows Server 2019 from Microsoft as a security patch. See Security patch process.

gkectl prepare windows --base-vm-template $BASE_WINDOWS_VM_TEMPLATE_NAME --bundle-path BUNDLE_PATH --kubeconfig ADMIN_CLUSTER_KUBECONFIG

Run the command below for upgrading the user cluster:

gkectl upgrade cluster --kubeconfig ADMIN_CLUSTER_KUBECONFIG --config USER_CLUSTER_CONFIG

Replace the following:

  • ADMIN_CLUSTER_KUBECONFIG with the path of your admin kubeconfig file
  • ADMIN_CLUSTER_CONFIG with the path of your admin cluster configuration file

Accessing Windows nodes

The standard way to access Windows nodes is with a username and password, which differs from Linux nodes, which are typically accessed via ssh key pairs for authentication.

For Windows nodes on vSphere, the username is Administrator. The password is generated by the clusterapi-controller and stored in the windows-node-password secret in the admin cluster's user namespace. The command for getting the password from that secret is:

kubectl get secret windows-node-password -n [USER_CLUSTER_NAME] --kubeconfig admin-kubeconfig.yaml -o jsonpath={.data.*} | base64 -d

You can also use get the password using the vCenter user interface. Navigate to the VM that you want to log into, and then you can find the password at the password vApp property of that VM.

Once you have the username and password, you can access your Windows VM using any of the following approaches:

Using Remote Desktop Protocol

Because the RDP has been enabled during the template build, you can access your Windows VM by using a RDP client.

Using SSH

To ssh into a Windows VM:

ssh Administrator@[VM_IP_ADDRESS]

Follow the prompt to type in the password to connect to your VM.

Transferring files from and to your Windows VM

You can transfer files from and to your Windows VM via SCP:

Upload files to Windows VM:


Download files from Windows VM:


Type the password when prompted.

Alternatively, you can also transfer files by using Cloud Storage or using RDP, as described in  Transferring files to Windows VMs.


Cannot SSH/RDP to the Windows VM

Check if the VM has a network connection by running Test-NetConnection on your vCenter web console.

The result should contain PingSucceeded: true if there is a network connection. If the VM does not have a network connection, check the network adapter used for this VM. Make sure that the network allows inbound connections to the VM from your workstation where you want to run the SSH/RDP.

Verify that kubelet, kube-proxy, and flanneld are running on the Windows VM.

Connect to your VM following the steps here and run the following commands:

# Check that kubelet and kube-proxy services have status 'Running'
Get-Service kubelet
Get-Service kube-proxy
# Check that the flanneld process exists
Get-Process flanneld

Using the snapshot tool

Use the snapshot tool to grab the snapshot tarball. This tarball contains the log files on nodes as well as outputs for troubleshooting commands running on the node.

gkectl diagnose snapshot --scenario system-with-logs --cluster-name [USER_CLUSTER_NAME] --kubeconfig [PATH_TO_KUBECONFIG]

Windows VM creation fails

Check the logs from the vsphere-controller-manager container in the clusterapi-controllers Pod in the user namespace of the admin cluster.

kubectl --kubeconfig ADMIN_CLUSTER_KUBECONFIG -n USER_CLUSTER_NAME logs clusterapi-controllers-POD_NAME_SUFFIX vsphere-controller-manager

Ensure that your VM template is located in the same datacenter and datastore as specified in your user cluster configuration file.

Windows VM is created, but the node fails to start properly or show up

  • Check the startup logs on the node located at C:\var\log\startup.log to see if anything failed to start.

    • If flanneld is not running, try to rerun the startup script located at C:\etc\startup\startup-script.ps1
    • If kubelet is not running, check the kubelet logs under `C:\var\log`.
    • If kube-proxy is not running, check the kube-proxy logs under C:\var\log.

Logging and Monitoring

Anthos clusters on VMware supports logging and monitoring for Windows nodes and Pods, as it does for Linux nodes and Pods.

When logging and monitoring are configured, agents are deployed on Windows nodes. These agents collect, process, and export the node's logs and metrics.

Windows logging Agent

The Windows logging agent collects the following logs:

  • Pod resource type: system and user application workloads.

    Note that Windows user application workloads logs are collected by default and currently cannot be enabled or disabled.

  • Node resource type: kubelet, kube-proxy and Windows event-logs

You can access logs using the Logs Explorer in the console. See Access logs for more information.

Windows monitoring Agent

The Windows monitoring agent collect a different set of CPU and memory usage metrics than the Linux monitoring agent. To monitor the Windows nodes and Pods status, use the prepared dashboards. From the console, select Monitoring > Dashboards, and then select "GKE on-prem Windows node status" and "GKE on-prem Windows pod status" from the All Dashboards list.

These dashboards are automatically created during admin cluster installation if Cloud Monitoring is enabled. If you already have an admin cluster running, follow these instructions to create these dashboards, using the following json files:

See the full list of metrics collected by the Windows agents.

Node Problem Detector on Windows nodes

The Node Problem Detector daemon is available on Windows nodes. If you have upgraded to version 1.9, Node Problem Detector is enabled automatically. Node Problem Detector helps with quick detection of some common node problems. Node Problem Detector keeps checking for possible problems and reports them as events and conditions on the node. When a node misbehaves, you can use the kubectl command to find corresponding events and conditions.

The following monitor configurations are enabled for Node Problem Detector:

To get events and conditions on a node:

kubectl --kubeconfig KUBECONFIG describe nodes NODE_NAME


  • KUBECONFIG with the path of the kubeconfig file for the cluster that contains the node.
  • NODE_NAME with the name of the node.

To identify the events generated by the Node Problem Detector monitors, look for the monitor name in the reason field of a rule specified in the rules section.

Node Problem Detector monitors also generate the following conditions on the node. Each of these is set to true if Node Problem Detector detects the corresponding failure scenario on the node.

  • KubeletUnhealthy
  • KubeProxyUnhealthy
  • ContainerRuntimeUnhealthy

Whenever one of the conditions is set to true, the node's Ready condition will become false, which prevents new Pods from being scheduled on the node.

When an unhealthy condition is found, Node Problem Detector attempts to auto-repair the node by restarting the relevant system service.

Node Problem Detector logs are located in the node's C:\var\log\node-problem-detector folder. If logging and monitoring are enabled, the log is exported to Cloud Logging and you can view them in the Logs Explorer.

Use this filter to get Node Problem Detector logs in Logs Explorer:


Replace PROJECT_NAME with the project name.

Security patch process

Besides the regular patch releases for the supported Anthos versions, the Anthos team also continuously qualifies newer Windows patch updates during non-release time periods, and publishes the results for your reference. If an urgent security patch update is needed between Anthos patch releases, you can build a new VM template using the latest version, and then perform a rolling update for the existing Windows node pools to use the new template.

The security patch process includes these steps:

  • Microsoft releases a new security patch for Windows Server 2019.
  • Anthos qualifies the latest security patch version and announces the qualification result.
  • If qualified, users will:
    • Download the latest patch version from Microsoft
    • Build a new Windows VM template using this patch version following the steps here.
    • Update the Windows node pools to use the new template by running:
gkectl update cluster --kubeconfig ADMIN_CLUSTER_KUBECONFIG --config USER_CLUSTER_CONFIG
  • If the new version requires changes from Anthos's side, you must wait for the next monthly Anthos patch release and upgrade the clusters.

  • If the new Windows version is not compatible with Anthos at all, the Anthos team will skip that version and wait for the next security update from Microsoft.

Active Directory domain join

Active Directory domain join requires the length of the VM hostname to be <= 15 characters. For IPAM mode, because the VM hostname is set in the user cluster configuration file, you must ensure the length is <= 15 characters. These instructions are based on the instructions for creating Windows node pools, with the additional step of supplying a customized script during the Windows VM template build.

Verify the Active Domain DNS server is reachable

Active Directory Domain Services (AD DS) uses Domain Name System (DNS) name resolution services to make it possible for clients to locate domain controllers and for the domain controllers that host the directory service to communicate with each other.

The DNS server was created when the AD DS role installed the root forest. For any Windows VM to join the AD domain, it must be able to reach the DNS server. Configure the DNS and firewall configurations following the guidance of the DNS service provider that you are using. You can verify whether the Windows VMs in the current network can contact the AD domain DNS server by running this command:

Server:  example-1-2-3-4.anthos

Step 1: Create Windows VM template with a customized script

  1. Execute a customized script before the Windows node joins the user cluster for Active Directory domain join. Store this script to a local path on your admin workstation. Note that:

    • You can replace the script with your own script for doing the Active Directory domain join.
    • It is recommended that you use a user account with the minimum permissions required for an Active Directory domain join, instead of using an Administrator user.
    • (Optional) To avoid storing the password as cleartext in this script, place the password in a file on the VM template, let the script read from that password file, then delete the file after the domain join.
    $domain = "[DOMAIN_NAME]"
    $password = "[PASSWORD]" | ConvertTo-SecureString -asPlainText -Force
    $username = "$domain\[USERNAME]"
    $credential = New-Object System.Management.Automation.PSCredential($username,$password)
    Add-Computer -DomainName $domain -Credential $credential -restart –force
  2. Create a Windows VM template with a customized script:

    gkectl prepare windows --base-vm-template BASE_WINDOWS_VM_TEMPLATE_NAME --bundle-path BUNDLE_PATH --kubeconfig ADMIN_CLUSTER_KUBECONFIG --customized-script CUSTOMIZED_SCRIPT_PATH

Replace BUNDLE_PATH with the path to the bundle.

Step 2: Create a Windows node pool

Proceed with the standard instructions in Steps 2-6 to create a Windows node pool using the customized Windows VM template.

Step 3: Verify the Active Domain joining for the Windows nodes

On the AD domain controller VM, run the following command:

PS C:\> Get-ADComputer -Filter 'Name -like "user-host-prefix*"'

DistinguishedName : CN=AD-VM-1,CN=Computers,DC=example,DC=org
DNSHostName       :
Enabled           : True
Name              : AD-VM-1
ObjectClass       : computer
ObjectGUID        : b3609717-d24b-4df6-bccb-26ca8e8b9eb0
SamAccountName    : AD-VM-1$
SID               : S-1-5-21-3236879623-1561052741-2808297733-1103

Step 4: Configure Group Managed Service Accounts (optional)

Follow these instructions: Configure GMSA for Windows Pods and containers. You can configure GMSA for Windows pods and containers after the nodes are domain joined.


Logs for the customized script execution of cloudbase-init are located at C:\Program Files\Cloudbase Solutions\Cloudbase-Init\log\cloudbase-init.log. Look for LocalScriptPlugin in the log file, and check the related logs. - Build a new Windows VM template. - Update the Windows node pools to use the new template by running:

gkectl update cluster --kubeconfig ADMIN_CLUSTER_KUBECONFIG --config USER_CLUSTER_CONFIG

Considerations for Windows containers

Some notable differences between Windows and Linux containers are:

  • Version compatibility of Windows container images and the host/node OS images.
    • The Windows Server OS version tuple has four parts: major, minor, build, and revision.
    • The Windows server container base image must match the first three parts of the version tuple of the host OS image. The revision need not match, although it is recommended that you update both the host and container base images.
    • Users need to rebuild their container images whenever the OS image version changes
  • Privileged containers and host namespaces are not supported.
    • Users cannot configure/change nodes by deploying containers, such as Daemonsets.

Limitations for Anthos clusters on VMware on vSphere Windows

  • User clusters must contain at least one Linux node pool.

    • You cannot create a cluster with only a Windows node pool
    • The Linux node pools are required to run critical add-ons.
  • Because 1.5 times more resources are reserved for Windows nodes than Linux nodes, the allocatable resources for Windows are lower.

  • The use of Windows nodes might require a larger minimum machine size than the Anthos clusters on VMware Linux minimum machine size. Windows nodes typically require more resources because of the higher overhead of running node components/services.

Known Issues

This section lists known issues with Windows nodes used with Anthos clusters on VMware, along with workarounds to avoid or recover from these issues.

Windows Pods cannot talk to external IP addresses

This issue is described in Microsoft documentation, which states "You need to exclude the external IP you are trying to query from the ExceptionList."

Contact Google Cloud Support to proceed with a workaround solution.

Windows containers are not cleaned up after removing Windows Pods

This is a known issue, in which docker RemoveContainer also tries to call CreateFile on Windows. As a workaround, log in to the Windows node that has the issue, run Restart-Service docker and the issue should be mitigated. From Anthos clusters on VMware 1.9, the fluent-bit-win container image version and docker version have been updated to pick up the upstream fixes for this issue, this shouldn't reproduce any more. If you encounter this issue, contact Google Cloud Support.

Windows nodes having IP address conflicta

This is a known issue that happens very rarely, if you encounter this during Windows node pool creation, you can mitigate this following the steps:

  • If you are using IPAM mode, you can manually remove the VMs that having IP conflicts from vCenter, new VMs will be created automatically which should have correct IP allocations. Or you could just wait for the node auto repair to detect this issue and recreate the Windows nodes.

  • If you are using DHCP mode, the newly created VMs are likely to have duplicated IPs again since the DHCP server is encountering issues for IP allocation, you can delete the pending Windows node pool by running gkectl update cluster, and add it back in user-cluster.yaml, run gkectl update cluster again for creating it, the newly created node pool should have correct IP allocations.