Version 1.6. This version is supported as outlined in the Anthos version support policy, offering the latest patches and updates for security vulnerabilities, exposures, and issues impacting Anthos clusters on VMware (GKE on-prem). Refer to the release notes for more details. This is not the most recent version.

Proxy and firewall rules

This page shows how to set up proxy and firewall rules for Anthos clusters on VMware (GKE on-prem).

Allowlisting addresses for your proxy

If your organization requires outbound traffic to pass through a proxy server, allowlist the following addresses in your proxy server. Note that www.googleapis.com is needed, instead of googleapis.com:

  • dl.google.com (required by Google Cloud SDK installer)
  • gcr.io
  • www.googleapis.com
  • accounts.google.com
  • cloudresourcemanager.googleapis.com
  • container.googleapis.com
  • gkeconnect.googleapis.com
  • gkehub.googleapis.com
  • iam.googleapis.com
  • iamcredentials.googleapis.com
  • logging.googleapis.com
  • monitoring.googleapis.com
  • oauth2.googleapis.com
  • securetoken.googleapis.com
  • servicecontrol.googleapis.com
  • serviceusage.googleapis.com
  • storage.googleapis.com
  • sts.googleapis.com
  • checkpoint-api.hashicorp.com
  • releases.hashicorp.com (Optional. Required only if you use Terraform to create an admin workstation.)

If you use gkeadm to install Anthos clusters on VMware, you don't need to allowlist the hashicorp URLs above.

Also, if your vCenter Server has an external IP address, allowlist its address in your proxy server.

Firewall rules

Set up your firewall rules to allow the following traffic.

Firewall rules for IP addresses available in the admin cluster

The IP addresses available in the admin cluster are listed in the IP block file. These IP addresses are used for the admin cluster control-plane node, admin cluster add-on nodes, and the user cluster control-plane node. Because the IP addresses for the admin cluster are not assigned to specific nodes, you must make sure that all of the firewall rules listed in the following table apply to all of the IP addresses available for the admin cluster.

From

To

Port

Protocol

Description

Admin cluster control-plane node

vCenter Server API

443

TCP/https

Cluster resizing.

Admin cluster add-on nodes

vCenter Server API

443

TCP/https

User cluster lifecycle managemet.

User cluster control-plane node

vCenter Server API

443

TCP/https

Cluster resizing.

Cloud Logging Collector, which runs on an admin cluster add-on node

oauth2.googleapis.com
logging.googleapis.com
stackdriver.googleapis.com
servicecontrol.googleapis.com
www.googleapis.com

443

TCP/https

Cloud Monitoring Collector, which runs on an admin cluster add-on node

oauth2.googleapis.com
monitoring.googleapis.com
stackdriver.googleapis.com
servicecontrol.googleapis.com

443

TCP/https

Admin cluster control-plane node

F5 BIG-IP API

443

TCP/https

User cluster control-plane node

F5 BIG-IP API

443

TCP/https

Admin cluster control-plane node

On-premises local Docker registry

Depends on your registry

TCP/https

Required if Anthos clusters on VMware is configured to use a local private Docker registry instead of gcr.io.

User cluster control-plane node

On-premises local Docker registry

Depends on your registry

TCP/https

Required if Anthos clusters on VMware is configured to use a local private Docker registry instead of gcr.io.

Admin cluster control-plane node

* gcr.io
*.googleusercontent.com
*.googleapis.com

443

TCP/https

Download images from public Docker registries.

Not required if using a private Docker registry.

*.googleusercontent.com is the canonical DNS name for gcr.io.

User cluster control-plane node

* gcr.io
*.googleusercontent.com
*.googleapis.com
443

TCP/https

Download images from public Docker registries.

Not required if using a private Docker registry.

*.googleusercontent.com is the canonical DNS name for gcr.io.

Admin cluster worker nodes

Admin cluster worker nodes

All

179 - bgp

443 - https

5473 - Calico/Typha

9443 - Envoy metrics

10250 - kubelet node port

All worker nodes must be layer-2 adjacent and without any firewall.

Admin cluster nodes

Admin cluster pod CIDR

all

any

External traffic get SNAT'ed on the first node and sent to pod IP.

Admin cluster worker nodes

User cluster nodes

22

ssh

API server to kubelet communication over an SSH tunnel.

Admin cluster nodes

IPs of Seesaw LB VMs of the admin cluster

20255,20257

TCP/http

LB config push and metrics monitoring. Only needed if you are using Bundled LB Seesaw.

Firewall rules for user cluster nodes

In the user cluster nodes, their IP addresses are listed in the IP block file.

As with the admin cluster nodes, you don't know which IP address will be used for which node. Thus, all of the rules in the user cluster nodes apply to each user cluster node.

From

To

Port

Protocol

Description

User cluster worker nodes

Admin workstation Docker registry

User cluster worker nodes

* gcr.io
*.googleusercontent.com
*.googleapis.com

443

TCP/https

Download images from public Docker registries.

Not required if using a private Docker registry.

*.googleusercontent.com is the canonical DNS name for gcr.io.

User cluster worker nodes

F5 BIG-IP API

443

TCP/https

User cluster worker nodes

VIP of the pushprox server, which runs in the Admin cluster.

8443

TCP/https

Prometheus traffic.

User cluster worker nodes

User cluster worker nodes

all

22 - ssh

179 - bgp

443 - https

5473 - calico-typha

9443 - envoy metrics

10250 - kubelet node port"

All worker nodes must be layer-2 adjacent and without any firewall.

User cluster nodes

User cluster pod CIDR

all

any

External traffic get SNAT'ed on the first node and sent to pod IP.

Cloud Logging Collector, which runs on a random user cluster worker node

oauth2.googleapis.com
logging.googleapis.com
stackdriver.googleapis.com
servicecontrol.googleapis.com
www.googleapis.com

443

TCP/https

Connect agent, which runs on a random user cluster worker node.

gkeconnect.googleapis.com
gkehub.googleapis.com
www.googleapis.com
oauth2.googleapis.com
accounts.google.com

443

TCP/https

Connect traffic.

Cloud Monitoring Collector, which runs on a random user cluster worker node

oauth2.googleapis.com
monitoring.googleapis.com
stackdriver.googleapis.com
servicecontrol.googleapis.com

443

TCP/https

User cluster nodes

IPs of Seesaw LB VMs of the user cluster

20255,20257

TCP/http

LB config push and metrics monitoring. Only needed if you are using Bundled LB Seesaw.

Firewall rules for remaining components

These rules apply to all other components not listed in the tables for the admin cluster and user cluster nodes.

From

To

Port

Protocol

Description

Admin cluster pod CIDR

Admin cluster pod CIDR

all

any

Inter-pod traffic does L2 forwarding directly using source and destination IP within Pod CIDR.

Admin cluster pod CIDR

Admin cluster nodes

all

any

Return traffic of external traffic.

User cluster pod CIDR

User cluster pod CIDR

all

any

Inter-pod traffic does L2 forwarding directly using source and destination IP within Pod CIDR.

User cluster pod CIDR

User cluster nodes

all

any

Return traffic of external traffic.

Clients and application end users

VIP of Istio ingress

80, 443

TCP

End user traffic to the ingress service of a user cluster.

Jump server to deploy the admin workstation

checkpoint-api.hashicorp.com
releases.hashicorp.com
vCenter Server API
ESXi VMkernel (mgt) IPs of hosts in target cluster

443

TCP/https

Terraform deployment of the admin workstation.

Admin workstation

* gcr.io
*.googleusercontent.com
*.googleapis.com

443

TCP/https

Download Docker images from public Docker registries.

*.googleusercontent.com is the canonical DNS name for gcr.io.

Admin workstation

vCenter Server API

F5 BIG-IP API

443

TCP/https

Cluster bootstrapping.

Admin workstation

ESXi VMkernel (mgt) IPs of hosts in target cluster

443

TCP/https

The admin workstation uploads the OVA to the datastore through the ESXi hosts.

Admin workstation

Node IP of Admin Cluster control-plane VM

443

TCP/https

Cluster bootstrapping.

Admin workstation

VIP of the admin cluster's Kubernetes API server

VIPs of user clusters' Kubernetes API servers

443

TCP/https

Cluster bootstrapping.

User cluster deletion.

Admin workstation

Admin cluster control-plane node and worker nodes

443

TCP/https

Cluster bootstrapping.

Control plane upgrades.

Admin workstation

All admin cluster nodes and all user cluster nodes

443

TCP/https

Network validation as part of the gkectl check-config command.

Admin workstation

VIP of the admin cluster's Istio ingress

VIP of user clusters' Istio ingress

443

TCP/https

Network validation as part of the gkectl check-config command.

Admin workstation

IPs of Seesaw LB VMs in both admin and user clusters

Seesaw LB VIPs of both admin and user clusters

20256,20258

TCP/http/gRPC

Health check of LBs. Only needed if you are using Bundled LB Seesaw.

Admin workstation

Node IP of the admin cluster control plane

22

TCP

Required if you need SSH access from the admin workstation to the admin cluster control plane.

LB VM IPs

node IPs of the corresponding cluster

10256: node health check

30000 - 32767: healthCheckNodePort

TCP/http

Node health check. healthCheckNodePort is for services with externalTrafficPolicy set to Local. Only needed if you are using Bundled LB Seesaw.

F5 Self-IP

All admin and all user cluster nodes

30000 - 32767

any

For the data plane traffic that F5 BIG-IP load balances via a virtual server VIP to the node ports on the Kubernetes cluster nodes.

Typically the F5 self-ip is on the same network/subnet as the Kubernetes cluster nodes.