Configuring an egress NAT gateway

Version 1.9 

• Version 1.28 (latest)
• Version 1.16
• Version 1.15
• Earlier versions (unsupported)

Preview

This product or feature is subject to the "Pre-GA Offerings Terms" in the General Service Terms section of the Service Specific Terms. Pre-GA products and features are available "as is" and might have limited support. For more information, see the launch stage descriptions.

With GKE on VMware, you can configure source network address translation (SNAT) so that certain egress traffic from your user cluster is given a predictable source IP address.

This document shows how to configure an egress NAT gateway for a user cluster.

Caution: You can't upgrade any version 1.8.x or 1.9.x cluster that has Anthos Network Gateway enabled to any later version. For example, you cannot upgrade a version 1.9.0 cluster that has Anthos Network Gateway enabled to version 1.9.x, nor can you upgrade it to version 1.10. You can only upgrade clusters that have Anthos Network Gateway enabled if they were created in version 1.10 or later. You can delete a version 1.8.x or version 1.9.x cluster, and then recreate it following the instructions in this document.

Introduction

Sometimes you have Pods running in a user cluster that need to send packets to components running in your organization, but outside of the cluster. You might want to design those external components so that they filter incoming network traffic according to a set of well-known source IP addresses.

Here are some scenarios:

• You have a firewall in front of a database that allows access only by IP address. And the team that manages the database firewall is different from the team that manages the user cluster.

• Workloads in your user cluster have to access a third-party API over the internet. For security reasons, the API provider authenticates and authorizes traffic by using IP address as the identity.

With an egress NAT gateway, you can have fine-grained control over the source IP addresses used for network traffic that leaves a cluster.

Pricing

There is no charge for using this feature during the preview.

How an egress NAT gateway works

Ordinarily, when a Pod sends a packet out of the cluster, the packet is SNAT translated with the IP address of the node where the Pod is running.

When an egress NAT gateway is in place, you can specify that certain outbound packets should be sent first to a dedicated gateway node. The network interface on the gateway node is configured with two IP addresses: the primary IP address and an egress source IP address.

When a packet that has been selected to use the egress NAT gateway, the packet leaves the cluster from the gateway node and is SNAT translated with the egress source IP address that is configured on the network interface.

The following diagram illustrates the packet flow:

The preceding diagram, you can see the flow of a packet that is sent from the Pod.

1. On a node with IP address 192.168.1.1, a Pod with IP address 10.10.10.1 generates an outbound packet.

2. The packet matches an egress rule, so the packet is forwarded to the gateway node.

3. The gateway node changes the source IP address to 192.168.1.100 and sends the packet out of the cluster.

4. Return traffic comes back to the gateway node with destination 192.168.1.100.

5. The gateway node uses conntrack to modify the destination IP address to 10.10.10.1.

6. The packet is treated as in-cluster traffic and forwarded to the original node and delivered back to the original Pod.

Personas

This topic refers to two personas:

• Cluster administrator. This person creates a user cluster and specifies floating IP addresses to be used by Anthos Network Gateway.

• Developer. This person runs workloads on the user cluster and creates egress policies.

Enable Anthos Network Gateway

This section is for cluster administrators.

Create a new user cluster with Anthos Network Gateway enabled.

In your user cluster configuration file, set enableDataplaneV2 and enableAnthosNetworkGateway to true

enableDataplaneV2: true
...
enableAnthosNetworkGateway: true

Create the user cluster.

Specify floating IP addresses

This section is for cluster administrators.

Choose a set of IP addresses that you would like to use as egress source addresses. These are called floating IP addresses, because Anthos Network Gateway assigns them, as needed, to the network interfaces of nodes that it chooses to be egress gateways.

Your floating IP addresses must be in the same subnet as your node IP addresses.

Your set of floating IP addresses must not overlap with the set of IP addresses you have specified for your nodes.

For example, suppose a subnet has the address range 192.168.1.0/24. And suppose you have chosen to use 192.168.1.1 through 192.168.1.99 for nodes. Then you could use 192.168.1.100 through 192.168.1.104 as floating IP addresses.

Create an AnthosNetworkGateway object

This section is for cluster administrators.

Here's an example of a manifest for an AnthosNetworkGateway object:

kind: AnthosNetworkGateway
apiVersion: networking.gke.io/v1alpha1
metadata:
  namespace: kube-system
  name: default
spec
  floatingIPs:
  - 192.168.1.100
  - 192.168.1.101
  - 192.168.1.102
  - 192.168.1.103
  - 192.168.1.104

Replace the floatingIPs array with your floating IP addresses, and save the manifest in a file named my-ang.yaml.

Create the AnthosNetworkGateway object:

kubectl --kubeconfig USER_CLUSTER_KUBECONFIG apply -f my-ang.yaml

Example of an egress NAT policy

This section is for developers.

Here's an example of an EgressNatPolicy custom resource:

kind: EgressNATPolicy
metadata:
  name: alice-paul
spec:
  egress:
  - namespaceSelector:
      matchLabels:
        user: alice
    podSelector:
      matchLabels:
        role: frontend
  - namespaceSelector:
      matchLabels:
        user: paul
    podSelector:
      matchLabels:
        role: frontend
  destinationCIDRs:
  - 8.8.8.0/24
  egressSourceIP: 192.168.1.100

In the preceding manifest, we see:

A Pod is a candidate for egress NAT if it satisfies one the following:

• The Pod has the label role: frontend, and the Pod is in a namespace that has the label user: alice.

• The Pod has the label role: frontend, and the Pod is in a namespace that has the label user: paul.

Traffic from a candidate Pod to an address in the 8.8.8.0/24 range is sent to the egress NAT gateway.

Note: The egressSourceIP that you specify in your EgressNATPolicy object must be listed in the AnthosNetworkGateway object, and it must be assigned to a node. If that is not the case, then you will get an UnknownEgressIP error in the Kubernetes events for the EgressNATPolicy object.

Create an EgressNATPolicy object

Create your own EgressNATPolicy manifest. Set metadata.name to "my-policy". Save your manifest in a file named my-policy.yaml.

Create the EgressNatPolicy object:

kubectl apply --kubeconfig USER_CLUSTER_KUBECONFIG -f my-policy.yaml

View information about your egress NAT policy

kubectl --kubeconfig USER_CLUSTER_KUBECONFIG get egressnatpolicy my-policy --output yaml

kubectl --kubeconfig USER_CLUSTER_KUBECONFIG get anthosnetworkgateway --namespace kube-system --output yaml

kubectl --kubeconfig USER_CLUSTER_KUBECONFIG describe egressnatpolicy my-policy

Order of operations

Egress NAT policy is compatible with network policy APIs. Network policy is evaluated before egress NAT policy. If a network policy says to drop a packet, the packet is dropped regardless of the egress NAT policy.

Known issues

Duplicate egressSourceIP addresses

When you create an EgressNATGateway object, you might specify an egressSourceIP address that is already in use for another EgressNATPolicy object. This will cause egress traffic routing conflicts. Coordinate with your development team to determine which floating IP addresses are available before specifying the egressSourceIP address.

Changes to custom resources

Changes to EgresNATPolicy custom resources are not populated to the Cilium custom resources.