Zonal network endpoint groups overview

You can use zonal network endpoint groups (NEGs) as the backend for a backend service. The primary use case for this configuration is deploying containers on your VMs so that you can run services in the containers. You can also distribute traffic in a granular fashion to applications running on the VMs.

This document discusses using zonal network endpoint groups with the following load balancer types:

• HTTP(S)
• Internal HTTP(S)
• TCP proxy
• SSL proxy

You cannot use zonal NEGs as a backend with the following load balancer types:

• Internal TCP/UDP
• Network

For information about internet NEGs, see Internet network endpoint groups overview.

For information about serverless NEGs, see Serverless network endpoint groups overview.

Overview

Zonal network endpoint groups (NEGs) are zonal resources that represent collections of IP address and port combinations for Google Cloud resources within a single subnet. Each IP address and port combination is called a network endpoint.

Zonal network endpoint groups can be used as backends in backend services for HTTP(S), Internal HTTP(S) Load Balancing, TCP proxy, and SSL proxy load balancers. You cannot use zonal NEGs as a backend with internal TCP/UDP load balancers. Because zonal NEG backends allow you to specify IP addresses and ports, you can distribute traffic in a granular fashion among applications or containers running within VM instances.

Endpoint relationships

When you create a network endpoint group, you select a zone, a network, and a subnet. Every endpoint IP address must be in the same subnet as the zonal NEG.

If the network you select is an auto mode network, you can omit specifying the subnet. However, a subnet is still associated with the zonal NEG. If you specify an auto mode network but do not specify a subnet when creating a zonal NEG, the subnet it uses is the automatically-created subnet in the region that contains the zone that you selected for the zonal NEG.

Network endpoint groups currently only support the GCE_VM_IP_PORT type. This means that individual endpoints must have IP addresses associated with Google Cloud VMs. When adding endpoints to a zonal NEG:

• You must specify the name of a VM instance. The VM instance must be in the same zone as the zonal NEG, and its network interface in the VPC network must be in a subnet in the same region that contains that zone.

• You can have up to 10,000 endpoints in a zonal NEG. All 10,000 endpoints can be on the same VM. Each endpoint is created by referencing the VM when adding an endpoint to an existing zonal NEG. Endpoints created in different zonal NEGs can point to the same VM.

  • You can specify an IP address or an IP address and port combination in addition to specifying the VM instance. Any IP address you specify must either be the primary internal IP address of the VM or an alias IP for the VM in the same subnet as the zonal NEG.

  • If you do not specify an IP address when you add an endpoint, the VM's primary internal IP address in the VPC network is used.

  • If you do not specify a port when adding an endpoint, you must have defined a default port for the zonal NEG. Endpoints use the default port unless they specify a port of their own. This means that you must either specify a default port when you create a zonal NEG, or you must specify a port for each and every endpoint you add to it.

  • Every endpoint in a zonal NEG must be a unique IP address and port combination. If a port is not specified for an IP address, the combination is created by using the IP and the default port.

Endpoints, containers, and services

To create a unique network endpoint for a container or application running in a VM, you must either use the primary IP address of the VM or a secondary IP assigned to the VM using the alias IP address feature. The software running in the container or the application running in the VM should be configured to bind to the IP address used by the network endpoint.

NEGs are especially useful for GKE. For information on using zonal NEGs with GKE, refer to Using Container-native Load Balancing.

NEGs are also useful because they allow you to create logical groupings of IP addresses and ports representing software services instead of entire VMs. IP addresses for microservices (running in Google Cloud VMs) managed by other orchestrators like Apache Mesos or Cloud Foundry can be endpoints.

Important: All endpoints in a zonal NEG must be internal (RFC 1918) IP addresses in the same subnet as the zonal NEG. NEGs cannot point to public IP addresses, and they cannot point to arbitrary private IP addresses outside of the ones in the chosen subnet and VPC network.

Load balancing

Zonal NEGs can be used as backends for backend services in the following types of load balancers:

• An external HTTP(S) load balancer
• An internal HTTP(S) load balancer
• An SSL proxy load balancer
• A TCP proxy load balancer

The primary use case for zonal NEGs is container-native load balancing so that you can distribute traffic among microservices running in containers on your VMs.

The following example demonstrates how load balancers distribute traffic among microservices running in containers on your VMs. The VMs are configured to use alias IP ranges from their subnets, and those ranges are the addresses used by the containers.

Load balancing zonal network endpoint groups with containers (click to enlarge)

Backend services

Zonal NEGs can be used as backends for backend services in a load balancer. When you use a zonal NEG as a backend for a backend service, all other backends in that backend service must also be zonal NEGs. You cannot use instance groups and zonal NEGs as backends in the same backend service.

You can add the same network endpoint (IP address and port combination) to more than one zonal NEG. You can use the same zonal NEG as a backend for more than one backend service.

Backend services using zonal NEGs for backends can only use balancing modes of RATE or CONNECTION. You cannot use a balancing mode of UTILIZATION for backend services that use zonal NEGs as backends.

HTTP(S), Internal HTTP(S), TCP Proxy, and SSL Proxy load balancing

You can use zonal network endpoint groups in load balancers using either Standard or Premium network service tiers.

The following illustrations show configuration components for HTTP(S) load balancers, TCP/SSL Proxy load balancers and Internal HTTP(S) load balancers where zonal NEGs are the backends:

• Each Premium Tier HTTP(S), SSL Proxy, and TCP Proxy load balancer has its own global external forwarding rule to direct traffic to the appropriate target proxy object.

• Each Standard Tier HTTP(S), SSL Proxy, and TCP Proxy load balancer has its own regional external forwarding rule to direct traffic to the appropriate target proxy object.

• Each internal HTTP(S) load balancer has its own regional internal managed forwarding rule to direct traffic to the appropriate target proxy object.

• For target HTTP(S) proxies, the backend service used is determined by checking the request host name and path in the URL map. External HTTP(S) and internal HTTP(S)load balancers can have multiple backend services referenced from the URL map.

• For target TCP or target SSL proxies, only one backend service can be defined.

• The backend service directs traffic to its backend zonal NEGs. For each request, the load balancer picks a network endpoint from one of the zonal NEGs and sends the traffic there.

Zonal network endpoint groups in load balancing (click to enlarge)

Container-native load balancing

Container-native load balancing enables load balancers to target Pods directly and to make load distribution decisions at the Pod-level instead of at the VM-level. There are two ways to configure container-native load balancing: either use NEGs managed by GKE Ingress, or use standalone NEGs.

Kubernetes Ingress with NEGs (Recommended)

When NEGs are used with Ingress, the Ingress controller facilitates the creation of all aspects of the L7 load balancer. This includes creating the virtual IP address, forwarding rules, health checks, firewall rules, and more. To learn how to configure this, see Container-native load balancing through Ingress.

Ingress is the recommended way to use NEGs for container-native load balancing as it has many features that simplify the management of NEGs. Standalone NEGs can be used if NEGs managed by Ingress do not serve your use case.

For instructions about how to set up a load balancer through Ingress, see Container-native load balancing through Ingress.

Standalone NEGs

When zonal NEGs are deployed with load balancers provisioned by anything other than GKE Ingress, they are considered standalone NEGs. Standalone zonal NEGs are deployed and managed through the NEG controller, while the forwarding rules, health checks, and other load balancing components are deployed manually.

The following list describes the high-level process required to set up load balancing with standalone NEGs:

1. Configure containers or services on the VMs. If multiple containers should run on each VM or if you need IP addresses for the containers, configure alias IP addresses for the VMs. If you are configuring services, you need two or more services running on the same VM such that at least the port numbers are different.
2. Create zonal network endpoint groups.
3. Add network endpoints to the zonal network endpoint groups.
4. Create a health check.
5. Create a backend service.
6. For HTTP(S) Load Balancing, create a URL map and connect the backend service to it.
7. Create the appropriate type of target proxy, such as a target HTTP proxy, target HTTPS proxy, target SSL proxy, or target TCP proxy. Link the target proxy to the URL map (for HTTP(S) load balancing) or the backend service (for TCP Proxy and SSL Proxy load balancing).
8. Create a forwarding rule and link it to the target proxy.

To attach a load balancer to standalone zonal NEGs, see the following:

• Attach an external HTTP(S) load balancer to standalone zonal NEGs
• Attach an internal HTTP(S) load balancer to standalone zonal NEGs

Restrictions

• You cannot use zonal NEGs with legacy networks.
• The IP address for a network endpoint must be a primary or alias IP that belongs to the specified VM instance.

Limits

• Zonal NEGs are only usable as backends for load balancers. The following load balancer types support zonal NEGs: external HTTP(S), internal HTTP(S), TCP Proxy, and SSL Proxy.
• Only RATE balancing mode is supported by zonal NEGs for HTTP(s) load balancing, while CONNECTION is supported for TCP/SSL load balancing. Utilization-based load balancing is not supported.
• A backend service that uses NEGs as backends cannot also use instance groups as backends.
• Only internal (RFC 1918) IP addresses can currently be added to a zonal NEG.
• For information about NEG quotas—such as NEGs per project, NEGs per backend service, and endpoints per NEG—see the load balancing quotas page.

Troubleshooting

Traffic does not reach the endpoints

After the service is configured, new endpoints generally become reachable after attaching them to the zonal NEG, provided they respond to health checks.

If traffic cannot reach the endpoints, which results in a 502 error code for HTTP(s) or rejected connections for TCP/SSL load balancers, check the following:

• Verify that firewall rules allow incoming TCP traffic to your endpoints from following ranges: 130.211.0.0/22 and 35.191.0.0/16.
• Verify that your endpoints are healthy by using gcloud, as shown below, or by calling the getHealth API on the backend service resource or the listEndpoints API on the zonal NEG with the showHealth parameter set to SHOW. The following gcloud command shows health information by network endpoint:

gcloud compute network-endpoint-groups list-network-endpoints NAME \
    --zone=ZONE

What's next

• For information about configuring zonal NEGs, see Setting Up zonal network endpoint groups in load balancing.
• For information about using zonal network endpoint groups in Google Kubernetes Engine, see Using container-native load balancing.