Routes overview

Google Cloud routes define the paths that network traffic takes from a virtual machine (VM) instance to other destinations. These destinations can be inside your Google Cloud Virtual Private Cloud (VPC) network (for example, in another VM) or outside it.

In a VPC network, a route consists of a single destination prefix in CIDR format and a single next hop. When an instance in a VPC network sends a packet, Google Cloud delivers the packet to the route's next hop if the packet's destination address is within the route's destination range.

This page provides an overview of how routes work in Google Cloud.

Routing in Google Cloud

Every VPC network uses a scalable, distributed virtual routing mechanism. There is no physical device that's assigned to the network. Some routes can be applied selectively, but the routing table for a VPC network is defined at the VPC network level.

Each VM instance has a controller that is kept informed of all applicable routes from the network's routing table. Each packet leaving a VM is delivered to the appropriate next hop of an applicable route based on a routing order. When you add or delete a route, the set of changes is propagated to the VM controllers by using an eventually consistent design.

Route types

The following tables summarize how Google Cloud categorizes routes in VPC networks.

Type and destination Next hop Notes
System-generated routes
System-generated default route
0.0.0.0/0
default-internet-gateway Applies to the whole VPC network

Can be removed or replaced
Subnet route
Created automatically for each subnet IP address range
VPC network
Forwards packets to VMs and internal load balancers
Applies to the whole VPC network

Created, updated, and removed automatically by Google Cloud when you create, modify, or delete a subnet or secondary IP address range of a subnet.
Custom routes
Static route
Supports various destinations
Valid static route next hop For next hop internal TCP/UDP load balancers, only applies to TCP and UDP traffic in the same region as the load balancer, unless global access is enabled.

For other static next hops: The route can apply to specific VMs if you specify network tags on the route. If you don't specify any network tag, the route applies to all VMs in the VPC network.
Dynamic route
Destinations that don't conflict with subnet routes or static routes
Peer of a BGP session on a Cloud Router Routes are added and removed automatically by Cloud Routers in your VPC network.

Routes apply to VMs according to the VPC network's dynamic routing mode.
Peering routes
Peering subnet route
Represents a subnet IP address range in a network connected using VPC Network Peering
Peer VPC network
Forwards packets to VMs and internal load balancers in the peer network
Applies to the whole VPC network

Created, updated, and removed automatically by Google Cloud when subnets are created, modified, or deleted in the peer network.
Peering custom route
Custom static or custom dynamic route in a network connected using VPC Network Peering
Next hop in the peer VPC network Peering custom routes backed by static routes in the peer network apply as follows:
  • If the peering custom route is backed by a custom static route whose next hop is an internal TCP/UDP load balancer, the route applies to TCP and UDP traffic in the same region as the load balancer, unless global access is enabled on the load balancer's forwarding rule.
  • If the peering custom route is backed by a custom static route whose next hop is not an internal TCP/UDP load balancer, and that custom static route doesn't have an associated tag: The route applies to all VMs in the VPC network that imports the route.
  • Static routes in a peer network that use network tags or whose next hops are the default internet gateway are never exported in a peering relationship.
Peering custom routes backed by dynamic routes in the peer network apply according to the dynamic routing mode of the VPC network that imports the routes.

System-generated default route

When you create a VPC network, it includes a system-generated default route. This default route serves two purposes:

  • It defines the path out of the VPC network, including the path to the internet. In addition to having this route, instances must meet additional requirements if they need internet access.

  • It provides the standard path for Private Google Access.

The system-generated default route has a priority of 1000. Because its destination is the broadest possible (0.0.0.0/0), Google Cloud only uses it if a route with a more specific destination does not apply to a packet. For information about how destination specificity and route priority are used to select a route, see routing order.

If you want to completely isolate your network from the internet or if you need to replace the default route with a custom route, you can delete the default route:

  • If you want to route internet traffic to a different next hop, you can replace the default route with a custom static or dynamic route. For example, you could replace it with a custom static route whose next hop is a proxy VM.

  • If you remove the default route and do not replace it, packets destined to IP ranges that are not covered by other routes are dropped.

Subnet routes

Subnet routes define paths to resources like VMs and internal load balancers in a VPC network.

Each subnet has at least one subnet route whose destination matches the primary IP range of the subnet. If the subnet has secondary IP ranges, there's a corresponding subnet route for each of its secondary IP address ranges. For more information about subnet IP ranges, see networks and subnets.

No other route can have a destination that matches or is more specific (has a longer subnet mask) than the destination of a subnet route.

Subnet routes always have the most specific destinations. They cannot be overridden by other routes, even if another route has a higher priority. This is because Google Cloud considers destination specificity before priority when selecting a route. Google Cloud uses 0 as the priority for all subnet routes.

The following points describe how subnet routes are created and removed:

  • When you add a subnet, Google Cloud creates a corresponding subnet route for the subnet's primary IP address range.

  • Auto mode VPC networks create a subnet route for the primary IP ranges of each of their automatically created subnets. You can delete these subnets only if you convert the auto mode VPC network to custom mode.

  • You cannot delete a subnet route unless you modify or delete the subnet:

    • When you remove a secondary range from a subnet, the subnet route for that secondary range is deleted automatically.

    • When you delete a subnet, all subnet routes for both primary and secondary ranges are deleted automatically. You cannot delete the subnet route for the subnet's primary range in any other way.

The number of subnet routes in a VPC network is limited by the Maximum number of subnet IP ranges (primary and secondary).

Static routes

Static routes are defined using static route parameters and support static route next hops. You can create static routes in one of two ways:

Dynamic routes

Dynamic routes are managed by Cloud Routers in the VPC network. Their destinations always represent IP address ranges outside your VPC network, received from a BGP peer. Dynamic routes are used by:

VPC networks ignore any destinations received by Cloud Routers when the destinations match either of these criteria:

  • The destination exactly matches a subnet IP address range.

  • The destination fits within (has a longer subnet mask than) a subnet IP address range.

It is acceptable for a dynamic route's destination to contain (to have a shorter subnet mask than) a subnet IP address range. In that case, packets are only sent to the dynamic route's next hop if they do not fit within the subnet route's destination. The broadest destination possible is 0.0.0.0/0.

Peering subnet routes

Peering subnet routes define paths to resources using subnets in another VPC network connected using VPC Network Peering. The other network is called a peer VPC network.

Peer networks share subnet routes in the following way:

  • Peer networks always export their subnet routes, for all primary and secondary IP address ranges.

  • Peer networks automatically import subnet routes as long as the route's destination (subnet IP address range) is not a privately used public IP address. You can configure a peer to import the subnet routes that use privately used public IP addresses if necessary.

  • Imported peering subnet routes cannot be removed unless you disable the peering. When you disable the peering, all imported peering subnet routes are automatically removed.

  • Google Cloud prevents subnet IP address ranges conflicts between peered networks.

The combined number of subnet routes in a VPC network and peering subnet routes imported from all peer networks is limited by the Maximum number of subnet IP ranges (primary and secondary).

Peering custom routes

You can configure peer networks to export or import custom routes. In this context, custom routes means both static and dynamic routes in a VPC network.

Even when a VPC network is configured to export its custom routes, it omits these types of routes:

The combined number of static and dynamic routes in a VPC network plus peering custom routes imported from all peer networks is limited by:

  • Maximum number of static routes in a peering group
  • Maximum number of dynamic routes in a peering group

Applicability and order

Applicable routes

Each instance has a set of applicable routes, which are a subset of all routes in the VPC network. Applicable routes are the possible egress paths that a packet can take when sent from the instance.

  • Subnet and peering subnet routes apply to all instances. Subnet routes and peering subnet routes correspond to subnets in your VPC network and those imported from peer networks.

  • The system-generated default route applies to all instances. You can replace the system-generated default route with a custom static route if you prefer.

  • Custom static routes can apply to all instances or specific instances. Static routes with a tag attribute apply to instances that have that same network tag. If the route doesn't have a network tag, the route applies to all instances in the network.

    • When a custom static route has a VM instance next hop, the route is always valid, even if the next-hop VM is deleted, powered off, or malfunctioning. For more information, see instances as next hops.

    • When a custom static route has a Cloud VPN tunnel next hop, the route is always valid if the tunnel is up. For information about how Google Cloud handles routes for tunnels when they are down, see when tunnels are down in the Cloud VPN documentation.

  • Dynamic routes apply to instances based on the dynamic routing mode of the VPC network. If a VPC network uses regional dynamic routing mode, all of its Cloud Routers only create dynamic routes for learned prefixes in their local region. If a VPC network uses global dynamic routing mode, all of its Cloud Routers create dynamic routes for learned prefixes in all regions.

    • Cloud Routers automatically discard learned prefixes that correspond to inaccessible next hops (Cloud VPN tunnels or Cloud Interconnect attachments). Depending on your network, Cloud Routers can take up to 40 seconds of processing time to remove a dynamic route with a next hop that's down.
  • For some load-balanced traffic, the applicable routes originate outside your VPC network. For more information, see load balancer return paths.

Routing order

When an instance sends a packet, Google Cloud attempts to select one route from the set of applicable routes according to the following routing order.

  1. Subnet routes and peering subnet routes are considered first because these routes have the most specific destinations. Google Cloud considers peering subnet routes in the same way as it considers local subnet routes.

    • If the destination IP address for a packet fits inside a subnet or peering subnet route's destination:

      • Packets are delivered to the internal IP address of a running VM instance or configured internal load balancer.

      • Packets destined for a stopped VM instance are dropped.

      • Packets destined for an internal IP address are dropped if no Google Cloud resource has been configured to use the IP address.

    • Google Cloud does not allow you to create a custom static route that has an equal or more specific destination than any subnet route or any peering subnet route.

    • Cloud Routers ignore learned prefixes that exactly match the destination of a subnet route or peering subnet route.

    • Cloud Routers ignore learned prefixes that fit within (have a longer subnet mask than) a subnet route or peering subnet route.

  2. If the packet cannot be routed by a subnet route or peering subnet route, Google Cloud looks for a static, dynamic, or peering custom route that has the most specific destination. For example,10.240.1.0/24 is a more specific destination than 10.240.0.0/16 for packets with the 10.240.1.4 destination.

  3. If more than one route has the same most specific destination, Google Cloud uses the following process to select a route from route candidates. Route candidates are the static, dynamic, and peering custom routes having the same most specific destination.

    1. If your VPC network is connected to peer networks, Google Cloud eliminates all route candidates except for those from a single VPC network.

      • If one or more route candidates exist in your local VPC network, Google Cloud disregards all route candidates from peer networks.

      • If none of the route candidates exist in your local VPC network, but instead exist in multiple peer networks, Google Cloud disregards all route candidates except for those defined in one of the peer networks. Google Cloud uses an internal algorithm to select the single peer network, and this algorithm does not consider route priority at this point in the process. If you peer with a new network or disconnect from an existing peer network, that might change the one VPC network Google Cloud chooses.

    2. Google Cloud eliminates all route candidates except for those that have the highest priority. If this results in a single remaining route, Google Cloud sends the packet to its next hop.

    3. If multiple route candidates have the same highest priority, Google Cloud continues the elimination process using the route's next hop and route's type. If this results in a single remaining route, Google Cloud sends the packet to its next hop.

      1. Custom static routes with next hops Next Hop Instance, Next Hop IP or Next Hop VPN Tunnel are most preferable. All other route candidates are eliminated if a route candidate using this next hop type exists.

      2. Custom dynamic routes are the second most preferable.

      3. Custom static routes with next hops internal TCP/UDP load balancer are the third most preferable.

      4. Custom static routes using the default internet gateway next hop are the least preferable.

    4. If more than one route remains from the route candidates, those routes all exist within the same VPC network, have the same highest priority, and are of the same route type or use the same next hop type. Google Cloud distributes traffic in the following way:

      • If the next hop is not an internal TCP/UDP load balancer, Google Cloud distributes packets among the next hops of the route candidates by using a five-tuple hash for affinity, implementing equal-cost multi-path (ECMP). Hash calculations are done for each packet, at the time it is sent, based on the number of route candidates at this stage. If the number of route candidates changes, the hash might direct a packet with the same five-tuple hash to a different next hop.

      • If the next hop is an internal TCP/UDP load balancer, Google Cloud uses an internal algorithm to select a single next hop load balancer, ignoring the other load balancer next hops, even though they are associated with routes having the same priority. See Considerations specific to internal TCP/UDP load balancer next hops for additional details about this situation.

  4. If no applicable destination is found, Google Cloud drops the packet, replying with an ICMP destination or network unreachable error.

Special return paths

VPC networks have special routes for certain services. These routes are defined outside of your VPC network, in Google's production network. They don't appear in your VPC network's routing table. You cannot remove or override them, even if you delete or replace a default route in your VPC network. However, you can control traffic to and from these services by using firewall rules.

Load balancer return paths

If you use one of the following Google Cloud load balancers, Google Cloud has special routes for the load balancers and their associated health checks, which are described in more detail in the following sections.

Return paths for HTTP(S), SSL proxy, and TCP proxy load balancers

For these load balancer types, Google Front End (GFE) systems serve as proxies. When a client sends a request to the load balancer, a proxy terminates the TCP session and opens a new TCP session with your backend VM. Routes defined outside your VPC network facilitate communication from GFE proxies to your backend VMs and from your backend VMs to GFE proxies.

For more information, see the following pages:

Return paths for network load balancer

When a client on the internet sends a TCP or UDP request through a network load balancer to a backend VM, Google Cloud uses routes defined outside your VPC network to facilitate communication from the client to your backend VM and from your backend VM to the client.

For more information, see Network Load Balancing.

Health checks for all load balancer types

Packets sent from Google Cloud health check probe systems have sources as described in Probe IP ranges and firewall rules. Routes that facilitate communication between Google Cloud health check probe systems and your backend VMs exist outside your VPC network, and cannot be removed. However, your VPC network must have ingress allow firewall rules to permit traffic from these systems.

IAP

A route for 35.235.240.0/20 is included to support TCP forwarding using IAP. Google's production network does not accept routes for 35.235.240.0/20 from other sources on the internet.

Cloud DNS

A route to 35.199.192.0/19 supports connectivity to forwarding targets that use private routing or alternative name servers that use private routing. Google's production network does not accept routes for 35.199.192.0/19 from other sources on the internet.

Static route parameters

Each static route consists of the following components:

  • Name and Description. These fields identify the route. A name is required, but a description is optional. Every route in your project must have a unique name.

  • Network. Each route must be associated with exactly one VPC network.

  • Destination range. The destination range is a single IPv4 CIDR block that contains the IP addresses of systems that receive incoming packets. Destinations must be expressed in CIDR notation. Destinations cannot exactly match a subnet IP address range, nor can they fit within a subnet IP address range (they cannot have a longer subnet mask). It is acceptable for a static route's destination to contain a subnet IP address range (have a shorter subnet mask). In that case, packets are only sent to the static route's next hop if they do not fit within the subnet route's destination. The broadest destination possible is 0.0.0.0/0.

  • Priority. If multiple routes have identical destinations, priority is used to determine which route should be used. Lower numbers indicate higher priorities; for example, a route with a priority value of 100 has higher priority than one with a priority value of 200. Highest route priority means the smallest possible non-negative integer. Because zero is the smallest possible non-negative integer, it has the highest priority.

  • Next hop. Static routes can have next hops that point to the default internet gateway, a Google Cloud instance, or a Cloud VPN tunnel. For more information, see static route next hops.

  • Tags. You can specify a list of network tags so that the route only applies to instances that have at least one of the listed tags. If you don't specify tags, Google Cloud applies the route to all instances in the network. Next hop internal TCP/UDP load balancers do not support network tags.

Static route next hops

The following are valid next hops for static routes. For more information about each type, see the gcloud reference documentation.

  • Next Hop Gateway (next-hop-gateway). You can specify a default internet gateway to define a path to external IP addresses.

  • Next Hop Instance (next-hop-instance). You can direct traffic to an existing instance in Google Cloud by specifying its name and zone. Traffic is directed to the primary internal IP address of the VM's network interface in the VPC network where you define the route.

    • If the primary internal IP address for the VM instance's network interface in the VPC network changes, Google Cloud updates the VPC network's routing table automatically so that traffic continues to be sent to the VM at its new IP address.

    • If the VM is replaced by a new VM with the same name in the same zone, Google Cloud updates the VPC network's routing table automatically so that traffic is directed to the replacement VM.

    • Google Cloud validates only that a next hop VM exists, and only when you create the route. It does not validate that the VM is able to process or forward packets. If the VM is later deleted but not replaced, the route still applies, which might result in dropped packets.

  • Next Hop Internal TCP/UDP Load Balancer (next-hop-ilb). For Internal TCP/UDP Load Balancing, you can use a load balancer's IP address as a next hop that distributes traffic among healthy backend instances. For example, you can use a custom static route whose next hop is an internal TCP/UDP load balancer to distribute traffic among backend VMs. Custom static routes that use this next hop cannot be scoped to specific instances by network tag.

  • Next Hop IP (next-hop-address). You can provide a primary internal IP address assigned to a Google Cloud VM interface as a next hop.

    • When you use next-hop-address, Google Cloud passes traffic to any VM instance assigned to that IP address. If you replace a VM instance with another one that uses the same primary internal IP address, traffic goes to the replacement, even if it has a different name. To define a next hop by VM name, use Next Hop Instance instead.

    • Google Cloud validates only that the IP address is a valid member of a subnet's primary or secondary IP range when you create the route. It does not validate that an instance exists at the Next Hop IP address. If the Next Hop IP address is not assigned to any instance, the route still applies, which might result in dropped packets. For more information, see Considerations for instance and load balancer next hops.

    • You cannot specify an arbitrary IP address next hop: Addresses outside of a subnet's IP address range in your VPC network are not allowed.

  • Next Hop VPN Tunnel (next-hop-vpn-tunnel). For Cloud VPN tunnels that use policy-based routing and route-based VPNs, you can direct traffic to the VPN tunnel by creating routes whose next hops refer to the tunnel by its name and region. Google Cloud ignores routes whose next hops are Cloud VPN tunnels that are down. For more examples about how routes and Cloud VPN tunnels interact, see the Cloud VPN routing examples.

Considerations for instance and load balancer next hops

Instance-based routing refers to a static route with a next hop that is a VM instance (next-hop-instance or next-hop-address).

internal TCP/UDP load balancer as a next hop refers to a static route with a next hop that is an internal TCP/UDP load balancer (next-hop-ilb).

When you configure instance-based routing or an internal TCP/UDP load balancer as a next hop, consider the following guidelines:

  • You must configure the backend VMs or the Next Hop Instance to be packets from any source IP address. You must enable IP forwarding (can-ip-forward) on a per-VM basis when you create the VM. To enable IP forwarding for VMs created automatically as part of a managed instance group, you must enable IP forwarding in the instance template used by the instance group. You must make this configuration change in addition to any operating system configuration necessary to route packets.

  • Software running on the backend VM or the Next Hop Instance must be configured appropriately. For example, third-party appliance VMs that act as routers or firewalls must be configured according to the manufacturer's instructions.

  • Backend VMs or the Next Hop Instance must have appropriate firewall rules. You must configure firewall rules that apply to the packets being routed. Keep the following in mind:

    • Ingress firewall rules applicable to instances that perform routing functions must include the IP addresses of routed packet sources. The implied deny ingress rule blocks all incoming packets, so you must at least create custom ingress allow firewall rules.
    • Egress firewall rules applicable to instances that perform routing functions must include the IP addresses of routed packet destinations. The implied allow egress rule permits this unless you've created a specific egress deny rule to override it.
    • Take into account whether the backend VM or Next Hop Instance is performing Network Address Translation (NAT) when creating your firewall rules.

    For more information, see Implied firewall rules.

Considerations specific to next hop instances

  • When you use an instance as a next hop (next-hop-instance or next-hop-address), remember that the instance is a zonal resource. Selecting a zone implies that you have selected a region. Google Cloud does not consider regional distance for routes that use a Next Hop Instance, so it is possible to create a route that sends traffic to a next hop in a different region. This adds egress costs and introduces network latency.

  • Google Cloud performs one of the following basic validations only when you create the route:

    • If you specify next-hop-instance, the instance must already exist.
    • If you specify next-hop-address, the IP address must be in an existing subnet's primary or secondary IP range.

    If you, for example, remove an instance or delete a subnet later, Google Cloud still considers any route that uses that instance as a next hop when it evaluates applicable routes. This might cause some or all packets for a given destination to be dropped, depending on the other routes in your network.

  • If software in a VM instance (such as the VM's operating system or the VM's process that routes packets) fails, packets destined for that instance are dropped. To enhance reliability, consider using an internal TCP/UDP load balancer as a next hop instead. An internal TCP/UDP load balancer requires a configured health check, and this health check determines how new connections are routed.

  • Disabling a network interface by configuring the guest operating system of the instance does not cause Google Cloud to stop considering the interface to be a route's next hop.

  • If you want to route traffic to an instance in a Shared VPC service project, use next-hop-address. Using next-hop-instance is not supported.

Considerations specific to internal TCP/UDP load balancer next hops

  • Network tags are not supported. A custom static route cannot use network tags if the route has an internal TCP/UDP load balancer next hop.

  • When all backends are unhealthy. When all backends of an internal TCP/UDP load balancer fail health checks, the routes using that load balancer next hop are still in effect. Packets processed by the route are sent to one of the next hop load balancer's backends according to traffic distribution.

  • Forwarding rules that use a common internal IP address (--purpose=SHARED_LOADBALANCER_VIP) are not supported. Next hop internal TCP/UDP load balancer and internal TCP/UDP load balancer forwarding rules with a common IP address are mutually exclusive features. A next hop internal TCP/UDP load balancer must use an IP address that is unique to the load balancer's forwarding rule so that only one backend service (one load balancer) is unambiguously referenced. It's possible for forwarding rules that use a common internal IP address to reference different backend services (different internal TCP/UDP load balancers).

  • Same destination and multiple next hop internal TCP/UDP load balancers. If you create two or more custom static routes with the same destination, using different internal TCP/UDP load balancer next hops, Google Cloud never distributes traffic among the load balancer next hops using ECMP. If the routes have unique priorities, Google Cloud uses the next hop internal TCP/UDP load balancer from the route with the highest priority. If the routes have equal priorities, Google Cloud still selects just one next hop internal TCP/UDP load balancer. In this latter situation, as illustrated in the diagram below, Google Cloud uses a deterministic, internal algorithm to select a single next hop forwarding rule (forwarding-rule-a), ignoring other routes with the same priority.

    Same destination, different internal TCP/UDP load balancer next hops
    Same destination, different internal TCP/UDP load balancer next hops
  • Multiple destinations and the same next hop internal TCP/UDP load balancer. You can use the same next hop internal TCP/UDP load balancer for multiple custom static routes as long as each route has a unique combination of destination and priority. You cannot create multiple custom static routes having the same combination of destination, priority, and internal TCP/UDP load balancer next hop. For example, route-x, route-y, and route-z can all be created, but route-x-copy cannot be created.

    Example routes using a common internal TCP/UDP load balancer next hop
    Example routes using a common internal TCP/UDP load balancer next hop

Considerations for Classic VPN tunnel next hops

Google Cloud does not consider regional distance for routes that use a next hop Classic VPN tunnel. Sending traffic to a next hop Classic VPN tunnel in another region adds egress costs and introduces network latency. As a best practice, use HA VPN tunnels or Classic VPN tunnels that use dynamic routing instead.

What's next