Order of routes

This page describes how custom routes with next hops of Cloud VPN tunnels work in Google Cloud.

Route types

A Cloud VPN tunnel can be a next hop for a custom route in your VPC network. Each custom route with a next hop Cloud VPN tunnel defines an egress path for packets leaving your VPC network:

  • A Classic VPN tunnel that uses dynamic routing or a HA VPN tunnel is always managed by a Cloud Router. The Cloud Router receives BGP advertisements from and sends BGP messages to a peer VPN gateway. Cloud Router automatically creates and removes custom dynamic routes in your VPC network — these are the routes with destinations in a peer network. It also advertises routes to a peer network — these are routes to the IP ranges of subnets in your VPC network, or to custom destinations that you specify in a Cloud Router configuration. The set of routes that Cloud Router imports and exports are controlled by the dynamic routing mode of your VPC network.

  • A policy-based or route-based Classic VPN tunnel uses static routing. If you create one of these tunnels using the Cloud Console, Google Cloud automatically creates custom static routes based on the remote IP ranges you specify in the Cloud VPN configuration. If you use gcloud commands to create one of these tunnels, you must manually create the static routes that use the tunnel as a next hop.

Example scenarios

Google Cloud follows a well-defined applicability and order when selecting the next hop to which a packet should be sent. The following examples demonstrate common interactions between routes and Cloud VPN.

Interaction with subnet routes

The following table demonstrates how subnet routes and custom routes interact. Each row represents a possible IP range of a subnet in a VPC network. The on-premises IP range is 10.2.0.0/16.

VPC network Cloud VPN tunnel routing
IP range of Google Cloud subnet Static (policy-based, route-based)
Classic VPN only
Dynamic (BGP)
Classic VPN or HA VPN
10.2.0.0/16
Same as on-premises IP range
Google Cloud does not allow you to create a custom static route whose destination is 10.2.0.0/16 and whose next hop is a Cloud VPN tunnel. The tunnel's associated Cloud Router ignores any advertised routes with a destination of 10.2.0.0/16. Traffic destined for 10.2.0.0/16 remains in your VPC network.
10.0.0.0/8
Broader than on-premises IP range
(smaller subnet mask)
Google Cloud does not allow you to create a custom static route whose destination is 10.2.0.0/16 and whose next hop is a Cloud VPN tunnel. The tunnel's associated Cloud Router ignores any advertised routes whose destinations match or fit within 10.0.0.0/8, including 10.2.0.0/16. Traffic destined for 10.0.0.0/8 remains in your VPC network.
10.2.99.0/24
Narrower than on-premises IP range
(longer subnet mask)
Google Cloud allows you to create a custom static route with the 10.2.0.0/16 destination and next hop Cloud VPN tunnel; however, traffic to IP addresses in 10.2.99.0/24 remains inside your VPC network. The tunnel's associated Cloud Router accepts an advertised route whose destination is 10.2.0.0/16; however, traffic to IP addresses in 10.2.99.0/24 remains inside your VPC network.

When tunnels are down

Dynamic (BGP) routing

When a Classic VPN tunnel that uses dynamic routing or a HA VPN tunnel goes down, the Cloud Router managing it automatically removes the learned custom dynamic routes within about 40 seconds. The custom dynamic routes can be re-added when the tunnel is re-established.

This process is fully automatic, but can still result in some packet loss during the time that it takes for Cloud Router to remove the affected routes.

Static routing

Google Cloud never considers a Cloud VPN tunnel that has not established an IKE security association (SA) as a valid next hop. Google Cloud considers a Cloud VPN tunnel operational if it has established an IKE SA. Note that an operational Cloud VPN tunnel only passes traffic if it is selected as the next hop according to the routing order. The next hop for a different route, with either a more specific destination or with a higher priority, might be used instead.

The following scenarios demonstrate expected behaviors:

  • If you have multiple custom static routes for the same destination, each having a different next hop Cloud VPN tunnel, Google Cloud will only consider those that have established IKE security associations (Phase 1). Tunnels that are down — that is, that do not have valid IKE security associations — are disregarded before considering route priority.

    For example, suppose that you have three custom static routes for the 192.168.168.0/24 destination: one with priority 10 whose next hop Cloud VPN tunnel is down, a second with priority 20 whose next hop tunnel is up, and a third with priority 30 whose next hop tunnel is also up. Google Cloud will send traffic to the second next hop, even though its route has a lower priority than the route whose next hop is down. If the first tunnel is re-established, then Google Cloud would consider it a valid next hop and use that route instead.

  • Traffic is dropped if all Cloud VPN tunnels, serving as next hops for custom static routes with the same destination, are down. This is true even if there is a custom static route for a broader destination with an operational next hop tunnel.

    For example, suppose that you have a custom static route for 192.168.168.0/24 whose next hop Cloud VPN tunnel is down (no valid IKE SA). Even if you have a route for 192.168.0.0/16 whose next hop is an operational Cloud VPN tunnel, traffic to 192.168.168.0/24 is dropped. This is because Google Cloud always selects routes with the most specific destinations.

When traffic switches from one next hop Cloud VPN tunnel to another, you should still expect packet loss. In-flight packets might be dropped and need to be re-transmitted.

ECMP over tunnels

For both dynamic and static routing, if more than one custom route exists for the same destination, having the same priority, and an active (IKE SA established) next hop tunnel, Google Cloud distributes packets among the tunnels using ECMP.

This balancing method is based on a hash, so all packets from the same flow use the same tunnel as long as that tunnel is up.

To test ECMP behavior, use iperf3 to send multiple simultaneous TCP streams, ideally from multiple Google Cloud VMs, through Cloud VPN tunnels. Do not perform tests using ICMP (ping) if you need to validate ECMP behavior. A "ping test" from one VM instance isn't sufficient to test ECMP-based egress through Cloud VPN tunnels, since only one tunnel is selected when you have a fixed source and destination. ICMP has no concept of ports and is a fixed protocol, so the hash is only calculated from two pieces of information: the source and destination addresses (a two-tuple hash).

What's next