This document is intended for network administrators, solutions architects, and operations professionals who run centralized network appliances on Google Cloud. Knowledge of Compute Engine and Virtual Private Cloud (VPC) networking in Google Cloud is required.
Corporate environments often need to route traffic to the internet, to an on-premises network, to other clouds, or even to other parts of the same cloud environment through virtualized, VM-based appliances that monitor, transform, or block traffic.
This document reviews several design options that segment VPC networks and interconnect them with a centralized, virtualized network appliance. All communication between VPC networks, on-premises networks, and the internet is routed through the centralized device. You can deploy a group of redundant appliances to get a high availability configuration. If you don't require high availability, you can deploy a single network appliance.
Routing traffic through virtualized appliances can be challenging. On Google Cloud, for example, you can replace the route pointing to the internet and redirect traffic to a set of virtualized appliances, but it's not possible to change the routing behavior between subnetworks within a VPC network. Routing between subnetworks is automatic and those routes can't be deleted or overridden.
Also, due to the crucial role of virtualized appliances, they must be deployed in highly available configurations. This is challenging because you need to ensure that all traffic gets routed through one of the virtual appliances and that failover between these appliances is automatic.
The following diagram shows a typical use case of several design options that feature a centralized, virtualized network appliance.
The preceding diagram shows the communication paths between segmented VPC networks, on-premises networks, and the internet, and how they are routed through the centralized, virtualized network appliance.
Main use cases for this architecture
You can use this architecture for multiple use cases that involve virtualized network appliances that traffic is routed through. Keep in mind the following considerations:
- Many appliances from different vendors can be found in the Google Cloud Marketplace.
- Some appliance vendors also offer customized Compute Engine images for Google Cloud on their website or support pages.
- You can create your own virtualized appliances by using open source networking software. You also can create your own images.
Vendors often provide their own reference architectures or deployment guides for running their virtualized appliances in a high availability configuration. For more information, see the vendor's website.
Reference architectures from the vendor might not encompass all the options presented in this document.
It's important to understand the advantages and disadvantages of each approach. Typical use cases for virtual appliances that traffic is routed through include the following:
- Next-generation firewalls (NGFWs). A centralized set of firewalls run as virtual machines that deliver features that aren't available if you use VPC firewall rules. This is the most common use case. Typical features of NGFW products include deep packet inspection (DPI) and firewalling on the application layer. Some NGFW firewalls also provide TLS/SSL traffic inspection, as well as other networking functions described in the use cases in the following bulleted items.
- Intrusion detection system/intrusion prevention system (IDS/IPS). A network-based IDS requires the visibility of all potentially malicious traffic. To prevent intrusions, traffic is usually blocked directly by an IPS in the traffic path.
- Transparent proxy. A transparent proxy is often used to monitor HTTP(S) traffic and to enforce restrictions on web access.
- Network address translation (NAT) gateway. A NAT gateway translates IP addresses and ports. This helps, for example, to avoid overlapping IP addresses. Google Cloud offers Cloud NAT as a managed service, but this service is available only for traffic going to the internet, not for on-premises traffic or for other VPC networks on Google Cloud.
- Web application firewall (WAF). A WAF blocks malicious HTTP traffic going to a web application. Google Cloud offers WAF functionality through Google Cloud Armor security policies. Exact functionality differs between WAF vendors, so it's important to determine exactly what you need.
Requirements for routing traffic through centralized virtual appliances differ based on the specific use case. However, the following requirements generally apply:
- All traffic between different network segments—for example, environments administered by different teams, with separate security requirements or between different modules of an application—must pass through a centralized virtual appliance.
- All traffic to or from the internet to secure environments must pass through a centralized virtual appliance.
- All traffic to or from on-premises systems connected to Google Cloud through Cloud VPN, Dedicated Interconnect, or Partner Interconnect must pass through a centralized virtual appliance.
- Multiple centralized virtual appliances are set up in a high availability configuration in either an active/active or active/passive configuration. Failover occurs automatically if any software or hardware issues arise on one of the virtualized appliances.
- All traffic is routed statefully, so that one traffic flow between two virtual machines is always passing through the same virtual appliance.
- The throughput of the system of centralized virtual appliances is
scalable by one of the two following options:
- Scaling the virtual appliances vertically: increasing the number of cores or the amount of memory on each virtual machine.
- Scaling the virtual appliances horizontally: increasing the number of virtual appliances used to distribute the traffic.
There are multiple options for achieving high availability between virtual appliances. There are also multiple options for attaching network segments to the virtual appliances.
You can combine any option for high availability with any option for attaching network segments. A common option described later in this document is an architecture that uses VPC Network Peering and the internal TCP/UDP load balancer as the next hop.
Choosing a high availability option
You can build an architecture to achieve high availability for the central appliance by using multiple instances in the following two ways:
Use an internal TCP/UDP load balancer as the next hop. In this approach, you place multiple virtual appliances in a managed instance group behind an internal TCP/UDP load balancer. This load balancer is used as the next hop for a default route that replaces the default route in the VPC networks connected to the appliances. You can use appliances either in an active/active configuration, which is the standard configuration, or in an active/passive configuration, by employing failover for Internal TCP/UDP Load Balancing. Health checks direct traffic to healthy instances. If you want to re-create appliances automatically on failure, you can use autohealing. If your device uses multiple network interfaces, an internal TCP/UDP load balancer as the next hop can have backends on any network interface.
The following diagram shows the topology of using an internal TCP/UDP load balancer as the next hop.
The preceding diagram shows a managed instance group in a VPC network, including multiple virtual appliances that are behind an internal TCP/UDP load balancer, which acts as the next hop.
Use routing. In this approach, Google Cloud routes direct the traffic to the virtual appliances from the connected VPC networks. You can use appliances either in an active/passive configuration by using different priority routes, or in an active/active configuration when routes are configured with the same priority, in which case you use equal-cost multi-path (ECMP) routing to distribute traffic. You can use autohealing to help ensure that appliances restart when they're unhealthy.
The following diagram shows the topology of using routing.
The preceding diagram shows a managed instance group in a VPC network with Google Cloud routes directing the traffic to the virtual appliances from the connected VPC network.
Using an internal TCP/UDP load balancer as the next hop has the following advantages over using Google Cloud routing for high availability:
- Health checks decide where the traffic is forwarded, helping to ensure that traffic is forwarded only to healthy instances. You can expose health checks so that the local instance is verified or an end-to-end path is verified.
- You can fine-tune the health check timers for faster failover. This approach provides the fastest failover times in case of unhealthy instances.
- With symmetric hashing, you can ensure that return traffic is routed to the same virtual machine as the outgoing traffic.
Using Google Cloud routing has the following advantage:
- Consistent, symmetric hashing ensures that packets for a given connection are processed by the same backend VM, for both requests and responses, depending on the protocol and session affinity selection. The protocol and session affinity selection determine whether or not connection tracking is used. Note that some use cases might benefit from having a source NAT (SNAT) configuration in the backend instances; for example, if a response connection from one network to another should match a separate request connection in the same direction between the same two networks.
Using Google Cloud routing has the following disadvantages:
- Health checks delete and re-create unhealthy instances from the instance pool. However, traffic flow doesn't change immediately in response to failed health checks because it takes time to delete unhealthy instances and create new, healthy ones. This leads to slower failover times.
- If you set health check timers to avoid unnecessary deletion and re-creation of instances, this will cause slower failover times.
Whether you use internal TCP/UDP load balancer as the next hop or Google Cloud routing, all protocol traffic is supported. For detailed information about this support, see Processing of TCP, UDP, and other protocol traffic.
Simiarly, both solutions support limiting routes to certain network tags. For example, VM instances can be segmented by region to use different sets of appliances.
Choosing an option for attaching network segments
Because routing between subnetworks can't be overridden, network segments are implemented by using separate VPC networks. Traffic between these VPC networks, to an on-premises environment, and to the internet, needs to be routed through the centralized appliances. Note that all of these network segments can be either standalone VPC networks or Shared VPC networks.
There are several options for attaching network segments, as follows:
Multiple network interfaces. The simplest way to connect multiple VPC networks through a virtual appliance is by using multiple network interfaces, with each interface connecting to one of the VPC networks. Internet and on-premises connectivity is provided over one or two separate network interfaces. With many NGFW products, internet connectivity is connected through an interface marked as untrusted in the NGFW software.
The following diagram shows this topology.
The preceding diagram shows multiple VPC networks connecting through a virtual appliance by using multiple network interfaces. Each interface connects to one of the VPC networks. The diagram also shows internet and on-premises connections over separate network interfaces, including an internet connection through an untrusted interface.
VPC Network Peering. This topology uses the hub-and-spoke concept in conjunction with VPC Network Peering. The network segments are implemented as a set of spoke VPC networks that are peered by using VPC Network Peering with a hub VPC network, in which traffic is routed through the centralized pool of virtualized appliances. The default route or routes pointing to the internal TCP/UDP load balancer as the next hop, or to the virtual appliances directly, are exported as custom routes over the VPC Network Peering. Internet and on-premises connectivity is provided over one or two separate network interfaces.
The following diagram shows this topology.
The preceding diagram shows a centralized pool of virtualized appliances with multiple network interfaces attached to multiple hub VPC networks. Each hub VPC network is attached to multiple VPC networks through VPC Network Peering. Traffic between any two VPC networks is routed through the centralized pool of virtualized appliances.
Combining multiple network interfaces and VPC Network Peering. This approach combines the two preceding options for additional scalability. Multiple network interfaces are attached to multiple hub VPC networks, each of which is connected to multiple spoke VPC networks through VPC Network Peering. For each hub-and-spoke VPC network, you use the same approach as described in the VPC Network Peering case.
The following diagram shows this topology.
The preceding diagram shows a hub VPC network attached to multiple VPC networks through VPC Network Peering. Traffic is routed through the centralized pool of virtualized appliances with one network interface in the hub network.
The following decision tree can help you decide on the best approach for attaching network segments.
Using multiple network interfaces—the first approach presented in the preceding cases—is the easiest approach to implement.
However, using multiple network interfaces has the following disadvantages:
- You're limited to 8 network interfaces per virtual machine instance. If you use one network interface for internet connectivity and one for on-premises connectivity, you can connect only 6 network segments on Google Cloud. Some appliances require an additional management interface, limiting you to 5 network segments.
- You can't add or remove network interfaces after an instance has been created.
Using VPC Network Peering has the following advantages:
- You can scale up to the limit of VPC Network Peering connections from a single VPC network. Contact the Google Cloud sales team if you have questions about increasing this limit.
- It's easy to add or remove VPC networks from this architecture by setting up VPC Network Peering.
However, using VPC Network Peering has the following disadvantages:
- This approach is slightly more complex than the approach that uses multiple network interfaces.
- The number of VPCs that can be connected is still limited as compared to the combined approach.
Using the combined approach—multiple network interfaces and VPC Network Peering—has the following advantage:
- It is the most scalable approach because the limit is a product of the limit of network interfaces and the limit of VPC peering connections. For example, you can connect 6 hub VPC networks to separate network interfaces, with each interface having 25 spoke VPC networks connected. This gives you a limit of 150 VPC networks exchanging traffic through one set of virtual appliances.
However, this approach has the following disadvantage:
- It is the most complex approach to implement.
Following are two example architectures. The first example architecture demonstrates how to use the internal TCP/UDP load balancer for high availability, combined with VPC Network Peering for attaching network segments. The second example architecture, a special use case, shows you how to route traffic from a single shared VPC network on Google Cloud towards an on-premises environment and to the internet.
Example architecture using VPC Network Peering and internal TCP/UDP load balancer as next hop
This architecture is a typical use case for enterprise environments, using the internal TCP/UDP load balancer for high availability, combined with VPC Network Peering for attaching network segments.
The following diagram shows the topology of this architecture.
The preceding diagram shows a hub VPC network and multiple spoke VPC networks that are peered with the hub VPC network by using VPC Network Peering. The hub VPC network has 2 instances of a virtual appliance in a managed instance group behind an internal TCP/UDP load balancer. A static default route points to the internal TCP/UDP load balancer as the next hop. This static default route is exported over the VPC Network Peering by using custom routes. The default route to the internet gateway in the spoke VPC networks is deleted.
You can meet the requirements in the following ways:
- Connectivity between spokes is automatically routed through the firewall because VPC Network Peering is non-transitive, and therefore, spoke VPC networks can't exchange data with each other directly. You can configure the virtual appliances to set the conditions and policies under which the spokes can exchange traffic.
- Connectivity to the internet is possible only through the virtual appliances because the default route to the internet gateway has been removed from the spoke VPC networks. The virtual appliances have a default route through a separate network interface, which can be marked as untrusted in the case of an NGFW.
- Connectivity to on-premises networks is achieved through a connectivity VPC network connected to the virtual appliance on a separate network interface. A Cloud VPN or Cloud Interconnect connection terminates in this connectivity VPC network.
- High availability is achieved through customized health checks on the internal load balancer. You can configure appliances as active/passive by using failover for Internal TCP/UDP Load Balancing. This helps to ensure that traffic always flows through a single virtual appliance.
You can use this example architecture if communication between the different VPC networks is TCP/UDP or other protocol traffic. It's scalable up to the limit of VPC Network Peering connections per VPC network.
For a sample implementation of this architecture with a NAT gateway as the virtual appliance, see Deploying centralized VM-based appliances using internal TCP/UDP load balancer as the next hop. You can use the same pattern to deploy other virtual appliances, as described in the preceding use cases section.
Special use case with one shared VPC network on Google Cloud
In one special use, no traffic between different VPC networks needs to pass through the virtual appliances. Instead, only traffic from a single shared VPC network is routed towards an on-premises environment and to the internet. You can implement this use case by using any of the high availability options described earlier in this document, combined with one network interface each, for connectivity to the shared VPC network, on premises, and Google Cloud. If you want visibility into traffic between subnets without running it through centralized appliances, Packet Mirroring can help.
The following diagram shows this topology.
The preceding diagram shows traffic from a single shared VPC network routed towards an on-premises network and to the internet through a pool of virtual appliances.
Implementation of an architecture with virtual appliances
To implement an architecture that uses virtual appliances, choose an option for high availability and combine it with an option for attaching network segments.
For the following combinations of options, there are implementation guides available:
- For information about using VPC Network Peering between segments and internal TCP/UDP load balancer as a high availability option, see Deploying centralized VM-based appliances using internal TCP/UDP load balancer as the next hop.
- For information about using VPC Network Peering between segments and using routing as a high availability option, see Deploying centralized VM-based appliances using VPC network peering.
The preceding tutorials use NAT gateways as a use case for the appliance, but you can adapt the gateway by using any of the other use cases.
If you want to deploy an appliance from a Google Cloud partner—for example, from the Cloud Marketplace—contact your appliance vendor, or search the vendor's website for guidelines about how to deploy their appliances on Google Cloud.
- Best practices and reference architectures for VPC design.
- Hybrid and multi-cloud network topologies.
- Best practices for enterprise organizations.
- VPC documentation.
- Internal TCP/UDP Load Balancing documentation.
- Explore reference architectures, diagrams, tutorials, and best practices about Google Cloud. Take a look at our Cloud Architecture Center.