This document helps cloud architects and operations professionals decide how to connect Google Cloud with other cloud service providers (CSP) such as Amazon Web Services (AWS) and Microsoft Azure. In a multi-cloud design, these connections allow data transfers between Virtual Private Cloud (VPC) networks. This document also provides information on how to implement the option that you choose.
Many organizations operate on multiple cloud platforms, either as a temporary measure during a migration or because the organization has a long-term multi-cloud strategy.
For data exchange between Google Cloud and other CSPs, there are multiple options discussed in this document:
- Transfer of data through public IP addresses over the internet.
- Using a managed VPN service between Google Cloud and the other CSP.
- Using Partner Interconnect on Google Cloud to connect to a partner that can directly connect with the other CSP.
- Using Dedicated Interconnect on Google Cloud to transfer data to the other CSP through a common point of presence (POP).
These options differ in terms of transfer speed, latency, reliability, service level agreements (SLAs), complexity, and costs. This document describes each option and its advantages and disadvantages in detail and ends with a comparison of all of the options.
This document covers data transfers between virtual machines residing in Google Cloud and other CSP's VPCs. For data stored in other Google Cloud products such as Cloud Storage and BigQuery, see the section covering these products.
This document can act as a guide to evaluate the options to transfer data between Google Cloud and one or more other CSPs, based on your requirements and capabilities.
The concepts in this document apply in multiple cases:
- When you are planning to transfer large amounts of data for a short period of time, for example, for a data migration project.
- If you run continuous data transfers between multiple cloud providers, for example, because your compute workloads run on another CSP while your big data workloads use Google Cloud.
Before you choose how to connect your cloud environments, it's important that you look at which regions you choose in each environment and that you have a strategy for transferring data that doesn't reside in VPC environments.
Choose cloud regions
If both Google Cloud and other CSP's resources are in regions that are geographically close to each other, there is a cost and latency advantage for data transfers between cloud providers.
The following diagram illustrates the flow of data between Google Cloud and other CSPs.
Regardless of the method of transfer, data flows in the following directions:
- From the Google Cloud region where resources are hosted to Google's edge POP.
- Through a third-party facility between Google Cloud and the other CSP.
- From the other CSP's edge POP to the region where resources are located inside the other CSP's network.
- Along the same paths mentioned previously in the list, but in the opposite direction. For example, from Google Cloud to the other CSP as well as from the other CSP to Google Cloud.
The end-to-end path determines the latency of the data transfer. For some solutions, the network costs also increase when the edge POPs of both providers aren't in one metropolitan area. Details are listed in the following pricing section of each solution.
Make sure you choose suitable regions in each cloud that can host your intended workloads. Visit the Locations page for Google Cloud and a similar pages for other CSPs, such as the AWS region table or Azure products by region. For general help in selecting one or multiple locations in Google Cloud, review Best practices for Compute Engine region selection.
Transfer of data in Cloud Storage and BigQuery
If you want to transfer another CSP's object storage, database, data warehouse, or other products, check their documentation to see if data can pass through their interconnect or managed VPN product. Otherwise, you might be able to pass this data through proxy virtual machines that you set up in the respective CSP's environment to make it pass through the connection you want.
Transfer through external IP addresses over the internet
The easiest way to transfer data between Google Cloud and another CSP is to use the internet and transfer the data by using external IP addresses.
The following diagram illustrates the elements for this solution.
Between Google's network edge and the other CSP's network edge, data passes through the internet or uses a direct peering between Google Cloud and the other CSP. Data can only pass between resources that have public IP addresses assigned.
How Google connects to other networks
Google's edge point-of-presence facilities (POP) are where Google's network connects to other networks that collectively form the internet. Google is present on over 90 internet exchanges and at over 140 interconnection facilities around the world.
On the internet, every network is assigned an autonomous system number (ASN) that encompasses the network's internal network infrastructure and routes. Google's primary ASN is 15169.
There are two primary ways a network can send or receive traffic to or from Google:
- Buy internet service from an internet service provider (ISP) that already has connectivity to Google (AS15169). This option is generally referred to as IP transit and is similar to what consumers and businesses purchase from access providers at their homes and businesses.
- Connect directly to Google (AS15169). This option, called peering, lets a network directly send and receive traffic to Google (AS15169) without using a third-party network. At scale, peering is generally preferred over transit because network operators have more control over how and where traffic is exchanged, allowing for optimization of performance and cost. Peering is a voluntary system; when choosing to peer, network operators decide jointly which facilities to connect in, how much bandwidth to provision, how to split infrastructure costs, and any other details required to set up the connections. AS15169 has an open peering policy, which means as long as a network meets the technical requirements, Google will peer with them.
Peering is a private, mutually beneficial agreement between two independent networks, and as such, networks generally do not disclose publicly who they peer with at particular locations, how much bandwidth is available, etc. But due to scale and an open policy, Google is directly peered with almost all major ISPs and cloud services providers in multiple locations and across regions. The Google network team works directly with their counterparts at these networks to provide sufficient peering capacity to meet demand.
You can read more about how internet peering works at The Internet Peering Playbook.
In this setup, all virtual machines that transfer data between Google Cloud and other cloud service providers must have a public IP address. On one end, the firewall must be opened to allow a connection from the public IP address of the other cloud provider. No extra steps are necessary because the data exchange happens over the existing internet connectivity.
Transfer speed and latency
While there is no guaranteed speed and latency on the path through the internet, typically, major CSPs and Google exchange data directly in multiple locations around the world. Capacity is shared with other customers and services, but, often due to the direct connection between both providers, latency is similar or lower than other options.
We recommend that you test the latency and bandwidth between Google Cloud and the other CSPs in your chosen regions. You can do a quick benchmark by using tools such as iperf or netperf, or run a more complete custom benchmark based on your app. While conditions might change over time, the benchmark can provide an indication of the performance you can expect and if this solution fulfills your needs. If you require a dedicated bandwidth between both environments, you should choose another solution.
Note that products from different vendors may have performance characteristics that may not always align. For example, at the time of writing, AWS VPNs have a per tunnel throughput limit of 1.25 Gbps. Microsoft Azure VPN tunnels have a limit of 1 Gbps. GCP VPN tunnels can handle up to 3Gbps. So connecting a GCP VPN to a third party VPN may introduce bottlenecks that would have to be mitigated by scaling up the number of tunnels, using ECMP, or other strategies.
Traffic over the internet isn't encrypted and might pass through third-party internet service providers (ISPs), autonomous systems, and facilities. Therefore, you should encrypt sensitive traffic at the application layer or choose another solution.
Reliability and SLA
Google Cloud generally has multiple diverse paths for internet connectivity from regions, and there are direct peering connections with other major CSPs at multiple locations around the globe.
However, Google Cloud doesn't provide any SLAs for connectivity to other CSPs over the internet. While you should check SLAs for your other CSPs, they typically only refer to internet connectivity as a whole and not specific providers.
Providers may have different routing policies which can impact availability. For example, on it's peeringdb page, Amazon explains that many AWS regions announce only local routes, because AWS VPCs are regional only (GCP VPCs are global.) This means customers may be relying on links at a single peering location, as traffic leaving GCP can only use those links to reach the destination. This is fine under normal operation with traffic being exchanged in-region, but it is advisable for customers to architect for multi-region deployments to tolerate regional failures. This could include setting up additional gateways, HA VPNs, VPC peering, or other multi-region topologies in the third-party cloud.
Applications should also be built in such a way that they 'fail open' as Google SRE recommends in the SRE book. For example, if you build an application that relies on being able to reach a third-party service via Internet routing, make sure that the application still functions, or at least returns helpful error messages to the user in the event of connectivity issues.
When issues with Internet routing do occur, the Google network team will attempt to restore connectivity with the third-party. However, not all issues will be under Google's control. And recall that Internet peering is a voluntary system, not a paid service. So in some cases, repair may depend on a third-party (ISP and/or cloud provider) taking restorative action. Customers have the most influence over how operators respond to outages, so make sure you have support coverage with all providers and a plan to escalate issues should something go wrong. Also perform periodic BCP (business continuity process) drills to ensure the resilience of applications architected on multi-cloud.
For data transfers over the internet, normal internet egress rates apply for traffic leaving Google Cloud. In cases where latency isn't critical, using the Standard Network Service Tier provides a lower pricing tier.
The other CSPs have their own charges for data transfers. In many cases, they only charge for traffic egressing their network. Consult your CSP's price list for example for AWS, see EC2 data transfer charges and for Azure, see Bandwidth Pricing Details.
Managed VPN between cloud providers
You can use managed VPN services from both cloud providers to have an encrypted channel between VPCs in both cloud environments and to transfer data by using private IP addresses. This is an extension to the previous solution of transferring over the internet without requiring any hardware or partners.
The following diagram illustrates the elements for this solution.
Using this solution, data is encrypted on Google Cloud using Cloud VPN and a VPN solution for the other CSP. The data transfer between Google Cloud and the other CSP uses the internet like in the previous solution.
Google offers Cloud VPN as a managed VPN service for encrypted IPsec tunnels, which can be used on the Google end. Other CSPs offer their own managed VPN products, which you can use to build IPsec VPN tunnels between both environments. For example, AWS offers AWS Site-to-Site VPN and Azure offers Azure VPN gateway. You can connect your VPCs between the environments by using one or multiple VPN tunnels.
IP addresses between both environments shouldn't overlap because there is no Network address translation (NAT) functionality available. On Cloud Router, overlapping routes could be removed from being exchanged between environments, but then no communication between the services using these IP addresses is possible.
With Cloud Router in global dynamic routing mode, you can reach all regions in a global VPC network with a single VPN tunnel, but for other CSPs you might need one VPN tunnel per region. If you have multiple VPCs in a cloud environment that aren't peered, you have to connect all VPCs that need to communicate with each other independently by using a VPN.
Google Cloud offers interoperability guides, which have step-by-step instructions for setting up VPN tunnels to other major cloud providers:
- Alibaba Cloud VPN gateway without redundancy: supports static routes only.
- Alibaba Cloud VPN gateway with redundancy: supports static routes only.
- AWS: supports static routes or dynamic routing with Cloud Router.
- Automated Network Deployment: Building a VPN Between Google Cloud and AWS: supports automatically deploying VPN connections between Google Cloud and AWS using Terraform.
- Microsoft Azure: supports static routes only.
Transfer speed and latency
When using managed VPN tunnels, data still follows the same internet path as if data was directly exchanged by using public IP addresses. The latency observed should be similar to the previous option with only a small latency overhead for the VPN tunnel. The available bandwidth is limited by the maximum bandwidth per VPN tunnel on Google Cloud (1.5 Gbps), the other CSP, and the available bandwidth over the internet path, whichever is lowest.
For higher bandwidth, you can deploy multiple tunnels in parallel. For more information on how to deploy such a solution, see Building high-throughput VPNs.
You can test latency and bandwidth as described in the last section, but conditions might vary over time, and there are no guarantees on latency or bandwidth.
Reliability and SLA
Cloud VPN offers an SLA of 99.9%-99.99% depending on if Classic VPN or HA VPN is selected. Other CSPs sometimes offer SLAs on their managed VPN product, for example, AWS Site-to-Site VPN SLA and Azure SLA for VPN Gateway. However, these SLAs only cover the VPN gateways availability and don't include connectivity to other CSPs over the internet, so there is no SLA on the end-to-end solution.
To increase reliability, consider using multiple VPN gateways and tunnels on both Google Cloud and the other CSPs.
For the managed VPN service, charges apply. For Google Cloud, an hourly charge applies see Cloud VPN pricing. For other CSPs, consult their price lists, for example, see AWS Site-to-Site VPN connection pricing or Azure VPN Gateway pricing.
In addition to hourly pricing for the VPN service, you have to pay for data transferred through the VPN gateways. For Google Cloud and many CSPs, standard internet data transfer charges apply as detailed in the previous section. In many cases, data transfer charges exceed the fixed cost for this solution.
Partner Interconnect with multi-cloud enabled partners
Partner Interconnect lets you connect a Virtual Private Cloud to other CSP's VPCs through the network of select partners that offer direct multi-cloud solutions. You connect by deploying one or more virtual routing instances that take care of the necessary Border Gateway Protocol (BGP) setup.
The following diagram shows a redundant setup using two Partner Interconnect connections.
Routes are exchanged between Cloud Router and the gateway on the other CSP's side through a virtual routing instance that is managed by the partner providing the interconnect. Traffic flows through the partner network between Google Cloud and the other CSP.
This solution requires you to set up multiple components:
- On the Google Cloud side, you set up Partner Interconnect with an interconnect service provider that's serving your Google Cloud regions and offering multi-cloud connectivity to the other CSP.
- On the other CSP, you have to use their interconnect product to connect to the same partner. For example, on AWS you can use Direct Connect and on Azure you can use ExpressRoute.
- On the service provider partner side, you have to configure the virtual routing equipment providing the BGP sessions to Google Cloud and to the other CSP.
IP address space between both CSP environments cannot overlap because there is no NAT functionality available.
Google offers implementation guides with step-by-step instructions for some partners, for example, Connecting multi-cloud VPCs with Megaport Cloud Router.
Transfer speed and latency
This solution offers dedicated capacity between Google Cloud and other CSPs. Depending on the partner and the other CSP, the available attachment capacity might vary. On the Google Cloud side, Partner Interconnect is available with an attachment capacity between 50 Mbps and 10 Gbps.
Latency for this solution is the sum of the following:
- Latency between the region in which your resources are hosted on Google Cloud and the interconnect location where the partner connects to Google Cloud.
- Latency in the partners network to, from, and through the virtual routing instance towards the other CSP.
- Latency from the other CSP's edge location where the interconnect with the partner takes place to the region where the resources are hosted in the CSP.
For the lowest possible latency, the edge locations of Google Cloud and the other CSP should be in the same metropolitan area, along with the virtual routing instance. For example, you might have a low latency connection if both CSP's cloud regions as well as the edge POP and the virtual routing instance are located in the Ashburn, Virginia area.
While Google Cloud and many other CSPs offer no latency guarantees for traffic towards their network edge, because there is a dedicated path and capacity through the partner, typically the latency in this solution is less variable than the previous options.
Because traffic over Partner Interconnect isn't encrypted, the use of application layer encryption for sensitive traffic is recommended.
If all traffic over Partner Interconnect needs to be encrypted, virtual appliances available from Google Cloud partners in the Google Cloud Marketplace can provide solutions for encryption on the Google Cloud side. Cloud VPN cannot be used over Dedicated Interconnect or Partner Interconnect. Some other CSPs allow you to use their managed VPN service over an interconnect, for example, AWS Site-to-Site VPN can be used over AWS Direct Interconnect. Alternatively, you can also use a virtual appliance encrypting the traffic on the other CSPs end.
Reliability and SLA
Depending on the other CSP and the Partner Interconnect SLAs, this solution is the only option that lets you have an SLA for each part of the connectivity solution.
Consult the documentation or terms of the service provider for the Partner Interconnect for their SLA on the availability of the connectivity and virtual routing instance. For example, see the Megaport Global Services Agreement.
On the Google Cloud side, there is a monthly fee for each Partner Interconnect attachment, depending on the bandwidth. Traffic egressing through the Partner Interconnect gets charged at a lower rate than internet traffic. For more information, see the Partner Interconnect pricing page.
Consult the other CSP's pricing page for their interconnect product, for example AWS Direct Connect pricing or Azure ExpressRoute pricing. Typically, pricing also has a monthly charge for the interconnect and data transfer charges through the interconnect at a lower rate than over the internet.
The partner provides interconnect services charges according to their own pricing, which can be found on their website or by consulting their sales team for a quote. Typically, if all data transfers happen in the same metropolitan area, charges are much lower than if data has to travel a longer distance on a partner network.
When data is transferred regularly at sufficiently high volumes, depending on the other CSP's prices, this solution can sometimes offer the lowest total cost due to the discounted egress rates. Even when adding in monthly costs for the interconnect for Partner Interconnect, the other CSP, and the service provider partner, using this solution can provide significant savings. As partners and other CSP's prices can change without notice, make your own comparison using up-to-date quotes from all involved parties.
Dedicated Interconnect through a common POP
By using one or more physical routing devices at a common interconnect facility between Google Cloud and the other CSP, you can connect both cloud providers by using Dedicated Interconnect on the Google Cloud side and an equivalent product at the other CSP. The interconnect location isn't necessarily at the same location as the region in which your resources are located.
The following diagram shows a redundant setup using two Dedicated Interconnect connections:
Routes are exchanged between Cloud Router and the gateway on the other CSP's side through a physical router that you place in a common interconnect facility. Traffic flows through this router between Google Cloud and the other CSP.
This solution requires that you host and maintain physical routing devices at a colocation facility where Google and the CSP you want to connect to are present. From this routing device, you order two physical connections with the facility: one toward Google Cloud using Dedicated Interconnect, and one towards the other service provider using an equivalent product, for example, AWS Direct Connect or Azure ExpressRoute. On your routing device, you need to configure BGP to allow route exchanges between the two cloud environments.
Due to various factors, including incompatibilities during link setup as well as in BGP parameters, it isn't possible to directly connect the cloud environments in a physical facility without active routing equipment in-between.
Check the Colocation facility locations list from Google Cloud and your other CSP, for example AWS Direct Connect Locations or Azure ExpressRoute peering locations, to identify suitable locations for this setup.
Typically IP address space allocation between both providers' VPCs shouldn't overlap, unless your physical routing device is able to perform NAT between the environments or you restrict some route exchanges between the environments.
This solution is effective if you use the dedicated connectivity to also connect back to your on-premises environment, in addition to the connection to another CSP.
In other cases, this solution is complex because it requires you to own and maintain physical equipment and have a contract with a colocation facility. We only recommend this solution if at least one of the following is true:
- You already have equipment in a suitable facility for another purpose and an existing contract with the facility.
- You transfer large amounts of data regularly so this is a cost effective option or you have bandwidth requirements that partners cannot fulfill.
Transfer speed and latency
This solution offers dedicated capacity between Google Cloud and another CSP. On the Google Cloud side, Dedicated Interconnect is available by using one or multiple 10 or 100 Gbps physical connections.
The latency for this solution is a sum of the following:
- Latency between the region in which your resources are hosted on Google Cloud and the interconnect location where you interconnect with Google Cloud.
- Latency through the facility and your physical equipment, which is usually negligible when compared with any latency due to fiber length.
- Latency from the interconnect location through the other CSP's network to the region where the resources are hosted in the CSP.
While no latency guarantees are offered, this solution typically allows for the lowest latency and highest transfer speeds between Google Cloud and other cloud environments over private IP addresses.
Because traffic over Dedicated Interconnect isn't encrypted, we recommend using application layer encryption for sensitive traffic.
If all traffic needs to be encrypted, like with the previous solution, virtual appliances available from Google Cloud partners in the Cloud Marketplace can provide solutions for encrypting the traffic to the other CSP's environment.
Reliability and SLA
The colocation facility or hardware vendor for the physical routing equipment might also offer SLAs on their services.
On the Google Cloud side, there is a monthly fee for each Dedicated Interconnect port as well as for each VLAN attachment connecting to a VPC environment. Traffic egressing through Dedicated Interconnect is charged at a lower rate than internet traffic. For more information, see the Dedicated Interconnect pricing page.
Consult the other CSP's pricing page for their interconnect product, for example, AWS Direct Connect pricing or Azure ExpressRoute pricing. Typically, pricing also has a monthly charge for the interconnect and data transfer charges through the interconnect at a lower rate than over the internet.
In addition, you need to factor in charges for the colocation facility services providing space, electrical power, and physical connections towards both cloud environments as well as any cost and ongoing service contracts with the vendor for the physical routing equipment. If the connection between both CSPs cannot happen in the same facility and you need to procure connectivity between facilities, pricing might be much higher for these services.
Comparison of options
The presented options vary in speed, availability, complexity, security, and pricing. You should evaluate all options thoroughly according to your needs.
The following diagram guides you through the process of choosing one of the solutions mentioned in this document through a simple flow chart.
Typically, we can recommend the following choices:
- For many scenarios where data is exchanged occasionally or at low volume and transfers aren't critical, transferring data over the internet is the simplest option and still provides relatively low latency and high bandwidth.
- If encryption or transfer of data using private IP addresses is required, consider using Cloud VPN and a managed VPN service on the other CSP's side.
- If you transfer larger amounts of data, using Partner Interconnect with a multi-cloud enabled partner provides multiple benefits: Dedicated capacity, lower cost for data transfers, and depending on the topology, an SLA for each part of the solution.
- If you connect your on-premises equipment to multiple clouds, using Dedicated Interconnect through a common POP is a common option. It comes with additional complexity of managing your own hardware and relationships with a colocation facility. Unless you already have the infrastructure in place, this solution is reserved to cases where typical data transfer rates are 10 Gbps or more.