Establishing 99.99% availability for Dedicated Interconnect

The following configuration uses Dedicated Interconnect to achieve 99.99% availability. Google recommends this configuration for production-level applications, such as mission-critical operations that have a low tolerance for downtime.

For Cloud Interconnect terminology used on this page, see Key terms.

Requirements

The following resources and settings are required to achieve 99.99% availability:

  • At least four Interconnect connections, two connections in one metropolitan area (metro) and two connections in another metro. Interconnect connections that are in the same metro must be placed in different edge availability domains (metro availability zones).

  • At least four Cloud Routers, two placed in each Google Cloud region. Even if you only have virtual machine (VM) instances in a single region, you must use two regions. If a region-wide issue occurs, Google Cloud can reroute traffic through the other region to your VMs. Each Cloud Router must be attached to a different Interconnect connection (four different VLAN attachments).

  • The dynamic routing mode for the Virtual Private Cloud (VPC) network must be global. With global dynamic routing, Cloud Router can advertise all subnets and propagate learned routes to all subnets regardless of the subnet's region.

  • Depending on your availability requirements, you can have two routers in your on-premises network, each router connecting to one of the Google Cloud regions.

Example topology

The following example topology shows four Interconnect connections in two different metros and different edge availability domains: lga-zone1-16, lga-zone2-1422, iad-zone1-1, and iad-zone2-1. The Cloud Routers (one for each connection) are in the vpc1 network in the us-central1 and us-east1 regions. Each router has its own Border Gateway Protocol (BGP) session.

Redundant connections for 99.99% availability (click to enlarge).
Redundant connections for 99.99% availability (click to enlarge)

Configuring the preferred path

With global dynamic routing, Cloud Router advertises routes to all subnets in the VPC network. Cloud Router advertises routes to remote subnets (subnets outside of the Cloud Router's region) with a lower priority compared to local subnets (subnets that are in the Cloud Router's region). For more information, see Dynamic routing mode in the Cloud Router documentation.

In the example topology, ingress traffic to the VPC network prefers to go through the us-central1 region because those routes have a higher priority compared to routes from the us-east1 region. Traffic traverses through us-east1 only if the connections in us-central1 fail.

You can change route priorities when you configure a Cloud Router's BGP session. The BGP session has an option for an advertised route priority (a MED value). A lower value means a higher priority. You can, for example, configure the BGP sessions in us-east1 with a higher priority, making that region the preferred path. For more information, see Route metrics in the Cloud Router documentation.

Creating a 99.99% topology

The following sections walk through the steps for creating this topology. To create a topology that's tailored for you, replace the sample input values with your own. For example, replace the vpc1 network name with your VPC network name.

Change VPC network's dynamic routing mode

For the vpc1 network, change its dynamic routing mode to global.

Console

  1. In the Google Cloud Console, go to the VPC networks page.

    Go to VPC networks

  2. Select the vpc1 network, and then click Edit.

  3. Under Dynamic routing mode, select Global.

  4. Click Save.

gcloud

gcloud compute networks update vpc1\
   --bgp-routing-mode global

Order Interconnect connections

Order duplicate connections by using the Cloud Console or the gcloud command-line tool.

Console

  1. In the Cloud Console, go to the Interconnect Hybrid Connectivity page.

    Go to Interconnect Hybrid Connectivity

  2. Click Get started.

  3. Select Dedicated Interconnect, and then click Continue.

  4. Select Order new Dedicated Interconnect, and then click Continue.

  5. Specify the details for the Interconnect connection:

    • Name: Enter int-lga1.
    • Description: Enter Example Interconnect connection in LGA ZONE1.
    • Location: Choose Digital Realty | Telx New York (111 8th Ave), which is the PeeringDB name for the lga-zone1-16 location.
    • Capacity: Use the default capacity (10 Gb/s).
  6. Click Next.

  7. Specify the details for the redundant Interconnect connection:

    • Name: Enter int-lga2.
    • Description: Enter Example Interconnect connection in LGA ZONE2.
    • Location: Choose zColo New York - 60 Hudson St, which is the PeeringDB name for the lga-zone2-1422 location.
    • Capacity: Use the default capacity (10 Gb/s).
  8. Click Next.

  9. Specify your contact information:

    • Company name: Enter The Customer. This name is used in the LOA-CFA as the party authorized to request an Interconnect connection.
    • Technical contact: Enter customer@customer.com. You don't need to enter your own address; you are included in all notifications.
  10. Click Next.

  11. Review your order. Check that your Interconnect connections details and contact information are correct. If everything is correct, click Place order. If not, go back and edit the connection details.

  12. On the order confirmation page, review the next steps, and then click Done.

  13. Order two more Interconnect connections (int-iad1 and int-iad2) in the location Equinix Ashburn (DC1-DC11), which is the PeeringDB name for the iad-zone1-1 and iad-zone2-1 locations. Because the Ashburn facility has two zones, the Cloud Console automatically selects the same facility but a different zone for the redundant connection.

gcloud

  • Order four Interconnect connections with the names int-lga1, int-lga2, int-iad1, and int-iad2:

    gcloud compute interconnects create int-lga1 \
      --customer-name "The Customer" \
      --description "Example Interconnect connection in LGA ZONE1" \
      --interconnect-type DEDICATED \
      --link-type ETHERNET_10G_LR \
      --location lga-zone1-16 \
      --requested-link-count 1 \
      --noc-contact-email customer@customer.com
    gcloud compute interconnects create int-lga2 \
      --customer-name "The Customer" \
      --description "Example Interconnect connection in LGA ZONE2" \
      --interconnect-type DEDICATED \
      --link-type ETHERNET_10G_LR \
      --location lga-zone2-1422 \
      --requested-link-count 1 \
      --noc-contact-email customer@customer.com
    gcloud compute interconnects create int-iad1 \
      --customer-name "The Customer" \
      --description "Example Interconnect connection in IAD ZONE1" \
      --interconnect-type DEDICATED \
      --link-type ETHERNET_10G_LR \
      --location iad-zone1-1 \
      --requested-link-count 1 \
      --noc-contact-email customer@customer.com
    gcloud compute interconnects create int-iad2 \
      --customer-name "The Customer" \
      --description "Example Interconnect connection in IAD ZONE2" \
      --interconnect-type DEDICATED \
      --link-type ETHERNET_10G_LR \
      --location iad-zone2-1 \
      --requested-link-count 1 \
      --noc-contact-email customer@customer.com

After you order Interconnect connections, Google emails you a confirmation and allocates ports for you. When the allocation is complete, Google generates LOA-CFAs for your connections and emails them to you. All the automated emails are sent to the NOC contact and the person who ordered the connections.

You are able to use the Interconnect connections only after your connections have been provisioned and tested. For more information about the provisioning process, see the Provisioning overview.

Create Cloud Routers

Create four Cloud Routers, one for each Interconnect connection.

Console

  1. In the Cloud Console, go to the Create a Cloud Router page.

    Go to Create a Cloud Router

  2. Specify the Cloud Router details:

    • Name: Enter rtr-cent1 for the name of the first Cloud Router.
    • Network: Select vpc1, which is the name of the network where the on-premises network extends to.
    • Region: Select us-central1, which is the region where the Cloud Router is configured, as shown in the example topology.
    • Google ASN: Enter 64513, which is the ASN for the example Cloud Router.
  3. Under Routes, select Advertise all subnets visible to the Cloud Router (default), and then click Create.

  4. Follow the same steps to create another Cloud Router, but use the name rtr-cent2.

  5. Create two more Cloud Routers with the same attributes in the us-east1 region with the names rtr-east1 and rtr-east2.

gcloud

  1. Create two Cloud Routers in the vpc1 network in the us-central1 region; use ASN 64513 for both Cloud Routers:

    gcloud compute routers create rtr-cent1 \
       --asn 64513 \
       --network vpc1 \
       --region us-central1
    gcloud compute routers create rtr-cent2 \
       --asn 64513 \
       --network vpc1 \
       --region us-central1
  2. Create two Cloud Routers in the vpc1 network in the us-east1 region; use ASN 64513 for both Cloud Routers:

    gcloud compute routers create rtr-east1 \
       --asn 64513 \
       --network vpc1 \
       --region us-east1
    gcloud compute routers create rtr-east2 \
       --asn 64513 \
       --network vpc1 \
       --region us-east1

Create VLAN attachments

After your Interconnect connections are ready to use (in the ACTIVE state), create VLAN attachments to connect the Interconnect connections with Cloud Routers. Create four attachments, one for each connection and Cloud Router pairing. The connections are described in the following list:

  • rtr-cent1 is connected to int-lga1
  • rtr-cent2 is connected to int-lga2
  • rtr-east1 is connected to int-iad1
  • rtr-east2 is connected to int-iad2

Console

  1. In the Cloud Console, go to the VLAN attachments tab.

    Go to VLAN attachments

  2. Click Add VLAN attachment.

  3. Select Dedicated Interconnect, and then click Continue.

  4. Select In this project.

  5. Select the int-lga1 Interconnect connection, and then click Continue.

  6. Select Add VLAN attachment:

    • Name: Enter attachment-lga1-central1.
    • Router: Select the rtr-cent1 Cloud Router.
  7. Click Create. The attachment takes a few moments to create.

  8. For the attachment, to add a BGP session to rtr-cent1, click Configure. The VLAN attachment has already populated the Google and Peer BGP IP addresses.

  9. Record the IP addresses, ASNs, and VLAN tag for each attachment. This information is required to configure the on-premises router.

  10. To add the BGP session, click Save configuration. The BGP sessions are inactive until you configure BGP on your on-premises router.

  11. Create three more attachments for the three other Interconnect connection-Cloud Router pairings. Record the information for each attachment.

gcloud

  1. Create four attachments:

    gcloud compute interconnects attachments create attachment-lga1-central1 \
       --interconnect int-lga1 \
       --router rtr-cent1 \
       --region us-central1
    gcloud compute interconnects attachments create attachment-lga2-central1 \
       --interconnect int-lga2 \
       --router rtr-cent2 \
       --region us-central1
    gcloud compute interconnects attachments create attachment-iad1-east1 \
       --interconnect int-iad1 \
       --router rtr-east1 \
       --region us-east1
    gcloud compute interconnects attachments create attachment-iad2-east2 \
       --interconnect int-iad2 \
       --router rtr-east2 \
       --region us-east1
  2. Describe the attachment-lga1-central1 attachment to retrieve the resources that it allocated, such as the VLAN ID and BGP peering addresses; use these values to configure the Cloud Router and on-premises router:

    gcloud compute interconnects attachments describe attachment-lga1-central1 \
       --region us-central1

    The previous command outputs the following information:

    cloudRouterIpAddress: 169.254.58.49/29
    creationTimestamp: '2017-08-15T08:34:11.137-07:00'
    customerRouterIpAddress: 169.254.58.50/29
    id: '5630382895290821276'
    interconnect:
    https://www.googleapis.com/compute/v1/projects/customer-project/global/interconnects/int-lga1
    kind: compute#interconnectAttachment
    name: attachment-lga1-central1
    operationalStatus: ACTIVE
    privateInterconnectInfo:
      tag8021q: 1000
    region: https://www.googleapis.com/compute/v1/projects/customer-project/regions/us-central1
    router: https://www.googleapis.com/compute/v1/projects/customer-project/regions/us-central1/routers/router1
    selfLink: https://www.googleapis.com/compute/v1/projects/customer-project/regions/us-central1/interconnectAttachments/attachment-lga1-central1
      
  3. Record the following values:

    • tag8021q: Specifies the VLAN ID, which is 1000. The VLAN ID identifies traffic that goes across this attachment. Use this value to configure a tagged VLAN subinterface on the on-premises router.
    • cloudRouterIpAddress: The Cloud Router IP address, which is 169.254.58.49/29. Assign this address to a Cloud Router interface, and specify this address as the BGP neighbor on the on-premises router.
    • customerRouterIpAddress: The customer router IP address, which is 169.254.58.50/29. On the Cloud Router, specify this address as the BGP peer address on the interface that has the Cloud Router IP address assigned to it. Assign this address to the VLAN subinterface on the on-premises router.
  4. On the rtr-cent1 Cloud Router, add an interface for the attachment-lga1-central1 attachment:

    gcloud compute routers add-interface rtr-cent1 \
       --interface-name rtr-cent1-1 \
       --interconnect-attachment attachment-lga1-central1 \
       --ip-address 169.254.58.49 \
       --mask-length 29 \
       --region us-central1
  5. Add a BGP peer to the interface; for this example, the on-premises ASN is 12345:

    gcloud compute routers add-bgp-peer rtr-cent1 \
       --interface rtr-cent1-1 \
       --peer-name central1 \
       --peer-ip-address 169.254.58.50 \
       --peer-asn 12345 \
       --advertised-route-priority 100 \
       --region us-central1
  6. Describe the other attachments, and then similarly configure a new BGP session on each of the Cloud Routers.

Configure on-premises routers

To configure the on-premises routers, use the information from the VLAN attachments. The following example is an excerpt from a Juniper router configuration. Use the sample as a starting point for configuring your own routers. For more information, see Configuring on-premises routers.

interface ae20 {
  flexible-vlan-tagging;
  native-vlan-id 1;
  aggregated-ether-options {
    lacp {
      active;
     }
   }
  unit 1000 {
    vlan-id 1000;
    family inet {
      address 169.254.58.50/29;
    }
  }
}

protocols bgp {
  group google {
    type external;
    multihop {
      ttl 4;
    }
    hold-time 60;
    peer-as 64513;
    local-as 12345;
    local-address 169.254.58.50;
    neighbor 169.254.58.49 {
      ...
    }
  }
}

What's next