Private connectivity

Overview

Private connectivity is a connection between your Virtual Private Cloud (VPC) network and Datastream's private network, enabling Datastream to communicate with resources by using internal IP addresses. Using private connectivity establishes a dedicated connection on the Datastream network, meaning no other customers can share it.

You can use private connectivity to connect Datastream to any source. However, only VPC networks that are peered directly can communicate with each other.

Transitive peering isn't supported. If the network that Datastream is peered with isn't the network where your source is hosted, then a proxy is required.

  • If your source is Cloud SQL, then you need a Cloud SQL Auth proxy.
  • If your source is hosted either in another VPC or outside of the Google network, and the VPC network to which Datastream is peered doesn't have direct connectivity to the VPC or network that hosts the source, then you need a reverse proxy.

In this page, you learn how to use proxies to establish private connectivity between Datastream and Cloud SQL, or between Datastream and sources that are hosted either in another VPC or outside of the Google network.

Why do you need a Cloud SQL Auth proxy?

When you configure a Cloud SQL instance to use private IP addresses, you use a VPC peering connection between your VPC network and the VPC network of the underlying Google services where your Cloud SQL instance resides.

Because Datastream's network can't be peered directly with Cloud SQL's private services network, and because VPC peering isn't transitive, a Cloud SQL Auth proxy is required to bridge the connection from Datastream to your Cloud SQL instance.

The following diagram illustrates using a Cloud SQL Auth proxy to establish a private connection between Datastream and Cloud SQL.

Datastream user flow diagram

Set up a Cloud SQL Auth proxy

  1. Identify the VPC network through which Datastream will connect to the source Cloud SQL instance. This VPC network should be able to connect to the instance.

  2. In this VPC network, create a new VM using the basic Debian or Ubuntu image. This VM will host the Cloud SQL Auth proxy client.

  3. Follow the steps in this guide to set up a Cloud SQL Auth proxy.

  4. Create a private connectivity configuration in Datastream to establish VPC peering between your VPC and Datastream's VPC.

  5. Create a connection profile in Datastream. For the connection details, enter the IP address and port of the VM that hosts the proxy client.

Why do you need a reverse proxy?

If Datastream's VPC network is peered with your VPC network ("Network1"), and your source is accessible from another VPC network ("Network2"), then Datastream can't use only VPC network peering to communicate with the source. A reverse proxy is also needed to bridge the connection between Datastream and the source.

The following diagram illustrates using a reverse proxy to establish a private connection between Datastream and a source that's hosted outside of the Google network.

Datastream user flow diagram

Set up a reverse proxy

  1. Identify the VPC network through which Datastream will connect to the source.
  2. In this VPC network, create a new VM using the basic Debian or Ubuntu image. This VM will host the reverse proxy.
  3. Verify that the subnet is in the same region as Datastream, and that the reverse proxy forwards traffic to the source (and not from it).
  4. Confirm that your VM can communicate with the source by running ping or a telnet command from the VM to the source's internal IP address and port.
  5. SSH into the reverse proxy and create a file, using the following script:
        #! /bin/bash
    
        export DB_ADDR=[IP]
        export DB_PORT=[PORT]
    
        export ETH_NAME=$(ip -o link show | awk -F': ' '{print $2}' | 
        grep -v lo)
    
        export REMOTE_IP_ADDR=$(getent hosts $DB_ADDR | awk '{print $1}')
    
        export LOCAL_IP_ADDR=$(ip -4 addr show $ETH_NAME |
        grep -Po 'inet \K[\d.]+')
    
        echo 1 > /proc/sys/net/ipv4/ip_forward
        iptables -t nat -A PREROUTING -p tcp -m tcp --dport $DB_PORT 
        -j DNAT --to-destination       $REMOTE_IP_ADDR:$DB_PORT
        iptables -t nat -A POSTROUTING -j SNAT --to-source $LOCAL_IP_ADDR
        
  6. Run the script.
  7. Create a private connectivity configuration in Datastream to establish VPC peering between your VPC and Datastream's VPC.
  8. Create a connection profile in Datastream. For the connection details, enter the internal IP address and port of the VM that hosts the proxy.