Overview
In this section, you learn how to create connection profiles to:
- A source Oracle, MySQL, SQL Server (Preview) and PostgreSQL database
- Destination datasets in BigQuery
- A destination bucket in Cloud Storage
By creating these connection profiles, Datastream can transfer data from the source database into the destination.
Create a connection profile for Oracle database
Go to the Connection profiles page in the Google Cloud Console.
Click CREATE PROFILE.
In the Create a connection profile page, click the Oracle profile type (because you want to create a connection profile for Oracle database).
Use the following table to populate the fields of the Define connection settings section of the Create Oracle profile page:
Field Description Connection profile name Enter the display name of the connection profile to the source Oracle database. This is used in the connection profile list as well as when an existing connection profile is selected in the creation of a stream. Connection profile ID Datastream populates this field automatically based on the connection profile name that you enter. You can keep the ID that's auto-generated or change it. Region Select the region where the connection profile is stored. Connection profiles, like all resources, are saved in a region, and a stream can only use connection profiles that are stored in the same region as the stream. Region selection doesn't impact whether Datastream can connect to the source or the destination, but can impact availability if the region experiences downtime. Hostname or IP Enter a hostname or IP address that Datastream can use to connect to the source Oracle database.
If you're using private connectivity to communicate with the source database, then specify the private (internal) IP address for the source database.
For other connectivity methods, such as IP allowlisting, provide the public IP address.
Port Enter the port number that's reserved for the source database (The default port is typically 1521.). Username Enter the username of the account for the source database (for example, ROOT). This is the Datastream user that you created for the database.For more information about creating this user, see Configure your source Oracle database.
Password Enter the password of the account for the source database.
System identifier (SID) Enter the service that ensures that the source Oracle database is protected and monitored. For Oracle databases, the database service is typically ORCL. For pluggable databases, SID is the pluggable database name. Optionally, use labels to organize your Datastream resources.
- To create a label, click ADD LABEL, and then enter the key-value pair for the label.
- To remove the label, click the trashcan icon to the right of the row that contains the label.
In the Define connection settings section, click CONTINUE. The Secure your connection to your source page is active.
From the Encryption type menu, select one of the following:
Encryption type Description None Datastream connects to the source database without encryption. Server-only When Datastream connects to the source database, Datastream authenticates the source, ensuring that it is connecting to the correct host securely. This prevents person-in-the-middle attacks.
For server-only authentication, the source doesn't authenticate Datastream. To use server-only authentication, you must provide the x509 PEM-encoded certificate of the certificate authority (CA) that signed the source's certificate. Make sure you use a single certificate, Datastream doesn't support certificate chains.
In the Secure your connection to your source section, click CONTINUE. The Define connectivity method section of the Create Oracle profile page is active.
From the Connectivity method drop-down menu, select a network connectivity method. This method defines how Datastream will connect to the source database. Current network connectivity methods include:
- IP allowlisting: This method works by configuring the source database server to accept connections from Datastream. If you select this network connectivity method, then configure your source database to allow incoming connections from the Datastream public IP addresses for the region that you specified for the connection profile.
- Forward-SSH tunnel: This method establishes secure, encrypted connectivity between Datastream and the source database, using an SSH tunnel to either a tunnel server or to the database server. If you select this network connectivity method, then:
- Enter the hostname or IP address, and port of the tunnel host server.
- Enter the username of the account for the tunnel host server.
- Select the authentication method for the SSH tunnel. If you select Password as the method, then enter the password of the account for the bastion host VM. If you select Private/Public key pair as the method, then provide a private key.
- Configure your tunnel host to allow incoming connections from the Datastream public IP addresses for the region that you specified for the connection profile.
Private connectivity (VPC peering): This method establishes secure connectivity between Datastream and the source database (internally within Google Cloud, or with external sources connected over VPN or Interconnect). This communication happens through a VPC peering connection.
If you select this network connectivity method, and you have created a private connectivity configuration, then select it from the list of configurations. This type of configuration contains information that Datastream uses to communicate with the source database over a private network.
If you haven't created a private connectivity configuration, create one by clicking CREATE PRIVATE CONNECTIVITY CONFIGURATION at the bottom of the drop-down list, and then perform the same steps as in Create a private connectivity configuration.
In the Define connectivity method section, click CONTINUE. The Test connection profile section of the Create Oracle profile page is active.
Click RUN TEST to verify that Datastream can communicate with the source.
If the test fails, then it indicates which part of the process had an issue. Refer to the Diagnose issues page for troubleshooting steps. Necessary changes can be made and then re-tested on the Create Oracle profile page.
Navigate to the part of the flow in question to correct the issue, and then retest.
Click CREATE.
Create a connection profile for MySQL database
Go to the Connection profiles page in the Google Cloud Console.
Click CREATE PROFILE.
In the Create a connection profile page, click the MySQL profile type (because you want to create a connection profile for MySQL database).
Use the following table to populate the fields of the Define connection settings section of the Create MySQL profile page:
Field Description Connection profile name Enter the display name of the connection profile to the source MySQL database. This is used in the connection profile list as well as when an existing connection profile is selected in the creation of a stream. Connection profile ID Datastream populates this field automatically based on the connection profile name that you enter. You can keep the ID that's auto-generated or change it. Region Select the region where the connection profile is stored. Connection profiles, like all resources, are saved in a region, and a stream can only use connection profiles that are stored in the same region as the stream. Region selection doesn't impact whether Datastream can connect to the source or the destination, but can impact availability if the region experiences downtime. Hostname or IP Enter a hostname or IP address that Datastream can use to connect to the source MySQL database.
If you're using private connectivity to communicate with the source database, then specify the private (internal) IP address for the source database.
For other connectivity methods, such as IP allowlisting or Forward-SSH, provide the public IP address.
Port Enter the port number that's reserved for the source database (The default port is typically 3306.). Username Enter the username of the account for the source database (for example,
root). This is the Datastream user that you created for the database.For more information about creating this user, see Configure a source MySQL database.
Password Enter the password of the account for the source database.
Optionally, use labels to organize your Datastream resources.
- To create a label, click ADD LABEL, and then enter the key-value pair for the label.
- To remove the label, click the trashcan icon to the right of the row that contains the label.
In the Define connection settings section, click CONTINUE. The Secure your connection to your source page is active.
From the Encryption type menu, select one of the following:
Encryption type Description None Datastream connects to the source database without encryption. Server-only When Datastream connects to the source database, Datastream authenticates the source, ensuring that it is connecting to the correct host securely. This prevents person-in-the-middle attacks.
For server-only authentication, the source doesn't authenticate Datastream. To use server-only authentication, you must provide the x509 PEM-encoded certificate of the certificate authority (CA) that signed the source's certificate. Make sure you use a single certificate, Datastream doesn't support certificate chains.
Server-client When Datastream connects to the source, Datastream authenticates the source and the source authenticates Datastream.
Server-client authentication provides the strongest security. However, if you don't want to provide the client certificate and private key when you create the connection profile, you can still use server-only authentication.
To use server-client authentication, you must provide the following items when you create the source connection profile:
- The certificate of the CA that signed the source database server's certificate (the CA certificate).
- The certificate used by the instance to authenticate against the source database server (the client certificate).
- The private key associated with the client certificate (the client key).
In the Secure your connection to your source section, click CONTINUE. The Define connectivity method section of the Create MySQL profile page is active.
From the Connectivity method drop-down menu, select a network connectivity method. This method defines how Datastream will connect to the source database. Current network connectivity methods include:
- IP allowlisting: This method works by configuring the source database server to accept connections from Datastream. If you select this network connectivity method, then configure your source database to allow incoming connections from the Datastream public IP addresses for the region that you specified for the connection profile.
- Forward-SSH tunnel: This method establishes secure, encrypted connectivity between Datastream and the source database, using an SSH tunnel to either a tunnel server or to the database server. If you select this network connectivity method, then:
- Enter the hostname or IP address, and port of the tunnel host server.
- Enter the username of the account for the tunnel host server.
- Select the authentication method for the SSH tunnel. If you select Password as the method, then enter the password of the account for the bastion host VM. If you select Private/Public key pair as the method, then provide a private key.
- Configure your tunnel host to allow incoming connections from the Datastream public IP addresses for the region that you specified for the connection profile.
Private connectivity (VPC peering): This method establishes secure connectivity between Datastream and the source database (internally within Google Cloud, or with external sources connected over VPN or Interconnect). This communication happens through a VPC peering connection.
If you select this network connectivity method, and you have created a private connectivity configuration, then select it from the list of configurations. This type of configuration contains information that Datastream uses to communicate with the source database over a private network.
If you haven't created a private connectivity configuration, create one by clicking CREATE PRIVATE CONNECTIVITY CONFIGURATION at the bottom of the drop-down list, and then perform the same steps as in Create a private connectivity configuration.
In the Define connectivity method section, click CONTINUE. The Test connection profile section of the Create MySQL profile page is active.
Click RUN TEST to verify that Datastream can communicate with the source.
If the test fails, then it indicates which part of the process had an issue. Refer to the Diagnose issues page for troubleshooting steps. You can make the necessary changes and then re-test the connection profile.
Navigate to the part of the flow in question to correct the issue, and then retest.
Click CREATE.
Create a connection profile for PostgreSQL database
Go to the Connection profiles page in the Google Cloud Console.
Click CREATE PROFILE.
In the Create a connection profile page, click the PostgreSQL profile type (because you want to create a connection profile for PostgreSQL database).
Use the following table to populate the fields of the Define connection settings section of the Create PostgreSQL profile page:
Field Description Connection profile name Enter the display name of the connection profile to the source PostgreSQL database. This is used in the connection profile list as well as when an existing connection profile is selected in the creation of a stream. Connection profile ID Datastream populates this field automatically based on the connection profile name that you enter. You can keep the ID that's auto-generated or change it. Region Select the region where the connection profile is stored. Connection profiles, like all resources, are saved in a region, and a stream can only use connection profiles that are stored in the same region as the stream. Region selection doesn't impact whether Datastream can connect to the source or the destination, but can impact availability if the region experiences downtime. Hostname or IP Enter a hostname or IP address that Datastream can use to connect to the source PostgreSQL database.
If you're using private connectivity to communicate with the source database, then specify the private (internal) IP address for the source database.
For other connectivity methods, such as IP allowlisting or Forward-SSH, provide the public IP address.
Port Enter the port number that's reserved for the source database (The default port for PostgreSQL is typically 5432.). Username Enter the username of the account for the source database (for example,
root). This is the Datastream user that you created for the database.For more information about creating this user, see Configure your source PostgreSQL database.
Password Enter the password of the account for the source database.
Database Enter the name that identifies the database instance. For PostgreSQL databases, this is typically
postgres.Optionally, use labels to organize your Datastream resources.
- To create a label, click ADD LABEL, and then enter the key-value pair for the label.
- To remove the label, click the trashcan icon to the right of the row that contains the label.
Click CONTINUE.
In the Define connection settings section, click CONTINUE. The Define connectivity method section of the Create PostgreSQL profile page is active.
From the Connectivity method drop-down menu, select a network connectivity method. This method defines how Datastream will connect to the source database. Current network connectivity methods include:
- IP allowlisting: This method works by configuring the source database server to accept connections from Datastream. If you select this network connectivity method, then configure your source database to allow incoming connections from the Datastream public IP addresses for the region that you specified for the connection profile.
- Forward-SSH tunnel: This method establishes secure, encrypted connectivity between Datastream and the source database, using an SSH tunnel to either a tunnel server or to the database server. If you select this network connectivity method, then:
- Enter the hostname or IP address, and port of the tunnel host server.
- Enter the username of the account for the tunnel host server.
- Select the authentication method for the SSH tunnel. If you select Password as the method, then enter the password of the account for the bastion host VM. If you select Private/Public key pair as the method, then provide a private key.
- Configure your tunnel host to allow incoming connections from the Datastream public IP addresses for the region that you specified for the connection profile.
Private connectivity (VPC peering): This method establishes secure connectivity between Datastream and the source database (internally within Google Cloud, or with external sources connected over VPN or Interconnect). This communication happens through a VPC peering connection.
If you select this network connectivity method, and you have created a private connectivity configuration, then select it from the list of configurations. This type of configuration contains information that Datastream uses to communicate with the source database over a private network.
If you haven't created a private connectivity configuration, create one by clicking CREATE PRIVATE CONNECTIVITY CONFIGURATION at the bottom of the drop-down list, and then perform the same steps as in Create a private connectivity configuration.
In the Define connectivity method section, click CONTINUE. The Test connection profile section of the Create PostgreSQL profile page is active.
Click RUN TEST to verify that Datastream can communicate with the source.
If the test fails, then it indicates which part of the process had an issue. Refer to the Diagnose issues page for troubleshooting steps. You can make the necessary changes and then re-test the connection profile.
Navigate to the part of the flow in question to correct the issue, and then retest.
Click CREATE.
Create a connection profile for SQL Server database
Go to the Connection profiles page in the Google Cloud Console.
Click CREATE PROFILE.
In the Create a connection profile page, click the SQL Server profile type.
Use the following table to populate the fields of the Define connection settings section of the Create SQL Server profile page:
Field Description Connection profile name Enter the display name of the connection profile to the source SQL Server database. This is used in the connection profile list as well as when an existing connection profile is selected in the creation of a stream. Connection profile ID Datastream populates this field automatically based on the connection profile name that you enter. You can keep the ID that's auto-generated or change it. Region Select the region where the connection profile is stored. Connection profiles, like all resources, are saved in a region, and a stream can only use connection profiles that are stored in the same region as the stream. Region selection doesn't impact whether Datastream can connect to the source or the destination, but can impact availability if the region experiences downtime. Hostname or IP Enter a hostname or IP address that Datastream can use to connect to the source SQL Server database.
If you're using private connectivity to communicate with the source database, then specify the private (internal) IP address for the source database.
For IP allowlisting, provide the public IP address.
Port Enter the port number that's reserved for the source database (The default port is typically 1433.). Username Enter the username of the account for the source database (for example,
root). This is the Datastream user that you created for the database.For more information about creating this user, see see Configure your source SQL Server database.
Password Enter the password of the account for the source database.
Database Enter the name that identifies the database instance.
Optionally, use labels to organize your Datastream resources.
- To create a label, click ADD LABEL, and then enter the key-value pair for the label.
- To remove the label, click the trashcan icon to the right of the row that contains the label.
Click Continue.
In the Define connectivity method section, from the Connectivity method drop-down menu, select a network connectivity method. This method defines how Datastream will connect to the source database. Network connectivity methods include:
- IP allowlisting: This method works by configuring the source database server to accept connections from Datastream. If you select this network connectivity method, then configure your source database to allow incoming connections from the Datastream public IP addresses for the region that you specified for the connection profile.
Private connectivity (VPC peering): This method establishes secure connectivity between Datastream and the source database (internally within Google Cloud, or with external sources connected over VPN or Interconnect). This communication happens through a VPC peering connection.
If you select this network connectivity method, and you have created a private connectivity configuration, then select it from the list of configurations. This type of configuration contains information that Datastream uses to communicate with the source database over a private network.
If you haven't created a private connectivity configuration, create one by clicking CREATE PRIVATE CONNECTIVITY CONFIGURATION at the bottom of the drop-down list, and then perform the same steps as in Create a private connectivity configuration.
Click CONTINUE. The Test connection profile section of the Create SQL Server profile page is active.
Click RUN TEST to verify that Datastream can communicate with the source.
If the test fails, then it indicates which part of the process had an issue. Refer to the Diagnose issues page for troubleshooting steps. You can make the necessary changes and then re-test the connection profile.
Navigate to the part of the flow in question to correct the issue, and then retest.
Click CREATE.
Create a connection profile for BigQuery
Go to the Connection profiles page in the Google Cloud Console.
Click CREATE PROFILE.
In the Create a connection profile page, click the BigQuery profile type (because you want to create a connection profile for BigQuery).
Use the following table to populate the fields of the Create BigQuery profile page:
Field Description Connection profile name Enter the display name of the connection profile to the destination datasets in BigQuery. This is used in the connection profile list as well as when an existing connection profile is selected in the creation of a stream. Connection profile ID Keep the auto-generated value that's populated in this field. Region Select the region where the connection profile is stored. Connection profiles, like all resources, are saved in a region, and a stream can only use connection profiles that are stored in the same region as the stream. Region selection doesn't impact whether Datastream can connect to the source or the destination, but can impact availability if the region experiences downtime. Region selection is also independent of the location type you selected for your BigQuery destination dataset. Optionally, use labels to organize your Datastream resources.
- To create a label, click ADD LABEL, and then enter the key-value pair for the label.
- To remove the label, click the trashcan icon to the right of the row that contains the label.
Click CREATE.
Create a connection profile for Cloud Storage
Go to the Connection profiles page in the Google Cloud Console.
Click CREATE PROFILE.
In the Create a connection profile page, click the Cloud Storage profile type (because you want to create a connection profile for Cloud Storage).
Use the following table to populate the fields of the Create Cloud Storage profile page:
Field Description Connection profile name Enter the display name of the connection profile to the destination bucket in Cloud Storage. This is used in the connection profile list as well as when an existing connection profile is selected in the creation of a stream. Connection profile ID Keep the auto-generated value that's populated in this field. Region Select the region where the connection profile is stored. Connection profiles, like all resources, are saved in a region, and a stream can only use connection profiles that are stored in the same region as the stream. Region selection doesn't impact whether Datastream can connect to the source or the destination, but can impact availability if the region experiences downtime. Bucket name Click BROWSE to create or select the destination bucket in Cloud Storage into which Datastream will transfer data from the source database.
If you select an existing bucket, then complete the following steps:
- Click the Search icon (which appears as a magnifying glass).
- In the Search by name field, enter the first few characters of the name of the bucket that you want to select. Datastream filters the list of buckets to reflect what you enter in the field.
- Select the bucket that you want to use as your destination.
- Click SELECT. Your bucket appears in the Bucket name field.
Alternatively, you can click the Create new bucket icon (which appears as a briefcase) to create a bucket.
Optionally, in the Connection profile path prefix field, you can provide a prefix for the path that will be appended to the bucket name when Datastream transfers data to the destination.
Optionally, use labels to organize your Datastream resources.
- To create a label, click ADD LABEL, and then enter the key-value pair for the label.
- To remove the label, click the trashcan icon to the right of the row that contains the label.
Click CREATE.
After creating connection profiles, you can view high-level and detailed information about them.