Manage connection profiles

Overview

In this section, you learn how to use the Datastream API to:

  • Create connection profiles for a source Oracle database, a source MySQL database, and a destination Cloud Storage bucket
  • Retrieve information about, update, and delete connection profiles
  • Discover the structure of source or destination connection profiles

There are two ways that you can use the Datastream API. You can make REST API calls or you can use the gcloud command-line tool (CLI).

To see high-level information about using gcloud to manage Datastream connection profiles, click here.

Create a connection profile for a source Oracle database

The following code shows a request to create a connection profile for an Oracle database, and specifies the connectivity method as using static IP addresses (IP allowlist).

REST

POST https://datastream.googleapis.com/v1alpha1/projects/[project-id]/locations/
[location]/connectionProfiles?connectionProfileId=[connection-profile-id]
{
  "displayName": "[connection-profile-display-name]",
  "oracleProfile": {
    "hostname": "[host-ip-address]",
    "port": [port],
    "username": "[username]",
    "password": "[password]",
    "databaseService": "[database]"
  },
  "staticServiceIpConnectivity": {}
}

For example:

POST https://datastream.googleapis.com/v1alpha1/projects/myProjectId/locations/
us-central1/connectionProfiles?connectionProfileId=myOracleCP
{
  "displayName": "my Oracle database",
  "oracleProfile": {
    "hostname": "1.2.3.4",
    "port": 1521,
    "username": "admin",
    "password": "12345"
    "databaseService": "ORCL",
  },
  "staticServiceIpConnectivity": {}
}

The following code shows a request to create a connection profile for an Oracle database, and specifies the connectivity method as private connectivity (VPC peering).

This method establishes secure connectivity between Datastream and the source database (internally within Google Cloud, or with external sources connected over VPN or Interconnect). This communication happens through a VPC peering connection.

POST https://datastream.googleapis.com/v1alpha1/projects/[project-id]/locations/
[location]/connectionProfiles?connectionProfileId=[connection-profile-id]
{
  "displayName": "[connection-profile-display-name]",
  "oracleProfile": {
    "hostname": "[host-ip-address]",
    "port": [port],
    "username": "[username]",
    "password": "[password]",
    "databaseService": "[database]"
  },
  "privateConnectivity": {
    "privateConnectionName": "https://datastream.googleapis.com/v1alpha1/projects/
    [project-id]/locations/[location]/privateConnections/
    [private-connectivity-configuration-id]"
  }
}

For example:

POST https://datastream.googleapis.com/v1alpha1/projects/myProjectId/locations/
us-central1/connectionProfiles?connectionProfileId=myOracleVPCPeeringCP
{
  "displayName": "my Oracle database",
  "oracleProfile": {
    "hostname": "1.2.3.4",
    "port": 1521,
    "username": "admin",
    "password": "12345"
    "databaseService": "ORCL",
  },
    "privateConnectivity": {
      "privateConnectionName": "https://datastream.googleapis.com/v1alpha1/projects/
       myProjectId/locations/us-central1/privateConnections/myPrivateConnection"
  }
}

After the connection profile is created, you can view information about it by calling the connectionProfiles/get method. Your output appears, similar to the following:

{
  "name": "projects/projectId/location/us-central1/connectionProfiles/myOracleVPCPeeringCP",
  "createTime": "2019-12-22T16:17:37.159786963Z",
  "updateTime": "2019-12-22T16:17:37.159786963Z",
  "displayName": "my Oracle database",
  "oracleProfile": {
    "hostname": "1.2.3.4",
    "port": 1521,
    "databaseService": "ORCL",
    "username": "admin"
  },
  "privateConnectivity": {
    "privateConnectionName": "https://datastream.googleapis.com/v1alpha1/projects/
     myProjectId/locations/us-central1/privateConnections/myPrivateConnection"
  }
}

Create a connection profile for a source MySQL database

The following code shows a request to create a connection profile for a MySQL database, and specifies the connectivity method as using static IP addresses (IP allowlist).

REST

POST https://datastream.googleapis.com/v1alpha1/projects/[project-id]/locations/
[location]/connectionProfiles?connectionProfileId=[connection-profile-id]
{
  "displayName": "[connection-profile-display-name]",
  "mysqlProfile": {
    "hostname": "[host-ip-address]",
    "port": [port],
    "username": "[username]",
    "password": "[password]",
  },
  "staticServiceIpConnectivity": {}
}

For example:

POST https://datastream.googleapis.com/v1alpha1/projects/myProjectId/locations/
us-central1/connectionProfiles?connectionProfileId=mysql-cp
{
  "displayName": "my MySQL database",
  "mysqlProfile": {
    "hostname": "1.2.3.4",
    "port": 3306,
    "username": "root",
    "password": "12345"
  },
  "staticServiceIpConnectivity": {}
}

Create a connection profile for a destination Cloud Storage bucket

The following code shows a request to create a connection profile for a Cloud Storage bucket. Because Cloud Storage is in the same network as Datastream, no special connectivity is required. Therefore, no connectivity method is specified.

REST

POST https://datastream.googleapis.com/v1alpha1/projects/[project-id/locations/
[location]/connectionProfiles?connectionProfileId=[connection-profile-id]
{
  "displayName": "connection-profile-display-name",
  "gcsProfile": {
    "bucketName": "bucket-name",
    "rootPath": "prefix-inside-bucket"
  }
}

For example:

POST https://datastream.googleapis.com/v1alpha1/projects/myProjectId/locations/
us-central1/connectionProfiles?connectionProfileId=myGcsCP
{
  "displayName": "my Cloud Storage bucket",
  "gcsProfile": {
    "bucketName": "myBucket",
    "rootPath": "prefix-inside-bucket"
  }
}

Get information about a connection profile

The following code shows a request to retrieve information about a connection profile for a source Oracle database, a source MySQL database, or a destination Cloud Storage bucket that has been created.

REST

GET https://datastream.googleapis.com/v1alpha1/projects/project-id/locations/
[location]/connectionProfiles/[connection-profile-id]

For example:

GET https://datastream.googleapis.com/v1alpha1/projects/myProjectId/locations/
us-central1/connectionProfiles/myOracleCP

gcloud

For more information on using gcloud to retrieve information about your connection profile, click here.

List connection profiles

The following code shows a request to retrieve information about all of your connection profiles.

REST

GET https://datastream.googleapis.com/v1alpha1/projects/[project-id]/locations/
[location]/connectionProfiles

gcloud

For more information on using gcloud to retrieve information about all of your connection profiles, click here.

Update a connection profile

The following code shows a request to change the username and password of an existing connection profile for a source database.

By using the updateMask parameter in the request, only the fields that you specify have to be included in the body of the request (for this example, the username and password fields, represented by the oracle_profile.username and oracle_profile.password flags, respectively).

REST

PATCH https://datastream.googleapis.com/v1alpha1/projects/[project-id]/locations/
[location]/connectionProfiles/[connection-profile-id]?
updateMask=oracle_profile.username,oracle_profile.password
{
  "oracleProfile": {
    "username": "username",
    "password": "password"
  }
}

For example:

PATCH https://datastream.googleapis.com/v1alpha1/projects/myProjectId/locations/
us-central1/connectionProfiles/myOracleCP?
updateMask=oracle_profile.username,oracle_profile.password
{
  "oracleProfile": {
    "username": "username",
    "password": "password"
  }
}

The following code shows a request to change the private connectivity configuration that's assigned to a connection profile. This connection profile uses the VPC peering network connectivity method.

For this example, you're assigning the new_private_connection configuration to the connection profile.

PATCH -d {\"private_connectivity\":{\"private_connection_name\":\
"https://datastream.googleapis.com/v1alpha1/
projects/[project-id]/locations/[location]/privateConnections/
new_private_connection\"}}" -H "Authorization: Bearer $TOKEN" -H 
"Content-Type: application/json" https://datastream.googleapis.com/v1alpha1/
projects/[project-id]/locations/[location]/connectionProfiles/
[connection-profile-id]?update_mask=private_connectivity.private_connection_name

For example:

PATCH -d {\"private_connectivity\":{\"private_connection_name\":\
"https://datastream.googleapis.com/v1alpha1/
projects/myProjectId/locations/us-central1/privateConnections/
new_private_connection\"}}" -H "Authorization: Bearer $TOKEN" -H 
"Content-Type: application/json"
https://datastream.googleapis.com/v1alpha1/projects/
myProjectId/locations/us-central1/connectionProfiles/
myOracleVPCPeeringCP?update_mask=private_connectivity.private_connection_name

Delete a connection profile

The following code shows a request to delete a connection profile. After the connection profile is deleted, any streams using it will fail.

REST

DELETE https://datastream.googleapis.com/v1alpha1/projects/[project-id]/locations/
[location]/connectionProfiles/[connection-profile-id]

For example:

DELETE https://datastream.googleapis.com/v1alpha1/projects/myProjectId/locations/
us-central1/connectionProfiles/myOracleCP

gcloud

For more information on using gcloud to delete your connection profile, click here.

Discover the structure of a source or destination connection profile

Use the discoverConnectionProfile API to retrieve the list of entities (for example, schemas and tables) from a source or destination connection profile, and the metadata associated with the entities.

The API can receive as a parameter either an ID of an existing connection profile or a complete connection profile object definition. The function can return a single level (for example, all schemas in a database or all tables in a schema), or all entities recursively (for example, schemas, tables, and columns).

REST

POST https://datastream.googleapis.com/v1alpha1/projects/[project-id]/locations/
[location]/connectionProfiles:discoverConnectionProfile?connection-profile-id

For example:

POST https://datastream.googleapis.com/v1alpha1/projects/myProjectId/locations/
us-central1/connectionProfiles:discoverConnectionProfile?myOracleCP