Manage connections

This page describes how to create and manage connections in Cloud Data Fusion version 6.5 and later.

To connect to data sources and sinks, a connection stores sensitive data, such as user credentials and host information. You can create and manage connections that ETL developers, data analysts, and data engineers can reuse in Wrangler and data pipelines.

In versions 6.5 and later, administrators can create and manage the connections for data pipelines, and developers can use these connections without needing to provide credentials.

For more information about supported connections, see the Connection Reference.

You can manage connections for each namespace in the Cloud Data Fusion interface:

• The Namespace Admin page lets you add, edit, delete, duplicate, export, and import connections for each namespace.

• The Wrangler page lets you add, edit, delete, duplicate, and export connections for each namespace.

• The Studio page lets you add and import connections for supported plugins for each namespace.

Note: Role-based access control (RBAC) is supported for connections. For more information, see the Role-based access control (RBAC) overview.

Before you begin

• Create an instance or upgrade your Cloud Data Fusion instance to version 6.5 or later.

Create a connection

1. In the Cloud Data Fusion web interface, click Menu > Namespace Admin > Connections.

2. Click Add connection.

3. Click the type of connection that you want to create, such as BigQuery.

4. Configure the connection properties.

   Note: Connection names must be unique.

5. Click Test Connection.

6. When you see the success message, click Create.

7. Optional: To export or duplicate connections, click More and follow the prompts.

Use a connection in a plugin

When you use a connection in a plugin, the plugin inherits the connection properties that you configured in the previous step.

To use a connection in one of the supported plugins, follow these steps:

1. In the Cloud Data Fusion interface, open a pipeline on the Studio page.

   Note: You can create a connection from the Studio, Wrangler, or Namespace Admin pages.

2. Under Source, click the source plugin for which you want to use a connection. Connections are available only for these batch source plugins:

   • Amazon S3
   • BigQuery
   • Spanner
   • Cloud Storage
   • Cloud SQL for MySQL
   • Cloud SQL for PostgreSQL
   • Database
   • Kafka
   • Microsoft SQL Server
   • MySQL
   • Oracle
   • PostgreSQL
   • Salesforce
   • SAP Ariba
   • SAP BW Open Hub Batch Source
   • SAP ODP
   • Zendesk

3. Hold the pointer over the source node that appears and click Properties.

4. Set Use Connection to Yes.

5. Click Browse Connections. You can select an existing connection or create a new one. After you select the connection, all properties associated with the connection no longer appear. The connection properties are stored in the connection.

6. Configure the rest of the required plugin properties.

7. Click Validate.

8. Click Close.

Add a connection

Before you add a connection for a database source, you must upload the JDBC driver to Cloud Data Fusion. You can upload the driver from the Hub or the Namespace Admin page.

To add a connection, follow these steps:

1. From the Namespace Admin page, click Connections.

2. Click Add Connection.

3. From the Add a connection page, click the type of connection that you want to create.

4. Configure the connection properties.

   Note: Connection names must be unique in a namespace and can only include letters, numbers, underscores, and hyphens.

5. As a best practice, click Test Connection to ensure the connection works.

6. Click Create.

   The connection appears in the Connection Type field and is available for all users in the namespace. The connection is available for use on the Wrangler page and in the corresponding plugin properties on the Studio page.

Edit a connection

To edit connection properties, follow these steps:

1. From the Namespace Admin page, click Connections.

2. Locate the connection that you want to edit.

3. Click More and click Edit.

   You can edit the connection properties and test the connection.

4. Click Save.

Duplicate a connection

To duplicate a connection, follow these steps:

1. From the Namespace Admin page, click Connections.

2. Locate the connection that you want to duplicate.

3. Click More and click Duplicate.

   Cloud Data Fusion creates a new connection with the properties from the original connection.

4. Enter a name for the connection.

   Note: Connection names must be unique in the namespace.

5. Optional: Edit the connection properties.

6. Click Test Connection and resolve any errors.

7. Click Create.

   The connection is available for use on the Wrangler page and in the corresponding source plugin properties on the Studio page.

Import a Connection

To import a connection, follow these steps:

1. From the Namespace Admin page, click Connections.

2. Click Import and navigate to the JSON file that you want to import into Cloud Data Fusion.

3. Edit the name of the connection and any other properties.

   Note: Connection names must be unique in the namespace.

Export a Connection

After you export a connection, you can share it with others on your team, add it to version control, or deploy it in another environment, such as testing or production.

To export a connection, follow these steps:

1. From the Namespace Admin page, click Connections.

2. Locate the connection that you want to export.

3. From the Namespace Admin page, click More and click Export.

   Cloud Data Fusion exports the connection properties to a JSON file.

Delete a connection

To delete a connection, follow these steps:

1. From the Namespace Admin page, click Connections.

2. Locate the connection that you want to delete.

3. From the Namespace Admin page, click More and click Delete.

   Cloud Data Fusion deletes the connection and it's no longer available for use.

What's next

• Learn more about Cloud Data Fusion.
• Follow one of the tutorials.