Managing connections

This page describes how to create and manage connections in Cloud Data Fusion version 6.5 and later.

To connect to data sources and sinks, a connection stores sensitive data, such as user credentials and host information. You can create and manage connections that ETL developers, data analysts, and data engineers can reuse in Wrangler and data pipelines.

In versions 6.5 and later, admins can create and manage the connections for data pipelines, and developers can use these connections without needing to provide credentials.

For more information about supported connections, see Connection Reference.

From the Namespace Admin page, you can add, edit, duplicate, import, export, and delete connections for each namespace. In Wrangler, you can add, edit, duplicate, export, and delete connections for each namespace. In Pipeline Studio, you can add and import connections for supported plugins for each namespace.

Before you begin

  • Create or upgrade your Cloud Data Fusion instance to version 6.5 or later.

  • You need the Cloud Data Fusion Admin role.

Create a connection

  1. In the Cloud Data Fusion UI, click Menu.

  2. Click Namespace Admin.

  3. Click the Connections tab.

  4. Click Add connection.

  5. Click the type of connection you want to create. For example, BigQuery.

  6. Configure the connection properties.

  7. Click Test Connection.

  8. When you see the success message, click Create.

  9. Optional: To export or duplicate connections, click More and follow the prompts.

Use a connection in a plugin

When you use a connection in a plugin, the plugin inherits all of the properties configured when the connection was created. To use a connection in one of the supported plugins, follow these steps:

  1. In the Pipeline Studio, open an existing pipeline.

  2. Under Source, click the source plugin for which you want to use a connection. Connections are available only for these batch source plugins:

    • BigQuery
    • Cloud Storage
    • Kafka
    • Amazon S3
    • Cloud Spanner
  3. Hold the pointer over the source node that appears and click Properties.

  4. Set Use Connection to Yes.

  5. Click Browse Connections. You can select an existing connection or create a new one. After you select the connection, all properties associated with the connection no longer appear. The connection properties are stored in the connection.

  6. Configure the rest of the required plugin properties.

  7. Click Validate.

  8. Click the X button in the upper-right corner of the dialog box.

  9. Continue creating your pipeline in the Pipeline Studio.

Add a connection

Before you add a connection for a database source, you must upload the JDBC driver to Cloud Data Fusion. You can upload the driver from the Hub or the Namespace Admin page.

To add a connection, follow these steps:

  1. From the Namespace Admin page, click Connections.

  2. Click Add Connection.

  3. From the Add a connection page, click the type of connection you want to create.

  4. Configure the connection properties.

  5. As a best practice, click Test Connection to ensure the connection works.

  6. Click Create.

    The connection appears under the connection type and is available for all users in the namespace. The connection is available for use in the Wrangler and in the corresponding plugin in the Pipeline Studio.

Edit a connection

To edit connection properties, follow these steps:

  1. From the Namespace Admin page, click Connections.

  2. Locate the connection you want to edit.

  3. Click More and click Edit.

    You can edit the connection properties and test the connection.

  4. Click Save.

Duplicate a connection

To duplicate a connection, follow these steps:

  1. From the Namespace Admin page, click Connections.

  2. Locate the connection you want to duplicate.

  3. Click More and click Duplicate.

    Cloud Data Fusion creates a new connection with the properties from the original connection.

  4. Enter a name for the connection.

  5. (Optional) Edit the connection properties.

  6. Click Test Connection and resolve any errors.

  7. Click Create.

    The connection is available for use in the Wrangler and in the corresponding source plugin in the Pipeline Studio.

Import a Connection

To import a connection, follow these steps:

  1. From the Namespace Admin page, click Connections.

  2. Click Import and navigate to the JSON file you want to import into Cloud Data Fusion.

  3. Edit the name of the connection and any other properties.

Export a Connection

If you want to share a connection with other team members, or add it to version control, or deploy it from a development environment to a test or production environment, then you must export the connection.

To export a connection, follow these steps:

  1. From the Namespace Admin page, click Connections.

  2. Locate the connection you want to export.

  3. From the Namespace Admin page, click More and click Export.

    Cloud Data Fusion exports the connection properties to a JSON file.

Delete a connection

To delete a connection, follow these steps:

  1. From the Namespace Admin page, click Connections.

  2. Locate the connection you want to delete.

  3. From the Namespace Admin page, click More and click Delete.

    Cloud Data Fusion deletes the connection and it is no longer available for use.

What's next