Schedule a Salesforce transfer

Preview

This feature is subject to the "Pre-GA Offerings Terms" in the General Service Terms section of the Service Specific Terms. Pre-GA features are available "as is" and might have limited support. For more information, see the launch stage descriptions.

Note: To get support or provide feedback for this feature, contact [e-mail].

The BigQuery Data Transfer Service for Salesforce connector lets you automatically schedule and manage recurring load jobs from Salesforce into BigQuery.

Limitations

Salesforce data transfers are subject to the following limitations:

• The BigQuery Data Transfer Service for Salesforce only supports the Salesforce Bulk API to connect to the Salesforce instance, and only supports the transfer of entities which are supported by the Salesforce Bulk API. For more information about what entities are supported, see 'Entity is not supported by the Bulk API' error.
• The minimum interval time between recurring data transfers is 15 minutes. The default interval for a recurring transfer is 24 hours.
• The BigQuery Data Transfer Service uses Salesforce Bulk API v1 to connect to the Salesforce endpoint to retrieve data.

Before you begin

The following sections describe the steps that you need to take before you create a Salesforce data transfer.

Create a Salesforce Connected App

You must create a Salesforce Connected App with the following required configurations:

• Configure the Basic Information in the Connected App. The Connected App Name and Contact Email fields are required for a Salesforce transfer.
• Enable OAuth Settings with the following configurations:
  • Select the Enable OAuth Settings checkbox.
  • In the Callback URL field, enter the following:
    • For a production environment, enter https://login.salesforce.com/services/oauth2/token.
    • For a sandbox environment, enter https://test.salesforce.com/services/oauth2/token.
• In the Selected OAuth Scopes section, select Manage user data via APIs (api).
• Clear the Required Proof Key for Code Exchange (PKCE) Extension for Supported Authorization Flows checkbox.
• Select the Enable Client Credentials Flow, then click OK on the notice that appears.

Once you have configured the Connected App with the required configurations, click Save. You are redirected to the detail page of your newly created Connected App.

Once you have created the Connected App, you must also configure the client credentials flow by doing the following:

1. Click Setup.
2. In the search bar, search for Connected Apps.
3. Click Manage Apps > Connected Apps. If you are using the Salesforce Lightning Experience, click Manage Connected Apps.
4. On the Connected App that you have created, click Edit.
5. The App details page appears. In the Client Credentials Flow section, enter your username in the Run As field. You can use the finder tool in this field to ensure that you have selected the correct user.
6. Click Save.

Required Salesforce information

You must have the following Salesforce information when creating a Salesforce data transfer:

Parameter Name	Description
myDomain	Your My Domain in Salesforce.
clientId	Consumer Key of the Salesforce connected application.
clientSecret	OAuth Client Secret or Consumer Secret of the Salesforce connected application.

To obtain your myDomain, clientID, and clientSecret values, select one of the following options:

Salesforce Classic

Retrieve myDomain details

To find your myDomain, do the following:

1. Sign in to the Salesforce platform.
2. Click Setup.
3. In the search bar, search for My Domain.
4. In the search results, click Domain Management > My Domain.

In the My Domain Details section, your myDomain appears as the prefix in Current My Domain URL. For example, if the My Domain URL is example.my.salesforce.com, the myDomain value to use is example.

Retrieve ClientId and ClientSecret details

To find your ClientId and ClientSecret values, do the following:

1. Sign in to the Salesforce platform.
2. Click Setup.
3. In the search bar, search for Apps.
4. In the Build section in the search results, click Create > Apps.
5. Click a Connected App Name.
6. In the Connected Apps details page, click Manage Consumer Details.
7. Verify your identity using one of the registered methods. You can view the consumer details page for up to five minutes before you're prompted to verify your identity again.
8. In the Consumer Details page, the Consumer Key is your ClientId value. The Customer Secret is your ClientSecret value.

Salesforce Lightning Experience

Retrieve myDomain details

To find your myDomain, do the following:

1. Sign in to the Salesforce platform.
2. Click Setup.

1. In the search bar, search for My Domain.
2. In the search results, click Company Settings > My Domain.

In the My Domain Details section, your myDomain appears as the prefix in Current My Domain URL. For example, if the My Domain URL is example.my.salesforce.com, the myDomain value to use is example.

Retrieve ClientId and ClientSecret details

1. Sign in to the Salesforce platform.
2. Click Setup.
3. In the search bar, search for Apps.
4. In the search results, click Apps > App Manager.
5. Find a connected app, then click View.
6. Click Manage Consumer Details.
7. Verify your identity using one of the registered methods. You can view the consumer details page for up to five minutes before you're prompted to verify your identity again.
8. In the Consumer Details page, the Consumer Key is your ClientId value. The Customer Secret is your ClientSecret value.

BigQuery prerequisites

• Verify that you have completed all actions required to enable the BigQuery Data Transfer Service.
• Create a BigQuery dataset to store your data.
• If you intend to set up transfer run notifications for Pub/Sub, ensure that you have the pubsub.topics.setIamPolicy Identity and Access Management (IAM) permission. Pub/Sub permissions are not required if you only set up email notifications. For more information, see BigQuery Data Transfer Service run notifications.

Required BigQuery roles

To get the permissions that you need to create a transfer, ask your administrator to grant you the BigQuery Admin (roles/bigquery.admin) IAM role. For more information about granting roles, see Manage access to projects, folders, and organizations.

This predefined role contains the permissions required to create a transfer. To see the exact permissions that are required, expand the Required permissions section:

Required permissions

The following permissions are required to create a transfer:

• bigquery.transfers.update on the user
• bigquery.datasets.get on the target dataset
• bigquery.datasets.update on the target dataset

You might also be able to get these permissions with custom roles or other predefined roles.

Set up a Salesforce data transfer

To create a Salesforce data transfer:

Console

1. Go to the Data transfers page in the Google Cloud console.

   Go to Data transfers

2. Click Create transfer.

3. In the Source type section, for Source, choose Salesforce.

4. In the Data source details section, do the following:

   • For My Domain, enter your Salesforce My Domain.
   • For Client ID, enter the Salesforce connected application Consumer Key.
   • For Client secret, enter the Salesforce connected application Consumer Secret.

   • For Salesforce objects to transfer, click Browse to select any objects to be transferred to the BigQuery destination dataset. You can also manually enter any objects to include in the data transfer in this field.

     

5. In the Destination settings section, for Dataset, choose the dataset that you created to store your data.

6. In the Transfer config name section, for Display name, enter a name for the data transfer.

7. In the Schedule options section:

   • In the Repeat frequency list, select an option to specify how often this data transfer runs. To specify a custom repeat frequency, select Custom. If you select On-demand, then this transfer runs when you manually trigger the transfer.
   • If applicable, select either Start now or Start at set time, and provide a start date and run time.

8. In the Service Account list, select a service account that is associated with your Google Cloud project. The selected service account must have the required roles to run this data transfer.

   If you signed in with a federated identity, then a service account is required to create a data transfer. If you signed in with a Google Account, then a service account for the transfer is optional. For more information about using service accounts with data transfers, see Use service accounts.

9. Optional: In the Notification options section, do the following:

   • To enable email notifications, click the Email notification toggle. When you enable this option, the transfer administrator receives an email notification when a transfer run fails.
   • To enable Pub/Sub transfer run notifications for this transfer, click the Pub/Sub notifications toggle. You can select your topic name, or you can click Create a topic to create one.

10. Click Save.

bq

Enter the bq mk command and supply the transfer creation flag --transfer_config:

bq mk \
    --transfer_config \
    --project_id=PROJECT_ID \
    --data_source=DATA_SOURCE \
    --display_name=NAME \
    --target_dataset=DATASET \
    --params='PARAMETERS'

Where:

• PROJECT_ID (optional): your Google Cloud project ID. If --project_id isn't supplied to specify a particular project, the default project is used.
• DATA_SOURCE: the data source — salesforce.
• NAME: the display name for the data transfer configuration. The transfer name can be any value that lets you identify the transfer if you need to modify it later.
• DATASET: the target dataset for the transfer configuration.

• PARAMETERS: the parameters for the created transfer configuration in JSON format. For example: --params='{"param":"param_value"}'. The following are the parameters for a Salesforce data transfer:

  • connector.authentication.oauth.clientId: The Consumer Key of the Salesforce connected application.
  • connector.authentication.oauth.clientSecret: OAuth Client Secret or Consumer Secret of the Salesforce connected application.
  • connector.authentication.oauth.myDomain: the Salesforce My Domain. For example, if your domain URL is example.my.salesforce.com, then the value is example.
  • assets: the path to the Salesforce objects to be transferred to BigQuery.

For example, the following command creates a Salesforce data transfer in the default project with all the required parameters:

bq mk \
    --transfer_config \
    --target_dataset=mydataset \
    --data_source=salesforce \
    --display_name='My Transfer' \
    --params='{"assets":["Account"],
        "connector.authentication.oauth.clientId": "1234567890",
        "connector.authentication.oauth.clientSecret":"ABC12345",
        "connector.authentication.oauth.myDomain":"MyDomainName",}'

API

Use the projects.locations.transferConfigs.create method and supply an instance of the TransferConfig resource.

Pricing

There is no cost to transfer Salesforce data into BigQuery while this feature is in Preview.

Troubleshoot transfer setup

If you are having issues setting up your data transfer, see Salesforce transfer issues.

What's next

• For an overview of the BigQuery Data Transfer Service, see Introduction to BigQuery Data Transfer Service.
• For information on using transfers including getting information about a transfer configuration, listing transfer configurations, and viewing a transfer's run history, see Working with transfers.
• Learn how to load data with cross-cloud operations.