Working with transfers

After creating a transfer using the BigQuery Data Transfer Service, you can:

• Get information about a transfer configuration
• List transfer configurations
• View a transfer's run history
• View transfer run details such as log messages
• Update a transfer
• Set up a backfill
• Update credentials
• Disable a transfer
• Delete a transfer

Get information about transfers

After you create a transfer, you can get information about the transfer's configuration. The configuration includes the values you supplied when you created the transfer.

Required permissions

Getting information about a transfer requires bigquery.transfers.get permissions. The following predefined IAM roles include bigquery.transfers.get permissions:

• bigquery.admin
• bigquery.user

A user granted the bigquery.transfers.get permissions can view information about all transfers in a project.

For more information on IAM roles in BigQuery Data Transfer Service, see Access control.

Get information about a transfer

To get information about a transfer configuration:

Console

1. Go to the Data transfers page.

   Go to Data transfer

2. Select the transfer for which you want to get the details.

3. To see the transfer configuration and the data source details, click Configuration on the Transfer details page. The following example shows the configuration properties for a Google Ads transfer:

   

bq

Enter the bq show command and provide the transfer configuration's resource name. The --format flag can be used to control the output format.

bq show \
--format=prettyjson \
--transfer_config resource_name

Where:

• resource_name is the transfer's resource name (also referred to as the transfer configuration). If you do not know the transfer's resource name, find the resource name with: bq ls --transfer_config --transfer_location=location.

For example, enter the following command to display transfer configuration for projects/myproject/locations/us/transferConfigs/1234a123-1234-1a23-1be9-12ab3c456de7.

bq show \
--format=prettyjson \
--transfer_config projects/myproject/locations/us/transferConfigs/1234a123-1234-1a23-1be9-12ab3c456de7

API

Use the projects.locations.transferConfigs.get method and supply the transfer configuration using the name parameter.

Java

Before trying this sample, follow the Java setup instructions in the BigQuery quickstart using client libraries. For more information, see the BigQuery Java API reference documentation.

To authenticate to BigQuery, set up Application Default Credentials. For more information, see Set up authentication for a local development environment.

View on GitHub Feedback

import com.google.api.gax.rpc.ApiException;
import com.google.cloud.bigquery.datatransfer.v1.DataTransferServiceClient;
import com.google.cloud.bigquery.datatransfer.v1.GetTransferConfigRequest;
import com.google.cloud.bigquery.datatransfer.v1.TransferConfig;
import java.io.IOException;

// Sample to get config info.
public class GetTransferConfigInfo {

  public static void main(String[] args) throws IOException {
    // TODO(developer): Replace these variables before running the sample.
    String configId = "MY_CONFIG_ID";
    // i.e projects/{project_id}/transferConfigs/{config_id}` or
    // `projects/{project_id}/locations/{location_id}/transferConfigs/{config_id}`
    getTransferConfigInfo(configId);
  }

  public static void getTransferConfigInfo(String configId) throws IOException {
    try (DataTransferServiceClient dataTransferServiceClient = DataTransferServiceClient.create()) {
      GetTransferConfigRequest request =
          GetTransferConfigRequest.newBuilder().setName(configId).build();
      TransferConfig info = dataTransferServiceClient.getTransferConfig(request);
      System.out.print("Config info retrieved successfully." + info.getName() + "\n");
    } catch (ApiException ex) {
      System.out.print("config not found." + ex.toString());
    }
  }
}

List transfer configurations

Required permissions

Listing all the transfers, or transfer configurations, in a project requires the bigquery.transfers.get permission. The following predefined IAM roles include bigquery.transfers.get permissions:

• bigquery.admin
• bigquery.user

A user granted the bigquery.transfers.get permissions can list all transfers in a project.

For more information on IAM roles in BigQuery Data Transfer Service, see Access control.

List transfers

To list transfer configurations in a project:

Console

1. In the Google Cloud console, go to the BigQuery page.

   Go to the BigQuery page

2. Click Transfers.

3. If there are any transfer configurations in the project, a list of the transfer configurations appears on the Transfers page.

bq

To list all transfer configurations for a project by location, enter the bq ls command and supply the --transfer_location and --transfer_config flags. You can also supply the --project_id flag to specify a particular project. If --project_id isn't specified, the default project is used. The --format flag can be used to control the output format.

To list transfer configurations for particular data sources, supply the --filter flag.

To view a particular number of transfer configurations in paginated format, supply the --max_results flag to specify the number of transfers. The command returns a page token you supply using the --page_token flag to see the next n configurations. There is a limit of 1000 configurations that will be returned if --max_results is omitted, and --max_results will not accept values greater than 1000. If your project has more than 1000 configurations, use --max_results and --page_token to iterate through them all.

bq ls \
--transfer_config \
--transfer_location=location \
--project_id=project_id \
--max_results=integer \
--filter=dataSourceIds:data_sources

Where:

• location is the location of the transfer configurations. The location is specified when you create a transfer.
• project_id is your project ID.
• integer is the number of results to show per page.
• data_sources is one or more of the following:
  • amazon_s3 - Amazon S3 data transfer
  • azure_blob_storage - Azure Blob Storage data transfer
  • dcm_dt - Campaign Manager data transfer
  • google_cloud_storage - Cloud Storage data transfer
  • cross_region_copy - Dataset Copy
  • dfp_dt- Google Ad Manager data transfer
  • adwords - Google Ads data transfer
  • google_ads - Google Ads data transfer (preview)
  • merchant_center - Google Merchant Center data transfer
  • play - Google Play data transfer
  • scheduled_query - Scheduled queries data transfer
  • doubleclick_search- Search Ads 360 data transfer
  • youtube_channel - YouTube Channel data transfer
  • youtube_content_owner - YouTube Content Owner data transfer
  • redshift - Amazon Redshift migration
  • on_premises - Teradata migration

Examples:

Enter the following command to display all transfer configurations in the US for your default project. The output is controlled using the --format flag.

bq ls \
--format=prettyjson \
--transfer_config \
--transfer_location=us

Enter the following command to display all transfer configurations in the US for project ID myproject.

bq ls \
--transfer_config \
--transfer_location=us \
--project_id=myproject

Enter the following command to list the 3 most recent transfer configurations.

bq ls \
--transfer_config \
--transfer_location=us \
--project_id=myproject \
--max_results=3

The command returns a next page token. Copy the page token and supply it in the bq ls command to see the next 3 results.

bq ls \
--transfer_config \
--transfer_location=us \
--project_id=myproject \
--max_results=3 \
--page_token=AB1CdEfg_hIJKL

Enter the following command to list Ads and Campaign Manager transfer configurations for project ID myproject.

bq ls \
--transfer_config \
--transfer_location=us \
--project_id=myproject \
--filter=dataSourceIds:dcm_dt,adwords

API

Use the projects.locations.transferConfigs.list method and supply the project ID using the parent parameter.

Java

Before trying this sample, follow the Java setup instructions in the BigQuery quickstart using client libraries. For more information, see the BigQuery Java API reference documentation.

To authenticate to BigQuery, set up Application Default Credentials. For more information, see Set up authentication for a local development environment.

View on GitHub Feedback

import com.google.api.gax.rpc.ApiException;
import com.google.cloud.bigquery.datatransfer.v1.DataTransferServiceClient;
import com.google.cloud.bigquery.datatransfer.v1.ListTransferConfigsRequest;
import com.google.cloud.bigquery.datatransfer.v1.ProjectName;
import java.io.IOException;

// Sample to get list of transfer config
public class ListTransferConfigs {

  public static void main(String[] args) throws IOException {
    // TODO(developer): Replace these variables before running the sample.
    final String projectId = "MY_PROJECT_ID";
    listTransferConfigs(projectId);
  }

  public static void listTransferConfigs(String projectId) throws IOException {
    try (DataTransferServiceClient dataTransferServiceClient = DataTransferServiceClient.create()) {
      ProjectName parent = ProjectName.of(projectId);
      ListTransferConfigsRequest request =
          ListTransferConfigsRequest.newBuilder().setParent(parent.toString()).build();
      dataTransferServiceClient
          .listTransferConfigs(request)
          .iterateAll()
          .forEach(config -> System.out.print("Success! Config ID :" + config.getName() + "\n"));
    } catch (ApiException ex) {
      System.out.println("Config list not found due to error." + ex.toString());
    }
  }
}

Python

Before trying this sample, follow the Python setup instructions in the BigQuery quickstart using client libraries. For more information, see the BigQuery Python API reference documentation.

To authenticate to BigQuery, set up Application Default Credentials. For more information, see Set up authentication for a local development environment.

View on GitHub Feedback

from google.cloud import bigquery_datatransfer

transfer_client = bigquery_datatransfer.DataTransferServiceClient()

project_id = "my-project"
parent = transfer_client.common_project_path(project_id)

configs = transfer_client.list_transfer_configs(parent=parent)
print("Got the following configs:")
for config in configs:
    print(f"\tID: {config.name}, Schedule: {config.schedule}")

View the run history

As your scheduled transfers are run, a run history is kept for each transfer configuration that includes successful transfer runs and transfer runs that fail. Transfer runs more than 90 days old are automatically deleted from the run history.

Required permissions

Viewing the run history for a transfer configuration requires bigquery.transfers.get permissions. Both the bigquery.user and bigquery.admin predefined IAM roles include bigquery.transfers.get permissions.

For more information on IAM roles in BigQuery Data Transfer Service, see Access control.

View a run history

To view the run history for a transfer configuration:

Console

1. In the Google Cloud console, go to the BigQuery page.

   Go to the BigQuery page

2. Click Transfers.

3. On the Transfers page, click on an appropriate transfer in the list.

4. You will be on the RUN HISTORY page for the selected transfer.

bq

To list transfer runs for a particular transfer configuration, enter the bq ls command and supply the --transfer_run flag. You can also supply the --project_id flag to specify a particular project. If resource_name doesn't contain project information, the --project_id value is used. If --project_id isn't specified, the default project is used. The --format flag can be used to control the output format.

To view a particular number of transfer runs, supply the --max_results flag. The command returns a page token you supply using the --page_token flag to see the next n configurations.

To list transfer runs based on run state, supply the --filter flag.

bq ls \
--transfer_run \
--max_results=integer \
--transfer_location=location \
--project_id=project_id \
--filter=states:state, ... \
resource_name

Where:

• integer is the number of results to return.
• location is the location of the transfer configurations. The location is specified when you create a transfer.
• project_id is your project ID.
• state, ... is one of the following or a comma-separated list:
  • SUCCEEDED
  • FAILED
  • PENDING
  • RUNNING
  • CANCELLED
• resource_name is the transfer's resource name (also referred to as the transfer configuration). If you do not know the transfer's resource name, find the resource name with: bq ls --transfer_config --transfer_location=location.

Examples:

Enter the following command to display the 3 latest runs for transfer configuration projects/myproject/locations/us/transferConfigs/1234a123-1234-1a23-1be9-12ab3c456de7. The output is controlled using the --format flag.

bq ls \
--format=prettyjson \
--transfer_run \
--max_results=3 \
--transfer_location=us \
projects/myproject/locations/us/transferConfigs/1234a123-1234-1a23-1be9-12ab3c456de7

The command returns a next page token. Copy the page token and supply it in the bq ls command to see the next 3 results.

bq ls \
--format=prettyjson \
--transfer_run \
--max_results=3 \
--page_token=AB1CdEfg_hIJKL \
--transfer_location=us \
projects/myproject/locations/us/transferConfigs/1234a123-1234-1a23-1be9-12ab3c456de7

Enter the following command to display all failed runs for transfer configuration projects/myproject/locations/us/transferConfigs/1234a123-1234-1a23-1be9-12ab3c456de7.

bq ls \
--format=prettyjson \
--transfer_run \
--filter=states:FAILED \
--transfer_location=us \
projects/myproject/locations/us/transferConfigs/1234a123-1234-1a23-1be9-12ab3c456de7

API

Use the projects.locations.transferConfigs.runs.list method and specify the project ID using the parent parameter.

Java

Before trying this sample, follow the Java setup instructions in the BigQuery quickstart using client libraries. For more information, see the BigQuery Java API reference documentation.

To authenticate to BigQuery, set up Application Default Credentials. For more information, see Set up authentication for a local development environment.

View on GitHub Feedback

import com.google.api.gax.rpc.ApiException;
import com.google.cloud.bigquery.datatransfer.v1.DataTransferServiceClient;
import com.google.cloud.bigquery.datatransfer.v1.ListTransferRunsRequest;
import java.io.IOException;

// Sample to get run history from transfer config.
public class RunHistory {

  public static void main(String[] args) throws IOException {
    // TODO(developer): Replace these variables before running the sample.
    String configId = "MY_CONFIG_ID";
    // i.e projects/{project_id}/transferConfigs/{config_id}` or
    // `projects/{project_id}/locations/{location_id}/transferConfigs/{config_id}`
    runHistory(configId);
  }

  public static void runHistory(String configId) throws IOException {
    try (DataTransferServiceClient dataTransferServiceClient = DataTransferServiceClient.create()) {
      ListTransferRunsRequest request =
          ListTransferRunsRequest.newBuilder().setParent(configId).build();
      dataTransferServiceClient
          .listTransferRuns(request)
          .iterateAll()
          .forEach(run -> System.out.print("Success! Run ID :" + run.getName() + "\n"));
    } catch (ApiException ex) {
      System.out.println("Run history not found due to error." + ex.toString());
    }
  }
}

View transfer run details and log messages

When a transfer run appears in the run history, you can view the run details including log messages, warnings and errors, the run name, and the start and end time.

Required permissions

Viewing transfer run details requires bigquery.transfers.get permissions. Both the bigquery.user and bigquery.admin predefined IAM roles include bigquery.transfers.get permissions.

For more information on IAM roles in BigQuery Data Transfer Service, see Access control.

Additionally, viewing log messages through Google Cloud console requires permissions to view Cloud Logging data, the Logs Viewer role (roles/logging.viewer) gives you read-only access to all features of Logging. For more information on the IAM permissions and roles that apply to cloud logs data, see the Cloud Logging access control guide.

View details for a transfer run

To view transfer run details:

Console

1. In the Google Cloud console, go to the BigQuery page.

   Go to the BigQuery page

2. Click Transfers.

3. On the Transfers page, click on an appropriate transfer in the list.

4. You will be on the RUN HISTORY page for the selected transfer.

5. Click on an individual run of the transfer, and the Run details panel will open for that run of the transfer.

6. In the Run details, note any error messages. This information is needed if you contact Cloud Customer Care. The run details also include log messages and warnings.

   

bq

To view transfer run details, enter the bq show command and provide the transfer run's Run Name using the --transfer_run flag. The --format flag can be used to control the output format.

bq show \
--format=prettyjson \
--transfer_run run_name

Where:

• run_name is the transfer run's Run Name. You can retrieve the Run Name by using the bq ls command.

Example:

Enter the following command to display details for transfer run projects/myproject/locations/us/transferConfigs/1234a123-1234-1a23-1be9-12ab3c456de7/runs/1a2b345c-0000-1234-5a67-89de1f12345g.

bq show \
--format=prettyjson \
--transfer_run \
projects/myproject/locations/us/transferConfigs/1234a123-1234-1a23-1be9-12ab3c456de7/runs/1a2b345c-0000-1234-5a67-89de1f12345g

To view transfer log messages for a transfer run, enter the bq ls command with the --transfer_log flag. You can filter log messages by type using the --message_type flag.

To view a particular number of log messages, supply the --max_results flag. The command returns a page token you supply using the --page_token flag to see the next n messages.

bq ls \
--transfer_log \
--max_results=integer \
--message_type=messageTypes:message_type \
run_name

Where:

• integer is the number of log messages to return.
• message_type is the type of log message to view (a single value or a comma-separated list):
  • INFO
  • WARNING
  • ERROR
• run_name is the transfer run's Run Name. You can retrieve the Run Name using the bq ls command.

Examples:

Enter the following command to view the first 2 log messages for transfer run projects/myproject/locations/us/transferConfigs/1234a123-1234-1a23-1be9-12ab3c456de7/runs/1a2b345c-0000-1234-5a67-89de1f12345g.

bq ls \
--transfer_log \
--max_results=2 \
projects/myproject/locations/us/transferConfigs/1234a123-1234-1a23-1be9-12ab3c456de7/runs/1a2b345c-0000-1234-5a67-89de1f12345g

The command returns a next page token. Copy the page token and supply it in the bq ls command to see the next 2 results.

bq ls \
--transfer_log \
--max_results=2 \
--page_token=AB1CdEfg_hIJKL \
projects/myproject/locations/us/transferConfigs/1234a123-1234-1a23-1be9-12ab3c456de7/runs/1a2b345c-0000-1234-5a67-89de1f12345g

Enter the following command to view only error messages for transfer run projects/myproject/locations/us/transferConfigs/1234a123-1234-1a23-1be9-12ab3c456de7/runs/1a2b345c-0000-1234-5a67-89de1f12345g.

bq ls \
--transfer_log \
--message_type=messageTypes:ERROR \
projects/myproject/locations/us/transferConfigs/1234a123-1234-1a23-1be9-12ab3c456de7/runs/1a2b345c-0000-1234-5a67-89de1f12345g

API

Use the projects.transferConfigs.runs.transferLogs.list method and supply the transfer run's Run Name using the parent parameter.

Java

Before trying this sample, follow the Java setup instructions in the BigQuery quickstart using client libraries. For more information, see the BigQuery Java API reference documentation.

To authenticate to BigQuery, set up Application Default Credentials. For more information, see Set up authentication for a local development environment.

View on GitHub Feedback

import com.google.api.gax.rpc.ApiException;
import com.google.cloud.bigquery.datatransfer.v1.DataTransferServiceClient;
import com.google.cloud.bigquery.datatransfer.v1.GetTransferRunRequest;
import com.google.cloud.bigquery.datatransfer.v1.TransferRun;
import java.io.IOException;

// Sample to get run details from transfer config.
public class RunDetails {

  public static void main(String[] args) throws IOException {
    // TODO(developer): Replace these variables before running the sample.
    // runId examples:
    // `projects/{project_id}/transferConfigs/{config_id}/runs/{run_id}` or
    // `projects/{project_id}/locations/{location_id}/transferConfigs/{config_id}/runs/{run_id}`
    String runId = "MY_RUN_ID";
    runDetails(runId);
  }

  public static void runDetails(String runId) throws IOException {
    try (DataTransferServiceClient dataTransferServiceClient = DataTransferServiceClient.create()) {
      GetTransferRunRequest request = GetTransferRunRequest.newBuilder().setName(runId).build();
      TransferRun run = dataTransferServiceClient.getTransferRun(request);
      System.out.print("Run details retrieved successfully :" + run.getName() + "\n");
    } catch (ApiException ex) {
      System.out.print("Run details not found." + ex.toString());
    }
  }
}

Update a transfer

Once the transfer is added, you can edit the transfer. You can edit any of the fields populated during transfer creation except for the Source:

• Destination dataset
• Display name
• Any of the parameters specified for the specific transfer type
• Run notification settings

Required permissions

Updating a transfer requires bigquery.transfers.update permissions. The bigquery.admin predefined IAM role includes bigquery.transfers.update permissions.

For more information on IAM roles in BigQuery Data Transfer Service, see Access control.

Update a transfer

To update a transfer:

Console

1. In the Google Cloud console, go to the BigQuery page.

   Go to the BigQuery page

2. Click Transfers.

3. On the Transfers page, click on an appropriate transfer in the list.

4. Click EDIT to update the transfer configuration.

bq

Enter the bq update command, provide the transfer configuration's resource name using the --transfer_config flag, and supply the --display_name, --params, --refresh_window_days, --schedule, or --target_dataset flags:

bq update \
--display_name='NAME' \
--params='PARAMETERS' \
--refresh_window_days=INTEGER \
--schedule='SCHEDULE'
--target_dataset=DATASET_ID \
--transfer_config \
RESOURCE_NAME

Replace the following:

• NAME: the display name for the transfer configuration.
• PARAMETERS: the parameters for the transfer configuration in JSON format. For example: --params='{"param1":"param_value1"}'. The following parameters are editable:
  • Campaign Manager: bucket and network_id
  • Google Ad Manager: bucket and network_code
  • Google Ads: customer_id
  • Google Merchant Center: merchant_id
  • Google Play: bucket and table_suffix
  • Scheduled Query: destination_table_kms_key, destination_table_name_template, partitioning_field, partitioning_type, query, and write_disposition
  • Search Ads 360: advertiser_id, agency_id, custom_floodlight_variables, include_removed_entities, and table_filter
  • YouTube Channel: table_suffix
  • YouTube Content Owner: content_owner_id and table_suffix
• INTEGER: a value from 0 to 30. For information on setting the refresh window, see the documentation for your transfer type.
• SCHEDULE: a recurring schedule, such as --schedule="every 3 hours". For a description of the schedule syntax, see Formatting the schedule.
• DATASET_ID: the target dataset for the transfer configuration.
• RESOURCE_NAME: the transfer's resource name (also referred to as the transfer configuration). If you do not know the transfer's resource name, find the resource name with: bq ls --transfer_config --transfer_location=location.

Note: You cannot update notification settings using the bq tool.

Examples:

The following command updates the display name, target dataset, refresh window, and parameters for Google Ads transfer projects/myproject/locations/us/transferConfigs/1234a123-1234-1a23-1be9-12ab3c456de7:

bq update \
--display_name='My changed transfer' \
--params='{"customer_id":"123-123-5678"}' \
--refresh_window_days=3 \
--target_dataset=mydataset2 \
--transfer_config \
 projects/myproject/locations/us/transferConfigs/1234a123-1234-1a23-1be9-12ab3c456de7

The following command updates the parameters and schedule for Scheduled Query transfer projects/myproject/locations/us/transferConfigs/5678z567-5678-5z67-5yx9-56zy3c866vw9:

bq update \
--params='{"destination_table_name_template":"test", "write_disposition":"APPEND"}' \
--schedule="every 24 hours" \
--transfer_config \
projects/myproject/locations/us/transferConfigs/5678z567-5678-5z67-5yx9-56zy3c866vw9

API

Use the projects.transferConfigs.patch method and supply the transfer's resource name using the transferConfig.name parameter. If you do not know the transfer's resource name, find the resource name with: bq ls --transfer_config --transfer_location=location. You can also call the following method and supply the project ID using the parent parameter to list all transfers: projects.locations.transferConfigs.list.

Java

Before trying this sample, follow the Java setup instructions in the BigQuery quickstart using client libraries. For more information, see the BigQuery Java API reference documentation.

To authenticate to BigQuery, set up Application Default Credentials. For more information, see Set up authentication for a local development environment.

View on GitHub Feedback

import com.google.api.gax.rpc.ApiException;
import com.google.cloud.bigquery.datatransfer.v1.DataTransferServiceClient;
import com.google.cloud.bigquery.datatransfer.v1.TransferConfig;
import com.google.cloud.bigquery.datatransfer.v1.UpdateTransferConfigRequest;
import com.google.protobuf.FieldMask;
import com.google.protobuf.util.FieldMaskUtil;
import java.io.IOException;

// Sample to update transfer config.
public class UpdateTransferConfig {

  public static void main(String[] args) throws IOException {
    // TODO(developer): Replace these variables before running the sample.
    String configId = "MY_CONFIG_ID";
    TransferConfig transferConfig =
        TransferConfig.newBuilder()
            .setName(configId)
            .setDisplayName("UPDATED_DISPLAY_NAME")
            .build();
    FieldMask updateMask = FieldMaskUtil.fromString("display_name");
    updateTransferConfig(transferConfig, updateMask);
  }

  public static void updateTransferConfig(TransferConfig transferConfig, FieldMask updateMask)
      throws IOException {
    try (DataTransferServiceClient dataTransferServiceClient = DataTransferServiceClient.create()) {
      UpdateTransferConfigRequest request =
          UpdateTransferConfigRequest.newBuilder()
              .setTransferConfig(transferConfig)
              .setUpdateMask(updateMask)
              .build();
      TransferConfig updateConfig = dataTransferServiceClient.updateTransferConfig(request);
      System.out.println("Transfer config updated successfully :" + updateConfig.getDisplayName());
    } catch (ApiException ex) {
      System.out.print("Transfer config was not updated." + ex.toString());
    }
  }
}

Python

Before trying this sample, follow the Python setup instructions in the BigQuery quickstart using client libraries. For more information, see the BigQuery Python API reference documentation.

To authenticate to BigQuery, set up Application Default Credentials. For more information, see Set up authentication for a local development environment.

View on GitHub Feedback

from google.cloud import bigquery_datatransfer
from google.protobuf import field_mask_pb2

transfer_client = bigquery_datatransfer.DataTransferServiceClient()

transfer_config_name = "projects/1234/locations/us/transferConfigs/abcd"
new_display_name = "My Transfer Config"

transfer_config = bigquery_datatransfer.TransferConfig(name=transfer_config_name)
transfer_config.display_name = new_display_name

transfer_config = transfer_client.update_transfer_config(
    {
        "transfer_config": transfer_config,
        "update_mask": field_mask_pb2.FieldMask(paths=["display_name"]),
    }
)

print(f"Updated config: '{transfer_config.name}'")
print(f"New display name: '{transfer_config.display_name}'")

Set up backfills and refresh runs

Refresh

Note that backfills are not available for these types of transfers. However, you can set up a refresh run.

• Cloud Storage transfers
• Copying datasets

Backfill

You can manually initiate data backfills at any time. In addition to source limits, the BigQuery Data Transfer Service supports a maximum of 180 days per backfill request. Simultaneous backfill requests are not supported.

Caution: When backfilling large date ranges, break your backfill requests into 180 day chunks, and wait for the previous backfill request to finish before creating another one.

For information on how much data is available for backfill, see the documentation for your data source:

• Amazon S3
• Campaign Manager
• Cloud Storage
• Dataset Copy
• Google Ad Manager
• Google Ads (formerly AdWords)
• When you set up a backfill for Google Ads, snapshots of Match Tables are taken once a day and stored in the partition for the latest run date. When you schedule a backfill, Match Table snapshots for the following tables are NOT updated:
  • Ad
  • AdGroup
  • AgeRange
  • Audience
  • BidGoal
  • Budget
  • Campaign
  • Criteria
  • Customer
  • Gender
  • Keyword
  • LocationBasedCampaignCriterion
  • ParentalStatus
  • Placement
  • Video
• Google Ads (preview)
• Google Merchant Center
• Google Play
• Scheduled queries
• Search Ads 360
• When you set up a backfill for Search Ads 360, snapshots of Match Tables are taken once a day and stored in the partition for the latest run date. When you schedule a backfill, Match Table snapshots for the following tables are NOT updated:
  • Account
  • Ad
  • Advertiser
  • AdGroup
  • AdGroupTarget
  • BidStrategy
  • Campaign
  • CampaignTarget
  • FeedItem
  • FloodlightActivity
  • Keyword
  • NegativeAdGroupKeyword
  • NegativeAdGroupTarget
  • NegativeCampaignKeyword
  • NegativeCampaignTarget
  • ProductAdvertised
  • ProductGroup
  • ProductLeadAndCrossSellFloodlightStats
  • Video
  • Placement
  • Video
• YouTube Channel
• YouTube Content Owner
• Teradata
• Amazon Redshift

Note that backfills are not available for Cloud Storage transfers. However, you can set up a refresh run of a Cloud Storage transfer.

Required permissions

Scheduling a backfill requires bigquery.transfers.update permissions. The bigquery.admin predefined IAM role includes bigquery.transfers.update permissions.

For more information on IAM roles in BigQuery Data Transfer Service, see Access control.

Schedule a backfill

To schedule a backfill:

Console

1. In the Google Cloud console, go to the BigQuery page.

   Go to the BigQuery page

2. Click Transfers.

3. On the Transfers page, click on an appropriate transfer in the list.

4. Click MORE menu, and then select Schedule backfill.

5. In the Schedule backfill dialog, choose your Start date and End date.

   

bq

Enter the bq mk command, supply the --transfer_run flag, and specify the --start_time and --end_time.

bq mk \
--transfer_run \
--start_time='start_time' \
--end_time='end_time' \
resource_name

Where:

• start_time and end_time are timestamps that end in Z or contain a valid time zone offset. For example:

  • 2017-08-19T12:11:35.00Z
  • 2017-05-25T00:00:00+00:00

• resource_name is the transfer's resource name (also referred to as the transfer configuration). If you do not know the transfer's resource name, find the resource name with: bq ls --transfer_config --transfer_location=location.

Examples:

Enter the following command to schedule a backfill for transfer configuration projects/myproject/locations/us/transferConfigs/1234a123-1234-1a23-1be9-12ab3c456de7.

bq mk \
--transfer_run \
--start_time 2017-05-25T00:00:00Z \
--end_time 2017-05-25T00:00:00Z \
projects/myproject/locations/us/transferConfigs/1234a123-1234-1a23-1be9-12ab3c456de7

API

Use the projects.locations.transferConfigs.startManualRuns method and provide the transfer configuration resource using the parent parameter.

Java

Before trying this sample, follow the Java setup instructions in the BigQuery quickstart using client libraries. For more information, see the BigQuery Java API reference documentation.

To authenticate to BigQuery, set up Application Default Credentials. For more information, see Set up authentication for a local development environment.

View on GitHub Feedback

import com.google.api.gax.rpc.ApiException;
import com.google.cloud.bigquery.datatransfer.v1.DataTransferServiceClient;
import com.google.cloud.bigquery.datatransfer.v1.ScheduleTransferRunsRequest;
import com.google.cloud.bigquery.datatransfer.v1.ScheduleTransferRunsResponse;
import com.google.protobuf.Timestamp;
import java.io.IOException;
import org.threeten.bp.Clock;
import org.threeten.bp.Instant;
import org.threeten.bp.temporal.ChronoUnit;

// Sample to run schedule back fill for transfer config
public class ScheduleBackFill {

  public static void main(String[] args) throws IOException {
    // TODO(developer): Replace these variables before running the sample.
    String configId = "MY_CONFIG_ID";
    Clock clock = Clock.systemDefaultZone();
    Instant instant = clock.instant();
    Timestamp startTime =
        Timestamp.newBuilder()
            .setSeconds(instant.minus(5, ChronoUnit.DAYS).getEpochSecond())
            .setNanos(instant.minus(5, ChronoUnit.DAYS).getNano())
            .build();
    Timestamp endTime =
        Timestamp.newBuilder()
            .setSeconds(instant.minus(2, ChronoUnit.DAYS).getEpochSecond())
            .setNanos(instant.minus(2, ChronoUnit.DAYS).getNano())
            .build();
    scheduleBackFill(configId, startTime, endTime);
  }

  public static void scheduleBackFill(String configId, Timestamp startTime, Timestamp endTime)
      throws IOException {
    try (DataTransferServiceClient client = DataTransferServiceClient.create()) {
      ScheduleTransferRunsRequest request =
          ScheduleTransferRunsRequest.newBuilder()
              .setParent(configId)
              .setStartTime(startTime)
              .setEndTime(endTime)
              .build();
      ScheduleTransferRunsResponse response = client.scheduleTransferRuns(request);
      System.out.println("Schedule backfill run successfully :" + response.getRunsCount());
    } catch (ApiException ex) {
      System.out.print("Schedule backfill was not run." + ex.toString());
    }
  }
}

Python

Before trying this sample, follow the Python setup instructions in the BigQuery quickstart using client libraries. For more information, see the BigQuery Python API reference documentation.

To authenticate to BigQuery, set up Application Default Credentials. For more information, see Set up authentication for a local development environment.

View on GitHub Feedback

import datetime

from google.cloud import bigquery_datatransfer

transfer_client = bigquery_datatransfer.DataTransferServiceClient()

transfer_config_name = "projects/1234/locations/us/transferConfigs/abcd"
now = datetime.datetime.now(datetime.timezone.utc)
start_time = now - datetime.timedelta(days=5)
end_time = now - datetime.timedelta(days=2)

# Some data sources, such as scheduled_query only support daily run.
# Truncate start_time and end_time to midnight time (00:00AM UTC).
start_time = datetime.datetime(
    start_time.year, start_time.month, start_time.day, tzinfo=datetime.timezone.utc
)
end_time = datetime.datetime(
    end_time.year, end_time.month, end_time.day, tzinfo=datetime.timezone.utc
)

response = transfer_client.schedule_transfer_runs(
    parent=transfer_config_name,
    start_time=start_time,
    end_time=end_time,
)

print("Started transfer runs:")
for run in response.runs:
    print(f"backfill: {run.run_time} run: {run.name}")

Caution: Depending on the data source, and the amount of data you request, backfills may take several hours or days to complete.

Update credentials

A transfer uses the credentials of the user that created it. If you need to change the user attached to a transfer configuration, you can update the transfer's credentials. This is useful if the user who created the transfer is no longer with your organization.

Required permissions

Updating credentials requires bigquery.transfers.update permissions. The bigquery.admin predefined IAM role includes bigquery.transfers.update permissions.

For more information on IAM roles in BigQuery Data Transfer Service, see Access control.

Update a transfer's credentials

To update the credentials for a transfer:

Console

1. In the Google Cloud console, go to the BigQuery page.

   Go to the BigQuery page

2. Click Transfers.

3. On the Transfers page, click on an appropriate transfer in the list.

4. Click MORE menu, and then select Refresh credentials.

5. Sign into your Google account and click Allow to give the BigQuery Data Transfer Service permission to view your reporting data and to access and manage the data in BigQuery.

bq

Enter the bq update command, provide the transfer configuration's resource name using the --transfer_config flag, and supply the --update_credentials flag.

bq update \
--update_credentials=boolean \
--transfer_config \
resource_name

Where:

• boolean is a boolean value indicating whether the credentials should be updated for the transfer configuration.
• resource_name is the transfer's resource name (also referred to as the transfer configuration). If you do not know the transfer's resource name, find the resource name with: bq ls --transfer_config --transfer_location=location.

Examples:

Enter the following command to update the credentials for Google Ads transfer projects/myproject/locations/us/transferConfigs/1234a123-1234-1a23-1be9-12ab3c456de7.

bq update \
--update_credentials=true \
--transfer_config \
projects/myproject/locations/us/transferConfigs/1234a123-1234-1a23-1be9-12ab3c456de7

API

Use the projects.transferConfigs.patch method and supply the authorizationCode and updateMask parameters.

Java

Before trying this sample, follow the Java setup instructions in the BigQuery quickstart using client libraries. For more information, see the BigQuery Java API reference documentation.

To authenticate to BigQuery, set up Application Default Credentials. For more information, see Set up authentication for a local development environment.

View on GitHub Feedback

import com.google.api.gax.rpc.ApiException;
import com.google.cloud.bigquery.datatransfer.v1.DataTransferServiceClient;
import com.google.cloud.bigquery.datatransfer.v1.TransferConfig;
import com.google.cloud.bigquery.datatransfer.v1.UpdateTransferConfigRequest;
import com.google.protobuf.FieldMask;
import com.google.protobuf.util.FieldMaskUtil;
import java.io.IOException;

// Sample to update credentials in transfer config.
public class UpdateCredentials {

  public static void main(String[] args) throws IOException {
    // TODO(developer): Replace these variables before running the sample.
    String configId = "MY_CONFIG_ID";
    String serviceAccount = "MY_SERVICE_ACCOUNT";
    TransferConfig transferConfig = TransferConfig.newBuilder().setName(configId).build();
    FieldMask updateMask = FieldMaskUtil.fromString("service_account_name");
    updateCredentials(transferConfig, serviceAccount, updateMask);
  }

  public static void updateCredentials(
      TransferConfig transferConfig, String serviceAccount, FieldMask updateMask)
      throws IOException {
    try (DataTransferServiceClient dataTransferServiceClient = DataTransferServiceClient.create()) {
      UpdateTransferConfigRequest request =
          UpdateTransferConfigRequest.newBuilder()
              .setTransferConfig(transferConfig)
              .setUpdateMask(updateMask)
              .setServiceAccountName(serviceAccount)
              .build();
      dataTransferServiceClient.updateTransferConfig(request);
      System.out.println("Credentials updated successfully");
    } catch (ApiException ex) {
      System.out.print("Credentials was not updated." + ex.toString());
    }
  }
}

Python

Before trying this sample, follow the Python setup instructions in the BigQuery quickstart using client libraries. For more information, see the BigQuery Python API reference documentation.

To authenticate to BigQuery, set up Application Default Credentials. For more information, see Set up authentication for a local development environment.

View on GitHub Feedback

from google.cloud import bigquery_datatransfer
from google.protobuf import field_mask_pb2

transfer_client = bigquery_datatransfer.DataTransferServiceClient()

service_account_name = "[e-mail]"
transfer_config_name = "projects/1234/locations/us/transferConfigs/abcd"

transfer_config = bigquery_datatransfer.TransferConfig(name=transfer_config_name)

transfer_config = transfer_client.update_transfer_config(
    {
        "transfer_config": transfer_config,
        "update_mask": field_mask_pb2.FieldMask(paths=["service_account_name"]),
        "service_account_name": service_account_name,
    }
)

print("Updated config: '{}'".format(transfer_config.name))

Disable a transfer

When you disable a transfer, disabled is added to the transfer name. When the transfer is disabled, no new transfer runs are scheduled, and no new backfills are allowed. Any transfer runs in progress are completed.

Disabling a transfer does not remove any data already transferred to BigQuery. Data previously transferred incurs standard BigQuery storage costs until you delete the dataset or delete the tables.

Required permissions

Disabling a transfer requires bigquery.transfers.update permissions. The bigquery.admin predefined IAM role includes bigquery.transfers.update permissions.

For more information on IAM roles in BigQuery Data Transfer Service, see Access control.

Disabling a transfer

To disable a transfer:

Console

1. In the Google Cloud console, go to the BigQuery page.

   Go to BigQuery

2. Click Transfers.

3. On the Transfers page, click on the transfer in the list that you want to disable.

4. Click on DISABLE. To re-enable the transfer, click on ENABLE.

bq

Disabling a transfer is not supported by the CLI.

API

Use the projects.locations.transferConfigs.patch method and set disabled to true in the projects.locations.transferConfig resource.

Java

Before trying this sample, follow the Java setup instructions in the BigQuery quickstart using client libraries. For more information, see the BigQuery Java API reference documentation.

To authenticate to BigQuery, set up Application Default Credentials. For more information, see Set up authentication for a local development environment.

View on GitHub Feedback

import com.google.api.gax.rpc.ApiException;
import com.google.cloud.bigquery.datatransfer.v1.DataTransferServiceClient;
import com.google.cloud.bigquery.datatransfer.v1.TransferConfig;
import com.google.cloud.bigquery.datatransfer.v1.UpdateTransferConfigRequest;
import com.google.protobuf.FieldMask;
import com.google.protobuf.util.FieldMaskUtil;
import java.io.IOException;

// Sample to disable transfer config.
public class DisableTransferConfig {

  public static void main(String[] args) throws IOException {
    // TODO(developer): Replace these variables before running the sample.
    String configId = "MY_CONFIG_ID";
    TransferConfig transferConfig =
        TransferConfig.newBuilder().setName(configId).setDisabled(true).build();
    FieldMask updateMask = FieldMaskUtil.fromString("disabled");
    disableTransferConfig(transferConfig, updateMask);
  }

  public static void disableTransferConfig(TransferConfig transferConfig, FieldMask updateMask)
      throws IOException {
    try (DataTransferServiceClient dataTransferServiceClient = DataTransferServiceClient.create()) {
      UpdateTransferConfigRequest request =
          UpdateTransferConfigRequest.newBuilder()
              .setTransferConfig(transferConfig)
              .setUpdateMask(updateMask)
              .build();
      TransferConfig updateConfig = dataTransferServiceClient.updateTransferConfig(request);
      System.out.println("Transfer config disabled successfully :" + updateConfig.getDisplayName());
    } catch (ApiException ex) {
      System.out.print("Transfer config was not disabled." + ex.toString());
    }
  }
}

To re-enable the transfer:

Before trying this sample, follow the Java setup instructions in the BigQuery quickstart using client libraries. For more information, see the BigQuery Java API reference documentation.

To authenticate to BigQuery, set up Application Default Credentials. For more information, see Set up authentication for a local development environment.

View on GitHub Feedback

/*
 * Copyright 2020 Google LLC
 *
 * Licensed under the Apache License, Version 2.0 (the "License");
 * you may not use this file except in compliance with the License.
 * You may obtain a copy of the License at
 *
 * http://www.apache.org/licenses/LICENSE-2.0
 *
 * Unless required by applicable law or agreed to in writing, software
 * distributed under the License is distributed on an "AS IS" BASIS,
 * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
 * See the License for the specific language governing permissions and
 * limitations under the License.
 */

package com.example.bigquerydatatransfer;

import com.google.api.gax.rpc.ApiException;
import com.google.cloud.bigquery.datatransfer.v1.DataTransferServiceClient;
import com.google.cloud.bigquery.datatransfer.v1.TransferConfig;
import com.google.cloud.bigquery.datatransfer.v1.UpdateTransferConfigRequest;
import com.google.protobuf.FieldMask;
import com.google.protobuf.util.FieldMaskUtil;
import java.io.IOException;

// Sample to disable transfer config.
public class DisableTransferConfig {

  public static void main(String[] args) throws IOException {
    // TODO(developer): Replace these variables before running the sample.
    String configId = "MY_CONFIG_ID";
    TransferConfig transferConfig =
        TransferConfig.newBuilder().setName(configId).setDisabled(true).build();
    FieldMask updateMask = FieldMaskUtil.fromString("disabled");
    disableTransferConfig(transferConfig, updateMask);
  }

  public static void disableTransferConfig(TransferConfig transferConfig, FieldMask updateMask)
      throws IOException {
    try (DataTransferServiceClient dataTransferServiceClient = DataTransferServiceClient.create()) {
      UpdateTransferConfigRequest request =
          UpdateTransferConfigRequest.newBuilder()
              .setTransferConfig(transferConfig)
              .setUpdateMask(updateMask)
              .build();
      TransferConfig updateConfig = dataTransferServiceClient.updateTransferConfig(request);
      System.out.println("Transfer config disabled successfully :" + updateConfig.getDisplayName());
    } catch (ApiException ex) {
      System.out.print("Transfer config was not disabled." + ex.toString());
    }
  }
}

Delete a transfer

When a transfer is deleted, no new transfer runs are scheduled. Any transfer runs in progress are stopped.

Deleting a transfer does not remove any data already transferred to BigQuery. Data previously transferred incurs standard BigQuery storage costs until you delete the dataset or delete the tables.

Required permissions

Deleting a transfer requires bigquery.transfers.update permissions. The bigquery.admin predefined IAM role includes bigquery.transfers.update permissions.

For more information on IAM roles in BigQuery Data Transfer Service, see Access control.

Deleting a transfer

To delete a transfer:

Console

1. In the Google Cloud console, go to the BigQuery page.

   Go to BigQuery

2. Click Transfers.

3. On the Transfers page, click on the transfer in the list that you want to delete.

4. Click on DELETE. As a safety measure you will need to type the word "delete" into a box to confirm your intention.

   Caution: The delete operation cannot be undone.

bq

Enter the bq rm command and provide the transfer configuration's resource name. You can use the -f flag to delete a transfer config without confirmation.

bq rm \
-f \
--transfer_config \
resource_name

Where:

• resource_name is the transfer's Resource Name which is also referred to as the transfer configuration). If you do not know the transfer's Resource Name, issue the bq ls --transfer_config --transfer_location=location command to list all transfers.

For example, enter the following command to delete transfer configuration projects/myproject/locations/us/transferConfigs/1234a123-1234-1a23-1be9-12ab3c456de7.

bq rm \
--transfer_config \
projects/myproject/locations/us/transferConfigs/1234a123-1234-1a23-1be9-12ab3c456de7

API

Use the projects.locations.transferConfigs.delete method and supply the resource you're deleting using the name parameter.

Java

Before trying this sample, follow the Java setup instructions in the BigQuery quickstart using client libraries. For more information, see the BigQuery Java API reference documentation.

To authenticate to BigQuery, set up Application Default Credentials. For more information, see Set up authentication for a local development environment.

View on GitHub Feedback

import com.google.api.gax.rpc.ApiException;
import com.google.cloud.bigquery.datatransfer.v1.DataTransferServiceClient;
import com.google.cloud.bigquery.datatransfer.v1.DeleteTransferConfigRequest;
import java.io.IOException;

// Sample to delete a transfer config
public class DeleteTransferConfig {

  public static void main(String[] args) throws IOException {
    // TODO(developer): Replace these variables before running the sample.
    // i.e projects/{project_id}/transferConfigs/{config_id}` or
    // `projects/{project_id}/locations/{location_id}/transferConfigs/{config_id}`
    String configId = "MY_CONFIG_ID";
    deleteTransferConfig(configId);
  }

  public static void deleteTransferConfig(String configId) throws IOException {
    try (DataTransferServiceClient dataTransferServiceClient = DataTransferServiceClient.create()) {
      DeleteTransferConfigRequest request =
          DeleteTransferConfigRequest.newBuilder().setName(configId).build();
      dataTransferServiceClient.deleteTransferConfig(request);
      System.out.println("Transfer config deleted successfully");
    } catch (ApiException ex) {
      System.out.println("Transfer config was not deleted." + ex.toString());
    }
  }
}

Python

Before trying this sample, follow the Python setup instructions in the BigQuery quickstart using client libraries. For more information, see the BigQuery Python API reference documentation.

To authenticate to BigQuery, set up Application Default Credentials. For more information, see Set up authentication for a local development environment.

View on GitHub Feedback