BigQuery Data Transfer client libraries

This page shows how to get started with the Cloud Client Libraries for the BigQuery Data Transfer API. Read more about the client libraries for Cloud APIs, including the older Google API Client Libraries, in Client Libraries Explained.

Installing the client library


For more information, see Setting Up a C# Development Environment.

Install-Package Google.Cloud.BigQuery.DataTransfer.V1 -Pre


For more information, see Setting Up a Go Development Environment.

go get -u


For more information, see Setting Up a Java Development Environment.

If you are using Maven, add the following to your pom.xml file. For more information about BOMs, see The Google Cloud Platform Libraries BOM.



If you are using Gradle, add the following to your dependencies:

implementation platform('')

implementation ''

If you are using sbt, add the following to your dependencies:

libraryDependencies += "" % "google-cloud-bigquerydatatransfer" % "2.0.20"

If you're using Visual Studio Code, IntelliJ, or Eclipse, you can add client libraries to your project using the following IDE plugins:

The plugins provide additional functionality, such as key management for service accounts. Refer to each plugin's documentation for details.


For more information, see Setting Up a Node.js Development Environment.

npm install --save @google-cloud/bigquery-data-transfer


For more information, see Using PHP on Google Cloud.

composer require google/cloud-bigquerydatatransfer


For more information, see Setting Up a Python Development Environment.

pip install --upgrade google-cloud-bigquery-datatransfer


For more information, see Setting Up a Ruby Development Environment.

gem install google-cloud-bigquery-data_transfer

Setting up authentication

To run the client library, you must first set up authentication. One way to do that is to create a service account and set an environment variable, as shown in the following steps. For other ways to authenticate, see Authenticating as a service account.

Cloud Console

Create a service account:

  1. In the Cloud Console, go to the Create service account page.

    Go to Create service account
  2. Select a project.
  3. In the Service account name field, enter a name. The Cloud Console fills in the Service account ID field based on this name.

    In the Service account description field, enter a description. For example, Service account for quickstart.

  4. Click Create and continue.
  5. Click the Select a role field.

    Under Quick access, click Basic, then click Owner.

  6. Click Continue.
  7. Click Done to finish creating the service account.

    Do not close your browser window. You will use it in the next step.

Create a service account key:

  1. In the Cloud Console, click the email address for the service account that you created.
  2. Click Keys.
  3. Click Add key, then click Create new key.
  4. Click Create. A JSON key file is downloaded to your computer.
  5. Click Close.

Command line

You can run the following commands using the Cloud SDK on your local machine, or in Cloud Shell.

  1. Create the service account. Replace NAME with a name for the service account.

    gcloud iam service-accounts create NAME
  2. Grant permissions to the service account. Replace PROJECT_ID with your project ID.

    gcloud projects add-iam-policy-binding PROJECT_ID --member="" --role="roles/owner"
  3. Generate the key file. Replace FILE_NAME with a name for the key file.

    gcloud iam service-accounts keys create FILE_NAME.json

Provide authentication credentials to your application code by setting the environment variable GOOGLE_APPLICATION_CREDENTIALS. This variable only applies to your current shell session, so if you open a new session, set the variable again.

Linux or macOS


Replace KEY_PATH with the path of the JSON file that contains your service account key.

For example:

export GOOGLE_APPLICATION_CREDENTIALS="/home/user/Downloads/service-account-file.json"


For PowerShell:


Replace KEY_PATH with the path of the JSON file that contains your service account key.

For example:


For command prompt:


Replace KEY_PATH with the path of the JSON file that contains your service account key.

Using the client library

The following example shows how to use the client library.


using System;
using Google.Api.Gax;
using Google.Cloud.BigQuery.DataTransfer.V1;

namespace GoogleCloudSamples
    public class QuickStart
        public static void Main(string[] args)
            // Instantiates a client
            DataTransferServiceClient client = DataTransferServiceClient.Create();

            // Your Google Cloud Platform project ID
            string projectId = "YOUR-PROJECT-ID";

            ProjectName project = new ProjectName(projectId);
            var sources = client.ListDataSources(ParentNameOneof.From(project));
            Console.WriteLine("Supported Data Sources:");
            foreach (DataSource source in sources)
                    $"{source.DataSourceId}: " +
                    $"{source.DisplayName} ({source.Description})");


// Sample bigquery-quickstart creates a Google BigQuery dataset.
package main

import (


	// Imports the BigQuery Data Transfer client package.
	datatransfer ""
	datatransferpb ""

func main() {
	ctx := context.Background()

	// Sets your Google Cloud Platform project ID.
	projectID := "YOUR_PROJECT_ID"

	// Creates a client.
	client, err := datatransfer.NewClient(ctx)
	if err != nil {
		log.Fatalf("Failed to create client: %v", err)

	req := &datatransferpb.ListDataSourcesRequest{
		Parent: fmt.Sprintf("projects/%s", projectID),
	it := client.ListDataSources(ctx, req)
	fmt.Println("Supported Data Sources:")
	for {
		ds, err := it.Next()
		if err == iterator.Done {
		if err != nil {
			log.Fatalf("Failed to list sources: %v", err)
		fmt.Println("\tID: ", ds.DataSourceId)
		fmt.Println("\tFull path: ", ds.Name)
		fmt.Println("\tDescription: ", ds.Description)


// Imports the Google Cloud client library


public class QuickstartSample {
   * List available data sources for the BigQuery Data Transfer service.
  public static void main(String... args) throws Exception {
    // Sets your Google Cloud Platform project ID.
    // String projectId = "YOUR_PROJECT_ID";
    String projectId = args[0];

    // Instantiate a client. If you don't specify credentials when constructing a client, the
    // client library will look for credentials in the environment, such as the
    // GOOGLE_APPLICATION_CREDENTIALS environment variable.
    try (DataTransferServiceClient client = DataTransferServiceClient.create()) {
      // Request the list of available data sources.
      String parent = String.format("projects/%s", projectId);
      ListDataSourcesRequest request =
      ListDataSourcesPagedResponse response = client.listDataSources(request);

      // Print the results.
      System.out.println("Supported Data Sources:");
      for (DataSource dataSource : response.iterateAll()) {
        System.out.printf("\tID: %s%n", dataSource.getDataSourceId());
        System.out.printf("\tFull path: %s%n", dataSource.getName());
        System.out.printf("\tDescription: %s%n", dataSource.getDescription());


const bigqueryDataTransfer = require('@google-cloud/bigquery-data-transfer');
const client = new bigqueryDataTransfer.v1.DataTransferServiceClient();

async function quickstart() {
  const projectId = await client.getProjectId();

  // Iterate over all elements.
  const formattedParent = client.projectPath(projectId, 'us-central1');
  let nextRequest = {parent: formattedParent};
  const options = {autoPaginate: false};
  console.log('Data sources:');
  do {
    // Fetch the next page.
    const responses = await client.listDataSources(nextRequest, options);
    // The actual resources in a response.
    const resources = responses[0];
    // The next request if the response shows that there are more responses.
    nextRequest = responses[1];
    // The actual response object, if necessary.
    // const rawResponse = responses[2];
    resources.forEach(resource => {
      console.log(`  ${}`);
  } while (nextRequest);

  console.log('Sources via stream:');

    .listDataSourcesStream({parent: formattedParent})
    .on('data', element => {
      console.log(`  ${}`);


# Includes the autoloader for libraries installed with composer
require __DIR__ . '/vendor/autoload.php';

# Imports the Google Cloud client library
use Google\Cloud\BigQuery\DataTransfer\V1\DataTransferServiceClient;

# Instantiates a client
$bqdtsClient = new DataTransferServiceClient();

# Your Google Cloud Platform project ID
$projectId = 'YOUR_PROJECT_ID';
$parent = sprintf('projects/%s/locations/us', $projectId);

try {
    echo 'Supported Data Sources:', PHP_EOL;
    $pagedResponse = $bqdtsClient->listDataSources($parent);
    foreach ($pagedResponse->iterateAllElements() as $dataSource) {
        echo 'Data source: ', $dataSource->getDisplayName(), PHP_EOL;
        echo 'ID: ', $dataSource->getDataSourceId(), PHP_EOL;
        echo 'Full path: ', $dataSource->getName(), PHP_EOL;
        echo 'Description: ', $dataSource->getDescription(), PHP_EOL;
} finally {


from import bigquery_datatransfer

client = bigquery_datatransfer.DataTransferServiceClient()

# TODO: Update to your project ID.
project_id = "my-project"

# Get the full path to your project.
parent = client.common_project_path(project_id)

print("Supported Data Sources:")

# Iterate over all possible data sources.
for data_source in client.list_data_sources(parent=parent):
    print("\tID: {}".format(data_source.data_source_id))
    print("\tFull path: {}".format(
    print("\tDescription: {}".format(data_source.description))


# Imports the Google Cloud client library
require "google/cloud/bigquery/data_transfer"

# Your Google Cloud Platform project ID
# project_id = "YOUR_PROJECT_ID"

# Instantiate a client
data_transfer = Google::Cloud::Bigquery::DataTransfer.data_transfer_service

# Get the full path to your project.
project_path = data_transfer.project_path project: project_id

puts "Supported Data Sources:"

# Iterate over all possible data sources.
data_transfer.list_data_sources(parent: project_path).each do |data_source|
  puts "Data source: #{data_source.display_name}"
  puts "ID: #{data_source.data_source_id}"
  puts "Full path: #{}"
  puts "Description: #{data_source.description}"

Additional resources