BigQuery Data Transfer Client Libraries

This page shows how to get started with the Cloud Client Libraries for the BigQuery Data Transfer API. Read more about the client libraries for Cloud APIs, including the older Google APIs Client Libraries, in Client Libraries Explained.

Alpha

This is an alpha release of the Cloud Client Libraries for the BigQuery Data Transfer API. These libraries might be changed in backward-incompatible ways and are not recommended for production use. They are not subject to any SLA or deprecation policy.

Installing the client library

C#

For more information, see Setting Up a C# Development Environment.

Install-Package Google.Cloud.BigQuery.DataTransfer.V1 -Pre

Go

go get -u cloud.google.com/go/bigquery/datatransfer/apiv1

Java

For more information, see Setting Up a Java Development Environment. If you are using Maven, add this to your pom.xml file:

<dependency>
  <groupId>com.google.cloud</groupId>
  <artifactId>google-cloud-bigquerydatatransfer</artifactId>
  <version>0.58.0-beta</version>
</dependency>

If you are using Gradle, add this to your dependencies:

compile 'com.google.cloud:google-cloud-bigquerydatatransfer:0.58.0-beta'

If you are using SBT, add this to your dependencies:

libraryDependencies += "com.google.cloud" % "google-cloud-bigquerydatatransfer" % "0.58.0-beta"

Note: Cloud Java client libraries do not currently support Android.

Node.js

For more information, see Setting Up a Node.js Development Environment.

npm install --save @google-cloud/bigquery-data-transfer

PHP

composer require google/cloud-bigquerydatatransfer

Python

For more information, see Setting Up a Python Development Environment.

pip install --upgrade google-cloud-bigquery-datatransfer

Ruby

For more information, see Setting Up a Ruby Development Environment.

gem install google-cloud-bigquery-data_transfer

Setting up authentication

To run the client library, you must first set up authentication by creating a service account and setting an environment variable.

GCP Console

1. Go to the Create service account key page in the GCP Console.

   Go to the Create Service Account Key page
2. From the Service account drop-down list, select New service account.
3. Enter a name into the Service account name field.

4. From the Role drop-down list, select Project > Owner.

   Note: The Role field authorizes your service account to access resources. You can view and change this field later using GCP Console. If you are developing a production application, specify more granular permissions than Project > Owner. For more information, see granting roles to service accounts.
5. Click Create. A JSON file that contains your key downloads to your computer.

Command line

You can run the following commands using the Cloud SDK on your local machine, or within Cloud Shell.

1. Create the service account. Replace [NAME] with your desired service account name.

   gcloud iam service-accounts create [NAME]

2. Grant permissions to the service account. Replace [PROJECT_ID] with your project ID.

   gcloud projects add-iam-policy-binding [PROJECT_ID] --member "serviceAccount:[NAME]@[PROJECT_ID].iam.gserviceaccount.com" --role "roles/owner"

   Note: The Role field authorizes your service account to access resources. You can view and change this field later using GCP Console. If you are developing a production application, specify more granular permissions than Project > Owner. For more information, see granting roles to service accounts.

3. Generate the key file. Replace [FILE_NAME] with a name for the key file.

   gcloud iam service-accounts keys create [FILE_NAME].json --iam-account [NAME]@[PROJECT_ID].iam.gserviceaccount.com

Provide authentication credentials to your application code by setting the environment variable GOOGLE_APPLICATION_CREDENTIALS. Replace [PATH] with the file path of the JSON file that contains your service account key, and [FILE_NAME] with the filename. This variable only applies to your current shell session, so if you open a new session, set the variable again.

Linux or macOS

export GOOGLE_APPLICATION_CREDENTIALS="[PATH]"

For example:

export GOOGLE_APPLICATION_CREDENTIALS="/home/user/Downloads/[FILE_NAME].json"

Windows

With PowerShell:

$env:GOOGLE_APPLICATION_CREDENTIALS="[PATH]"

For example:

$env:GOOGLE_APPLICATION_CREDENTIALS="C:\Users\username\Downloads\[FILE_NAME].json"

With command prompt:

set GOOGLE_APPLICATION_CREDENTIALS=[PATH]

Using the client library

The following example shows how to use the client library.

C#

Before trying this sample, follow the C# setup instructions in the BigQuery Data Transfer Quickstart Using Client Libraries . For more information, see the BigQuery Data Transfer C# API reference documentation .

View on GitHub Feedback

using System;
using Google.Api.Gax;
using Google.Cloud.BigQuery.DataTransfer.V1;

namespace GoogleCloudSamples
{
    public class QuickStart
    {
        public static void Main(string[] args)
        {
            // Instantiates a client
            DataTransferServiceClient client = DataTransferServiceClient.Create();

            // Your Google Cloud Platform project ID
            string projectId = "YOUR-PROJECT-ID";

            ProjectName project = new ProjectName(projectId);
            var sources = client.ListDataSources(ParentNameOneof.From(project));
            Console.WriteLine("Supported Data Sources:");
            foreach (DataSource source in sources)
            {
                Console.WriteLine(
                    $"{source.DataSourceId}: " +
                    $"{source.DisplayName} ({source.Description})");
            }
        }
    }
}

Go

Before trying this sample, follow the Go setup instructions in the BigQuery Data Transfer Quickstart Using Client Libraries . For more information, see the BigQuery Data Transfer Go API reference documentation .

View on GitHub Feedback

// Sample bigquery-quickstart creates a Google BigQuery dataset.
package main

import (
	"fmt"
	"log"

	"golang.org/x/net/context"
	"google.golang.org/api/iterator"

	// Imports the BigQuery Data Transfer client package.
	datatransfer "cloud.google.com/go/bigquery/datatransfer/apiv1"
	datatransferpb "google.golang.org/genproto/googleapis/cloud/bigquery/datatransfer/v1"
)

func main() {
	ctx := context.Background()

	// Sets your Google Cloud Platform project ID.
	projectID := "YOUR_PROJECT_ID"

	// Creates a client.
	client, err := datatransfer.NewClient(ctx)
	if err != nil {
		log.Fatalf("Failed to create client: %v", err)
	}

	req := &datatransferpb.ListDataSourcesRequest{
		Parent: fmt.Sprintf("projects/%s", projectID),
	}
	it := client.ListDataSources(ctx, req)
	fmt.Println("Supported Data Sources:")
	for {
		ds, err := it.Next()
		if err == iterator.Done {
			break
		}
		if err != nil {
			log.Fatalf("Failed to list sources: %v", err)
		}
		fmt.Println(ds.DisplayName)
		fmt.Println("\tID: ", ds.DataSourceId)
		fmt.Println("\tFull path: ", ds.Name)
		fmt.Println("\tDescription: ", ds.Description)
	}
}

Java

Before trying this sample, follow the Java setup instructions in the BigQuery Data Transfer Quickstart Using Client Libraries . For more information, see the BigQuery Data Transfer Java API reference documentation .

View on GitHub Feedback

// Imports the Google Cloud client library

import com.google.cloud.bigquery.datatransfer.v1.DataSource;
import com.google.cloud.bigquery.datatransfer.v1.DataTransferServiceClient;
import com.google.cloud.bigquery.datatransfer.v1.DataTransferServiceClient.ListDataSourcesPagedResponse;
import com.google.cloud.bigquery.datatransfer.v1.ListDataSourcesRequest;

public class QuickstartSample {
  /**
   * List available data sources for the BigQuery Data Transfer service.
   */
  public static void main(String... args) throws Exception {
    // Sets your Google Cloud Platform project ID.
    // String projectId = "YOUR_PROJECT_ID";
    String projectId = args[0];

    // Instantiate a client. If you don't specify credentials when constructing a client, the
    // client library will look for credentials in the environment, such as the
    // GOOGLE_APPLICATION_CREDENTIALS environment variable.
    try (DataTransferServiceClient client = DataTransferServiceClient.create()) {
      // Request the list of available data sources.
      String parent = String.format("projects/%s", projectId);
      ListDataSourcesRequest request =
          ListDataSourcesRequest.newBuilder()
              .setParent(parent)
              .build();
      ListDataSourcesPagedResponse response = client.listDataSources(request);

      // Print the results.
      System.out.println("Supported Data Sources:");
      for (DataSource dataSource : response.iterateAll()) {
        System.out.println(dataSource.getDisplayName());
        System.out.printf("\tID: %s%n", dataSource.getDataSourceId());
        System.out.printf("\tFull path: %s%n", dataSource.getName());
        System.out.printf("\tDescription: %s%n", dataSource.getDescription());
      }
    }
  }
}

Node.js

Before trying this sample, follow the Node.js setup instructions in the BigQuery Data Transfer Quickstart Using Client Libraries . For more information, see the BigQuery Data Transfer Node.js API reference documentation .

View on GitHub Feedback

if (
  !process.env.GCLOUD_PROJECT ||
  !process.env.GOOGLE_APPLICATION_CREDENTIALS
) {
  throw new Error(
    'Usage: GCLOUD_PROJECT=<project_id> GOOGLE_APPLICATION_CREDENTIALS=<path to key json file> node #{$0}'
  );
}

const bigqueryDataTransfer = require('@google-cloud/bigquery-data-transfer');

const client = new bigqueryDataTransfer.v1.DataTransferServiceClient({
  // optional auth parameters.
});
const projectId = process.env.GCLOUD_PROJECT;

// Iterate over all elements.
const formattedParent = client.locationPath(projectId, 'us-central1');

client.listDataSources({parent: formattedParent}).then(responses => {
  const resources = responses[0];
  for (let i = 0; i < resources.length; i += 1) {
    console.log(resources[i]);
  }
});

const options = {autoPaginate: false};
const callback = responses => {
  // The actual resources in a response.
  const resources = responses[0];
  // The next request if the response shows that there are more responses.
  const nextRequest = responses[1];
  // The actual response object, if necessary.
  // const rawResponse = responses[2];
  for (let i = 0; i < resources.length; i += 1) {
    console.log(resources[i]);
  }
  if (nextRequest) {
    // Fetch the next page.
    return client.listDataSources(nextRequest, options).then(callback);
  }
};
client.listDataSources({parent: formattedParent}, options).then(callback);

client.listDataSourcesStream({parent: formattedParent}).on('data', element => {
  console.log(element);
});

PHP

Before trying this sample, follow the PHP setup instructions in the BigQuery Data Transfer Quickstart Using Client Libraries . For more information, see the BigQuery Data Transfer PHP API reference documentation .

View on GitHub Feedback

# Includes the autoloader for libraries installed with composer
require __DIR__ . '/vendor/autoload.php';

# Imports the Google Cloud client library
use Google\Cloud\BigQuery\DataTransfer\V1\DataTransferServiceClient;

# Instantiates a client
$bqdtsClient = new DataTransferServiceClient();

# Your Google Cloud Platform project ID
$projectId = 'YOUR_PROJECT_ID';
$parent = sprintf('projects/%s/locations/us', $projectId);

try {
    echo 'Supported Data Sources:', PHP_EOL;
    $pagedResponse = $bqdtsClient->listDataSources($parent);
    foreach ($pagedResponse->iterateAllElements() as $dataSource) {
        echo 'Data source: ', $dataSource->getDisplayName(), PHP_EOL;
        echo 'ID: ', $dataSource->getDataSourceId(), PHP_EOL;
        echo 'Full path: ', $dataSource->getName(), PHP_EOL;
        echo 'Description: ', $dataSource->getDescription(), PHP_EOL;
    }
} finally {
    $bqdtsClient->close();
}

Python

Before trying this sample, follow the Python setup instructions in the BigQuery Data Transfer Quickstart Using Client Libraries . For more information, see the BigQuery Data Transfer Python API reference documentation .

View on GitHub Feedback

from google.cloud import bigquery_datatransfer

client = bigquery_datatransfer.DataTransferServiceClient()

project = 'my-project'  # TODO: Update to your project ID.

# Get the full path to your project.
parent = client.project_path(project)

print('Supported Data Sources:')

# Iterate over all possible data sources.
for data_source in client.list_data_sources(parent):
    print('{}:'.format(data_source.display_name))
    print('\tID: {}'.format(data_source.data_source_id))
    print('\tFull path: {}'.format(data_source.name))
    print('\tDescription: {}'.format(data_source.description))

Ruby

Before trying this sample, follow the Ruby setup instructions in the BigQuery Data Transfer Quickstart Using Client Libraries . For more information, see the BigQuery Data Transfer Ruby API reference documentation .

View on GitHub Feedback

# Imports the Google Cloud client library
require "google/cloud/bigquery/data_transfer"

# Your Google Cloud Platform project ID
project_id = "YOUR_PROJECT_ID"

# Instantiate a client
BigQueryDataTransfer = Google::Cloud::Bigquery::DataTransfer # Alias the module
data_transfer = BigQueryDataTransfer.new

# Get the full path to your project.
project_path = BigQueryDataTransfer::V1::DataTransferServiceClient.project_path(
  project_id
)

puts "Supported Data Sources:"

# Iterate over all possible data sources.
data_transfer.list_data_sources(project_path).each do |data_source|
  puts "Data source: #{data_source.display_name}"
  puts "ID: #{data_source.data_source_id}"
  puts "Full path: #{data_source.name}"
  puts "Description: #{data_source.description}"
end

Additional Resources

C#

• API Reference Documentation
• Source Code
• GitHub Issue Tracker
• Stack Overflow

Go

• API Reference Documentation
• Source Code
• GitHub Issue Tracker
• Stack Overflow

Java

• API Reference Documentation
• Source Code
• GitHub Issue Tracker
• Stack Overflow

Node.js

• API Reference Documentation
• Source Code
• GitHub Issue Tracker
• Stack Overflow

PHP

• API Reference Documentation
• Source Code
• GitHub Issue Tracker
• Stack Overflow

Python

• API Reference Documentation
• Source Code
• GitHub Issue Tracker
• Stack Overflow

Ruby

• API Reference Documentation
• Source Code
• GitHub Issue Tracker
• Stack Overflow