BigQuery Data Transfer Client Libraries

This page shows how to get started with the Cloud Client Libraries for the BigQuery Data Transfer API. Read more about the client libraries for Cloud APIs, including the older Google APIs Client Libraries, in Client Libraries Explained.

Alpha 版

此产品或功能目前处于预发布状态，可能会更改或仅提供有限支持。如需了解详情，请参阅产品发布阶段。

Installing the client library

C#

For more information, see Setting Up a C# Development Environment.

Install-Package Google.Cloud.BigQuery.DataTransfer.V1 -Pre

Go

go get -u cloud.google.com/go/bigquery/datatransfer/apiv1

Java

For more information, see Setting Up a Java Development Environment. 如果您使用的是 Maven，请将以下代码添加到您的 pom.xml 文件中：

 <dependencyManagement>
  <dependencies>
    <dependency>
      <groupId>com.google.cloud</groupId>
      <artifactId>libraries-bom</artifactId>
      <version>0.118.0-beta</version>
      <type>pom</type>
      <scope>import</scope>
     </dependency>
   </dependencies>
</dependencyManagement>

<dependency>
  <groupId>com.google.cloud</groupId>
  <artifactId>google-cloud-bigquerydatatransfer</artifactId>
</dependency>

如果您使用的是 Gradle，请将以下代码添加到您的依赖项中：

compile 'com.google.cloud:google-cloud-bigquerydatatransfer:0.118.0-beta'

如果您使用的是 SBT，请将以下代码添加到您的依赖项中：

libraryDependencies += "com.google.cloud" % "google-cloud-bigquerydatatransfer" % "0.118.0-beta"

如果您使用的是 IntelliJ 或 Eclipse，请通过以下 IDE 插件将客户端库添加到您的项目中：

• Cloud Code for IntelliJ
• Cloud Tools for Eclipse

上述插件还提供其他功能，例如服务帐号密钥管理。如需了解详情，请参阅各个插件相应的文档。

注意：Cloud Java 客户端库目前不支持 Android。

Node.js

For more information, see Setting Up a Node.js Development Environment.

npm install --save @google-cloud/bigquery-data-transfer

PHP

composer require google/cloud-bigquerydatatransfer

Python

For more information, see Setting Up a Python Development Environment.

pip install --upgrade google-cloud-bigquery-datatransfer

Ruby

For more information, see Setting Up a Ruby Development Environment.

gem install google-cloud-bigquery-data_transfer

Setting up authentication

To run the client library, you must first set up authentication by creating a service account and setting an environment variable. Complete the following steps to set up authentication. For other ways to authenticate, see the GCP authentication documentation.

GCP Console

1. 在 GCP Console 中，转到创建服务帐号密钥页面。

   转到“创建服务帐号密钥”页面
2. 从服务帐号列表中，选择新的服务帐号。
3. 在服务帐号名称字段中，输入一个名称。

4. 从角色列表中，选择项目 > 所有者。

   注意：角色字段为您的服务帐号授予资源访问权限。稍后您可以使用 GCP Console 查看和更改此字段。如果您开发的是正式版应用，请指定比项目 > 所有者更为精细的权限。如需了解详情，请参阅为服务帐号授予角色。
5. 点击创建。包含密钥的 JSON 文件就会下载到计算机。

命令行

您可以使用本地机器上的 Cloud SDK 或在 Cloud Shell 中运行以下命令。

1. 创建服务帐号。将 [NAME] 替换为服务帐号的名称。

   gcloud iam service-accounts create [NAME]

2. 向服务帐号授予权限。将 [PROJECT_ID] 替换为您的项目 ID。

   gcloud projects add-iam-policy-binding [PROJECT_ID] --member "serviceAccount:[NAME]@[PROJECT_ID].iam.gserviceaccount.com" --role "roles/owner"

   注意：角色字段为您的服务帐号授予资源访问权限。稍后您可以使用 GCP Console 查看和更改此字段。如果您开发的是正式版应用，请指定比项目 > 所有者更为精细的权限。如需了解详情，请参阅为服务帐号授予角色。

3. 生成密钥文件。将 [FILE_NAME] 替换为密钥文件的名称。

   gcloud iam service-accounts keys create [FILE_NAME].json --iam-account [NAME]@[PROJECT_ID].iam.gserviceaccount.com

通过设置环境变量 GOOGLE_APPLICATION_CREDENTIALS，向您的应用代码提供身份验证凭据。将 [PATH] 替换为包含服务帐号密钥的 JSON 文件的文件路径，将 [FILE_NAME] 替换为文件名。此变量仅适用于当前的 shell 会话，因此，如果您打开新的会话，请重新设置该变量。

Linux 或 macOS

export GOOGLE_APPLICATION_CREDENTIALS="[PATH]"

例如：

export GOOGLE_APPLICATION_CREDENTIALS="/home/user/Downloads/[FILE_NAME].json"

Windows

使用 PowerShell：

$env:GOOGLE_APPLICATION_CREDENTIALS="[PATH]"

例如：

$env:GOOGLE_APPLICATION_CREDENTIALS="C:\Users\username\Downloads\[FILE_NAME].json"

使用命令提示符：

set GOOGLE_APPLICATION_CREDENTIALS=[PATH]

Using the client library

The following example shows how to use the client library.

C#

View on GitHub Feedback

using System;
using Google.Api.Gax;
using Google.Cloud.BigQuery.DataTransfer.V1;

namespace GoogleCloudSamples
{
    public class QuickStart
    {
        public static void Main(string[] args)
        {
            // Instantiates a client
            DataTransferServiceClient client = DataTransferServiceClient.Create();

            // Your Google Cloud Platform project ID
            string projectId = "YOUR-PROJECT-ID";

            ProjectName project = new ProjectName(projectId);
            var sources = client.ListDataSources(ParentNameOneof.From(project));
            Console.WriteLine("Supported Data Sources:");
            foreach (DataSource source in sources)
            {
                Console.WriteLine(
                    $"{source.DataSourceId}: " +
                    $"{source.DisplayName} ({source.Description})");
            }
        }
    }
}

Go

View on GitHub Feedback

// Sample bigquery-quickstart creates a Google BigQuery dataset.
package main

import (
	"fmt"
	"log"

	"golang.org/x/net/context"
	"google.golang.org/api/iterator"

	// Imports the BigQuery Data Transfer client package.
	datatransfer "cloud.google.com/go/bigquery/datatransfer/apiv1"
	datatransferpb "google.golang.org/genproto/googleapis/cloud/bigquery/datatransfer/v1"
)

func main() {
	ctx := context.Background()

	// Sets your Google Cloud Platform project ID.
	projectID := "YOUR_PROJECT_ID"

	// Creates a client.
	client, err := datatransfer.NewClient(ctx)
	if err != nil {
		log.Fatalf("Failed to create client: %v", err)
	}

	req := &datatransferpb.ListDataSourcesRequest{
		Parent: fmt.Sprintf("projects/%s", projectID),
	}
	it := client.ListDataSources(ctx, req)
	fmt.Println("Supported Data Sources:")
	for {
		ds, err := it.Next()
		if err == iterator.Done {
			break
		}
		if err != nil {
			log.Fatalf("Failed to list sources: %v", err)
		}
		fmt.Println(ds.DisplayName)
		fmt.Println("\tID: ", ds.DataSourceId)
		fmt.Println("\tFull path: ", ds.Name)
		fmt.Println("\tDescription: ", ds.Description)
	}
}

Java

View on GitHub Feedback

// Imports the Google Cloud client library

import com.google.cloud.bigquery.datatransfer.v1.DataSource;
import com.google.cloud.bigquery.datatransfer.v1.DataTransferServiceClient;
import com.google.cloud.bigquery.datatransfer.v1.DataTransferServiceClient.ListDataSourcesPagedResponse;
import com.google.cloud.bigquery.datatransfer.v1.ListDataSourcesRequest;

public class QuickstartSample {
  /**
   * List available data sources for the BigQuery Data Transfer service.
   */
  public static void main(String... args) throws Exception {
    // Sets your Google Cloud Platform project ID.
    // String projectId = "YOUR_PROJECT_ID";
    String projectId = args[0];

    // Instantiate a client. If you don't specify credentials when constructing a client, the
    // client library will look for credentials in the environment, such as the
    // GOOGLE_APPLICATION_CREDENTIALS environment variable.
    try (DataTransferServiceClient client = DataTransferServiceClient.create()) {
      // Request the list of available data sources.
      String parent = String.format("projects/%s", projectId);
      ListDataSourcesRequest request =
          ListDataSourcesRequest.newBuilder()
              .setParent(parent)
              .build();
      ListDataSourcesPagedResponse response = client.listDataSources(request);

      // Print the results.
      System.out.println("Supported Data Sources:");
      for (DataSource dataSource : response.iterateAll()) {
        System.out.println(dataSource.getDisplayName());
        System.out.printf("\tID: %s%n", dataSource.getDataSourceId());
        System.out.printf("\tFull path: %s%n", dataSource.getName());
        System.out.printf("\tDescription: %s%n", dataSource.getDescription());
      }
    }
  }
}

Node.js

View on GitHub Feedback

const bigqueryDataTransfer = require('@google-cloud/bigquery-data-transfer');
const client = new bigqueryDataTransfer.v1.DataTransferServiceClient();

async function quickstart() {
  const projectId = await client.getProjectId();

  // Iterate over all elements.
  const formattedParent = client.projectPath(projectId, 'us-central1');
  let nextRequest = {parent: formattedParent};
  const options = {autoPaginate: false};
  console.log('Data sources:');
  do {
    // Fetch the next page.
    const responses = await client.listDataSources(nextRequest, options);
    // The actual resources in a response.
    const resources = responses[0];
    // The next request if the response shows that there are more responses.
    nextRequest = responses[1];
    // The actual response object, if necessary.
    // const rawResponse = responses[2];
    resources.forEach(resource => {
      console.log(`  ${resource.name}`);
    });
  } while (nextRequest);

  console.log('\n\n');
  console.log('Sources via stream:');

  client
    .listDataSourcesStream({parent: formattedParent})
    .on('data', element => {
      console.log(`  ${element.name}`);
    });
}
quickstart();

PHP

View on GitHub Feedback

# Includes the autoloader for libraries installed with composer
require __DIR__ . '/vendor/autoload.php';

# Imports the Google Cloud client library
use Google\Cloud\BigQuery\DataTransfer\V1\DataTransferServiceClient;

# Instantiates a client
$bqdtsClient = new DataTransferServiceClient();

# Your Google Cloud Platform project ID
$projectId = 'YOUR_PROJECT_ID';
$parent = sprintf('projects/%s/locations/us', $projectId);

try {
    echo 'Supported Data Sources:', PHP_EOL;
    $pagedResponse = $bqdtsClient->listDataSources($parent);
    foreach ($pagedResponse->iterateAllElements() as $dataSource) {
        echo 'Data source: ', $dataSource->getDisplayName(), PHP_EOL;
        echo 'ID: ', $dataSource->getDataSourceId(), PHP_EOL;
        echo 'Full path: ', $dataSource->getName(), PHP_EOL;
        echo 'Description: ', $dataSource->getDescription(), PHP_EOL;
    }
} finally {
    $bqdtsClient->close();
}

Python

View on GitHub Feedback

from google.cloud import bigquery_datatransfer

client = bigquery_datatransfer.DataTransferServiceClient()

project = 'my-project'  # TODO: Update to your project ID.

# Get the full path to your project.
parent = client.project_path(project)

print('Supported Data Sources:')

# Iterate over all possible data sources.
for data_source in client.list_data_sources(parent):
    print('{}:'.format(data_source.display_name))
    print('\tID: {}'.format(data_source.data_source_id))
    print('\tFull path: {}'.format(data_source.name))
    print('\tDescription: {}'.format(data_source.description))

Ruby

View on GitHub Feedback

# Imports the Google Cloud client library
require "google/cloud/bigquery/data_transfer"

# Your Google Cloud Platform project ID
project_id = "YOUR_PROJECT_ID"

# Instantiate a client
BigQueryDataTransfer = Google::Cloud::Bigquery::DataTransfer # Alias the module
data_transfer = BigQueryDataTransfer.new

# Get the full path to your project.
project_path = BigQueryDataTransfer::V1::DataTransferServiceClient.project_path(
  project_id
)

puts "Supported Data Sources:"

# Iterate over all possible data sources.
data_transfer.list_data_sources(project_path).each do |data_source|
  puts "Data source: #{data_source.display_name}"
  puts "ID: #{data_source.data_source_id}"
  puts "Full path: #{data_source.name}"
  puts "Description: #{data_source.description}"
end

Additional resources

C#

• API Reference Documentation
• Source Code
• GitHub Issue Tracker
• Stack Overflow

Go

• API Reference Documentation
• Source Code
• GitHub Issue Tracker
• Stack Overflow

Java

• API Reference Documentation
• Source Code
• GitHub Issue Tracker
• Stack Overflow

Node.js

• API Reference Documentation
• Source Code
• GitHub Issue Tracker
• Stack Overflow

PHP

• API Reference Documentation
• Source Code
• GitHub Issue Tracker
• Stack Overflow

Python

• API Reference Documentation
• Source Code
• GitHub Issue Tracker
• Stack Overflow

Ruby

• API Reference Documentation
• Source Code
• GitHub Issue Tracker
• Stack Overflow