Storage Transfer Service client libraries

This page shows how to get started with the Cloud Client Libraries for the Google Storage Transfer API. Read more about the client libraries for Cloud APIs, including the older Google API Client Libraries, in Client Libraries Explained.

For information about updating from the Google API Client Library to the Cloud Client Library that is described on this page, see the Storage Transfer Service Migration Guide.

Installing the client library

C#

For more information, see Setting Up a C# Development Environment.

If you are using Visual Studio 2017 or higher, open a nuget package manager window and type the following:

Install-Package Google.Cloud.StorageTransfer.V1

If you are using .NET Core command-line interface tools to install your dependencies, run the following command:

dotnet add package Google.Cloud.StorageTransfer.V1

Go

For more information, see Setting Up a Go Development Environment.

go get cloud.google.com/go/storagetransfer

Java

For more information, see Setting Up a Java Development Environment.

If you are using Maven, add this to your pom.xml file:

<dependency>
    <groupId>com.google.cloud</groupId>
    <artifactId>google-cloud-storage-transfer</artifactId>
    <version>0.2.3</version>
</dependency>

If you are using Gradle without BOM, add this to your dependencies:

implementation 'com.google.cloud:google-cloud-storage-transfer:0.2.3'

Node.js

For more information, see Setting Up a Node.js Development Environment.

npm install --save @google-cloud/storage-transfer

PHP

For more information, see Using PHP on Google Cloud.

composer require google/cloud

Python

For more information, see Setting Up a Python Development Environment.

pip install --upgrade google-cloud-storage-transfer

Ruby

For more information, see Setting Up a Ruby Development Environment.

gem install google-cloud-storage_transfer

Setting up authentication

To run the client library, you must first set up authentication. One way to do that is to create a service account and set an environment variable, as shown in the following steps. For other ways to authenticate, see Authenticating as a service account.

Provide authentication credentials to your application code by setting the environment variable GOOGLE_APPLICATION_CREDENTIALS. This variable applies only to your current shell session. If you want the variable to apply to future shell sessions, set the variable in your shell startup file, for example in the ~/.bashrc or ~/.profile file.

Linux or macOS

export GOOGLE_APPLICATION_CREDENTIALS="KEY_PATH"

Replace KEY_PATH with the path of the JSON file that contains your service account key.

For example:

export GOOGLE_APPLICATION_CREDENTIALS="/home/user/Downloads/service-account-file.json"

Windows

For PowerShell:

$env:GOOGLE_APPLICATION_CREDENTIALS="KEY_PATH"

Replace KEY_PATH with the path of the JSON file that contains your service account key.

For example:

$env:GOOGLE_APPLICATION_CREDENTIALS="C:\Users\username\Downloads\service-account-file.json"

For command prompt:

set GOOGLE_APPLICATION_CREDENTIALS=KEY_PATH

Replace KEY_PATH with the path of the JSON file that contains your service account key.

Using the client library

The following examples show how to use the client library.

Before using these samples, follow the instructions in Configuring access to set up the required permissions.

C#

To learn how to install and use the client library for Storage Transfer Service, see Storage Transfer Service client libraries. For more information, see the Storage Transfer Service C# API reference documentation.

View on GitHub

using System;
using Google.Cloud.StorageTransfer.V1;

namespace StorageTransfer.Samples
{
    public class QuickstartSample
    {
        public TransferJob Quickstart(
            // Your Google Cloud Project ID
            string projectId = "my-project-id",
            // The GCS bucket to transfer data from
            string sourceBucket = "my-source-bucket",
            // The GCS bucket to transfer data to
            string sinkBucket = "my-sink-bucket")
        {
            TransferJob transferJob = new TransferJob
            {
                ProjectId = projectId,
                TransferSpec = new TransferSpec
                {
                    GcsDataSink = new GcsData { BucketName = sourceBucket },
                    GcsDataSource = new GcsData { BucketName = sinkBucket }
                },
                Status = TransferJob.Types.Status.Enabled
            };

            StorageTransferServiceClient client = StorageTransferServiceClient.Create();
            TransferJob response = client.CreateTransferJob(new CreateTransferJobRequest { TransferJob = transferJob });
            client.RunTransferJob(new RunTransferJobRequest
            {
                JobName = response.Name,
                ProjectId = projectId
            });

            Console.WriteLine($"Created and ran transfer job from {sourceBucket} to {sinkBucket} with name {response.Name}");

            return response;
        }
    }
}

Go

View on GitHub

import (
	"context"
	"fmt"
	"io"

	storagetransfer "cloud.google.com/go/storagetransfer/apiv1"
	storagetransferpb "google.golang.org/genproto/googleapis/storagetransfer/v1"
)

// quickstart creates and runs a transfer job between two GCS buckets.
func quickstart(w io.Writer, projectID string, sourceGCSBucket string, sinkGCSBucket string) (*storagetransferpb.TransferJob, error) {
	// Your Google Cloud Project ID
	// projectID := "my-project-id"

	// The name of the GCS bucket to transfer data from
	// sourceGCSBucket := "my-source-bucket"

	// The name of the GCS bucket to transfer data to
	// sinkGCSBucket := "my-sink-bucket"
	ctx := context.Background()
	client, err := storagetransfer.NewClient(ctx)
	if err != nil {
		return nil, fmt.Errorf("storagetransfer.NewClient: %v", err)
	}
	defer client.Close()

	req := &storagetransferpb.CreateTransferJobRequest{
		TransferJob: &storagetransferpb.TransferJob{
			ProjectId: projectID,
			TransferSpec: &storagetransferpb.TransferSpec{
				DataSource: &storagetransferpb.TransferSpec_GcsDataSource{
					GcsDataSource: &storagetransferpb.GcsData{BucketName: sourceGCSBucket}},
				DataSink: &storagetransferpb.TransferSpec_GcsDataSink{
					GcsDataSink: &storagetransferpb.GcsData{BucketName: sinkGCSBucket}},
			},
			Status: storagetransferpb.TransferJob_ENABLED,
		},
	}
	resp, err := client.CreateTransferJob(ctx, req)
	if err != nil {
		return nil, fmt.Errorf("failed to create transfer job: %v", err)
	}
	if _, err = client.RunTransferJob(ctx, &storagetransferpb.RunTransferJobRequest{
		ProjectId: projectID,
		JobName:   resp.Name,
	}); err != nil {
		return nil, fmt.Errorf("failed to run transfer job: %v", err)
	}

	fmt.Fprintf(w, "Created and ran transfer job from %v to %v with name %v", sourceGCSBucket, sinkGCSBucket, resp.Name)
	return resp, nil
}

Java

View on GitHub

import com.google.storagetransfer.v1.proto.StorageTransferServiceClient;
import com.google.storagetransfer.v1.proto.TransferProto.CreateTransferJobRequest;
import com.google.storagetransfer.v1.proto.TransferProto.RunTransferJobRequest;
import com.google.storagetransfer.v1.proto.TransferTypes.GcsData;
import com.google.storagetransfer.v1.proto.TransferTypes.TransferJob;
import com.google.storagetransfer.v1.proto.TransferTypes.TransferSpec;

public class QuickstartSample {
  /** Quickstart sample using transfer service to transfer from one GCS bucket to another. */
  public static void quickStartSample(
      String projectId, String gcsSourceBucket, String gcsSinkBucket) throws Exception {
    // Your Google Cloud Project ID
    // String projectId = "your-project-id";

    // The name of the source GCS bucket to transfer objects from
    // String gcsSourceBucket = "your-source-gcs-source-bucket";

    // The name of the  GCS bucket to transfer  objects to
    // String gcsSinkBucket = "your-sink-gcs-bucket";

    StorageTransferServiceClient storageTransfer = StorageTransferServiceClient.create();

    TransferJob transferJob =
        TransferJob.newBuilder()
            .setProjectId(projectId)
            .setTransferSpec(
                TransferSpec.newBuilder()
                    .setGcsDataSource(GcsData.newBuilder().setBucketName(gcsSourceBucket))
                    .setGcsDataSink(GcsData.newBuilder().setBucketName(gcsSinkBucket)))
            .setStatus(TransferJob.Status.ENABLED)
            .build();

    TransferJob response =
        storageTransfer.createTransferJob(
            CreateTransferJobRequest.newBuilder().setTransferJob(transferJob).build());

    storageTransfer
        .runTransferJobAsync(
            RunTransferJobRequest.newBuilder()
                .setProjectId(projectId)
                .setJobName(response.getName())
                .build())
        .get();
    System.out.println(
        "Created and ran transfer job between two GCS buckets with name " + response.getName());
  }
}

Node.js

View on GitHub


// Imports the Google Cloud client library
const {
  StorageTransferServiceClient,
} = require('@google-cloud/storage-transfer');

/**
 * TODO(developer): Uncomment the following lines before running the sample.
 */
// Your project id
// const projectId = 'my-project'

// The ID of the GCS bucket to transfer data from
// const gcsSourceBucket = 'my-source-bucket'

// The ID of the GCS bucket to transfer data to
// const gcsSinkBucket = 'my-sink-bucket'

// Creates a client
const client = new StorageTransferServiceClient();

/**
 * Creates a one-time transfer job.
 */
async function quickstart() {
  // Creates a request to transfer from the source bucket to
  // the sink bucket
  const createRequest = {
    transferJob: {
      projectId: projectId,
      transferSpec: {
        gcsDataSource: {bucketName: gcsSourceBucket},
        gcsDataSink: {bucketName: gcsSinkBucket},
      },
      status: 'ENABLED',
    },
  };

  // Runs the request and creates the job
  const [transferJob] = await client.createTransferJob(createRequest);

  const runRequest = {
    jobName: transferJob.name,
    projectId: projectId,
  };
  await client.runTransferJob(runRequest);

  console.log(
    `Created and ran a transfer job from ${gcsSourceBucket} to ${gcsSinkBucket} with name ${transferJob.name}`
  );
}

quickstart();

PHP

View on GitHub

use Google\Cloud\StorageTransfer\V1\StorageTransferServiceClient;
use Google\Cloud\StorageTransfer\V1\TransferJob;
use Google\Cloud\StorageTransfer\V1\TransferJob\Status;
use Google\Cloud\StorageTransfer\V1\TransferSpec;
use Google\Cloud\StorageTransfer\V1\GcsData;

/**
 * Creates and runs a transfer job between two GCS buckets
 *
 * @param string $projectId Your Google Cloud project ID.
 * @param string $sourceGcsBucketName The name of the GCS bucket to transfer objects from.
 * @param string $sinkGcsBucketName The name of the GCS bucket to transfer objects to.
 */
function quickstart($projectId, $sourceGcsBucketName, $sinkGcsBucketName)
{
    // $project = 'my-project-id';
    // $sourceGcsBucketName = 'my-source-bucket';
    // $sinkGcsBucketName = 'my-sink-bucket';
    $transferJob = new TransferJob([
        'project_id' => $projectId,
        'transfer_spec' => new TransferSpec([
            'gcs_data_sink' => new GcsData(['bucket_name' => $sourceGcsBucketName]),
            'gcs_data_source' => new GcsData(['bucket_name' => $sourceGcsBucketName])
        ]),
        'status' => Status::ENABLED
    ]);

    $client = new StorageTransferServiceClient();
    $response = $client->createTransferJob($transferJob);
    $client->runTransferJob($response->getName(), $projectId);

    printf('Created and ran transfer job from %s to %s with name %s ' . PHP_EOL, $sourceGcsBucketName, $sinkGcsBucketName, $response->getName());
}

Python

View on GitHub

from google.cloud import storage_transfer


def create_one_time_transfer(project_id: str, source_bucket: str,
                             sink_bucket: str):
    """Creates a one-time transfer job."""

    client = storage_transfer.StorageTransferServiceClient()

    # The ID of the Google Cloud Platform Project that owns the job
    # project_id = 'my-project-id'

    # Google Cloud Storage source bucket name
    # source_bucket = 'my-gcs-source-bucket'

    # Google Cloud Storage destination bucket name
    # sink_bucket = 'my-gcs-destination-bucket'

    transfer_job_request = storage_transfer.CreateTransferJobRequest({
        'transfer_job': {
            'project_id': project_id,
            'status': storage_transfer.TransferJob.Status.ENABLED,
            'transfer_spec': {
                'gcs_data_source': {
                    'bucket_name': source_bucket,
                },
                'gcs_data_sink': {
                    'bucket_name': sink_bucket,
                }
            }
        }
    })

    transfer_job = client.create_transfer_job(transfer_job_request)
    client.run_transfer_job({
        'job_name': transfer_job.name,
        'project_id': project_id
    })

    print(f'Created and ran transfer job: {transfer_job.name}')

Ruby

View on GitHub

def quickstart project_id:, gcs_source_bucket:, gcs_sink_bucket:
  # Your Google Cloud Project ID
  # project_id = "your-project_id"

  # The name of the source GCS bucket to transfer objects from
  # gcs_source_bucket = "your-source-gcs-source-bucket"

  # The name of the  GCS bucket to transfer objects to
  # gcs_sink_bucket = "your-sink-gcs-bucket"

  require "google/cloud/storage_transfer"

  transfer_job = {
    project_id: project_id,
    transfer_spec: {
      gcs_data_source: {
        bucket_name: gcs_source_bucket
      },
      gcs_data_sink: {
        bucket_name: gcs_sink_bucket
      }
    },
    status: :ENABLED
  }

  client = Google::Cloud::StorageTransfer.storage_transfer_service

  transfer_job_response = client.create_transfer_job transfer_job: transfer_job

  run_request = {
    project_id: project_id,
    job_name: transfer_job_response.name
  }
  client.run_transfer_job run_request

  puts "Created and ran transfer job between #{gcs_source_bucket} and #{gcs_sink_bucket} with name #{transfer_job_response.name}"
end

Storage Transfer Service client libraries

Installing the client library

C#

Go

Java

Node.js

PHP

Python

Ruby

Setting up authentication

Linux or macOS

Windows

Using the client library

C#

Go

Java

Node.js

PHP

Python

Ruby

Additional resources

C#

Go

Java

Node.js

PHP

Python

Ruby