Creating a Storage Transfer Service Client for a Google API library

Before you can make requests to Storage Transfer Service, you must make sure the Storage Transfer Service API is enabled for your project, and that your application is set up for authorization, using the OAuth 2.0 protocol. We recommend that you also implement Truncated exponential backoff to handle retry attempts in the event of request failures.

Enabling the service

Accessing Storage Transfer Service through a Google APIs Library, or directly using the Storage Transfer Service API, requires that you enable the Google Storage Transfer API.

Enable the Google Storage Transfer API.

Enable the API

Authentication flow

Because all Storage Transfer Service operations occur without the need for human intervention or user consent, the best authentication flow for a Storage Transfer Service application is server-to-server, using a service account. App Engine and Compute Engine have built-in service accounts that you can use, or you can create a service credential in the Google Cloud Console.

When you use a service account, the Google APIs client libraries handle the authentication and authorization process for you in the following scenarios:

App Engine application

If your application runs on App Engine, it authenticates with the default service account of your App Engine application.

The code also works with your App Engine local emulator as long as you're using Cloud SDK (mvn gcloud:run), by using a service account and setting the GOOGLE_APPLICATION_CREDENTIALS environment variable.

For more information, see Local or third-party host section.

Compute Engine application

If your application runs on a Compute Engine instance, it authenticates using the default Compute Engine service account associated with the project.

Configure your instance to use service accounts and add the scope listed in the Authorization scope section.

For more information, see Setting the scope of service account access for instances in the Compute Engine documentation.

Local or third-party host

If you're running a local client or if you're running your program in a non–Google Cloud environment, create a new service account, download its JSON key file, and then set the environment variable GOOGLE_APPLICATION_CREDENTIALS to point to the JSON key file.

For more information, see Authenticating as a service account.

Authorization scope

Applications that access Storage Transfer Service using OAuth 2.0 must specify the cloud-platform authorization scope.

Scope Meaning
https://www.googleapis.com/auth/cloud-platform Full access.

Create a client

You can use the sample code below to create a Storage Transfer Service client in any of the environments discussed in Authentication Flow. Because the code uses Google Application Default Credentials, it selects the appropriate service account to use.

The service account must be assigned one of the following roles:

You can check by going to the permissions page for your project.

Java

This sample uses the Google API Client Library for Java. The custom RetryHttpInitializerWrapper class is described in the Retry Handling section.

storage/storage-transfer/src/main/java/com/google/cloud/storage/storagetransfer/samples/CreateTransferClient.java 
import com.google.api.client.googleapis.util.Utils;
import com.google.api.client.http.HttpRequestInitializer;
import com.google.api.client.http.HttpTransport;
import com.google.api.client.json.JsonFactory;
import com.google.api.services.storagetransfer.v1.Storagetransfer;
import com.google.api.services.storagetransfer.v1.StoragetransferScopes;
import com.google.auth.http.HttpCredentialsAdapter;
import com.google.auth.oauth2.GoogleCredentials;
import com.google.common.base.Preconditions;
import java.io.IOException;

/** Create a client to make calls to Storage Transfer API. */
public final class CreateTransferClient {

  /**
   * Create a Storage Transfer client using application default credentials and other default
   * settings.
   *
   * @return a Storage Transfer client
   * @throws IOException there was an error obtaining application default credentials
   */
  public static Storagetransfer createStorageTransferClient() throws IOException {
    HttpTransport httpTransport = Utils.getDefaultTransport();
    JsonFactory jsonFactory = Utils.getDefaultJsonFactory();
    GoogleCredentials credential = GoogleCredentials.getApplicationDefault();
    return createStorageTransferClient(httpTransport, jsonFactory, credential);
  }

  /**
   * Create a Storage Transfer client using user-supplied credentials and other settings.
   *
   * @param httpTransport a user-supplied HttpTransport
   * @param jsonFactory a user-supplied JsonFactory
   * @param credential a user-supplied Google credential
   * @return a Storage Transfer client
   */
  public static Storagetransfer createStorageTransferClient(
      HttpTransport httpTransport, JsonFactory jsonFactory, GoogleCredentials credential) {
    Preconditions.checkNotNull(httpTransport);
    Preconditions.checkNotNull(jsonFactory);
    Preconditions.checkNotNull(credential);

    // In some cases, you need to add the scope explicitly.
    if (credential.createScopedRequired()) {
      credential = credential.createScoped(StoragetransferScopes.all());
    }
    // Please use custom HttpRequestInitializer for automatic
    // retry upon failures. We provide a simple reference
    // implementation in the "Retry Handling" section.
    HttpRequestInitializer initializer = new HttpCredentialsAdapter(credential);
    return new Storagetransfer.Builder(httpTransport, jsonFactory, initializer)
        .setApplicationName("storagetransfer-sample")
        .build();
  }
}

Python

This sample uses the Google API Client Library for Python.

storage/transfer_service/create_client.py 
import googleapiclient.discovery


def create_transfer_client():
    return googleapiclient.discovery.build('storagetransfer', 'v1')

Retry handling

In this case of RPC failures, you should implement code to handle retry attempts with a Truncated exponential backoff strategy.

Java

This sample uses the Google API Client Library for Java.The class RetryHttpInitializerWrapper handles retry attempts for you.

storage/storage-transfer/src/main/java/com/google/cloud/storage/storagetransfer/samples/RetryHttpInitializerWrapper.java 
import com.google.api.client.auth.oauth2.Credential;
import com.google.api.client.http.HttpBackOffIOExceptionHandler;
import com.google.api.client.http.HttpBackOffUnsuccessfulResponseHandler;
import com.google.api.client.http.HttpRequest;
import com.google.api.client.http.HttpRequestInitializer;
import com.google.api.client.http.HttpResponse;
import com.google.api.client.http.HttpUnsuccessfulResponseHandler;
import com.google.api.client.util.ExponentialBackOff;
import com.google.api.client.util.Sleeper;
import com.google.common.base.Preconditions;
import java.io.IOException;
import java.util.logging.Logger;

/**
 * RetryHttpInitializerWrapper will automatically retry upon RPC failures, preserving the
 * auto-refresh behavior of the Google Credentials.
 */
public class RetryHttpInitializerWrapper implements HttpRequestInitializer {

  private static final Logger LOG = Logger.getLogger(RetryHttpInitializerWrapper.class.getName());
  private final Credential wrappedCredential;
  private final Sleeper sleeper;
  private static final int MILLIS_PER_MINUTE = 60 * 1000;

  /**
   * A constructor using the default Sleeper.
   *
   * @param wrappedCredential the credential used to authenticate with a Google Cloud Platform
   *     project
   */
  public RetryHttpInitializerWrapper(Credential wrappedCredential) {
    this(wrappedCredential, Sleeper.DEFAULT);
  }

  /**
   * A constructor used only for testing.
   *
   * @param wrappedCredential the credential used to authenticate with a Google Cloud Platform
   *     project
   * @param sleeper a user-supplied Sleeper
   */
  RetryHttpInitializerWrapper(Credential wrappedCredential, Sleeper sleeper) {
    this.wrappedCredential = Preconditions.checkNotNull(wrappedCredential);
    this.sleeper = sleeper;
  }

  /**
   * Initialize an HttpRequest.
   *
   * @param request an HttpRequest that should be initialized
   */
  public void initialize(HttpRequest request) {
    request.setReadTimeout(2 * MILLIS_PER_MINUTE); // 2 minutes read timeout
    final HttpUnsuccessfulResponseHandler backoffHandler =
        new HttpBackOffUnsuccessfulResponseHandler(new ExponentialBackOff()).setSleeper(sleeper);
    request.setInterceptor(wrappedCredential);
    request.setUnsuccessfulResponseHandler(
        new HttpUnsuccessfulResponseHandler() {
          public boolean handleResponse(
              final HttpRequest request, final HttpResponse response, final boolean supportsRetry)
              throws IOException {
            if (wrappedCredential.handleResponse(request, response, supportsRetry)) {
              // If credential decides it can handle it, the return code or message indicated
              // something specific to authentication, and no backoff is desired.
              return true;
            } else if (backoffHandler.handleResponse(request, response, supportsRetry)) {
              // Otherwise, we defer to the judgement of our internal backoff handler.
              LOG.info("Retrying " + request.getUrl().toString());
              return true;
            } else {
              return false;
            }
          }
        });
    request.setIOExceptionHandler(
        new HttpBackOffIOExceptionHandler(new ExponentialBackOff()).setSleeper(sleeper));
  }
}

Python

storage/transfer_service/get_transfer_job_with_retries.py 

"""Command-line sample that gets a transfer job using retries
"""

import googleapiclient.discovery


def get_transfer_job_with_retries(project_id, job_name, retries):
    """Check the latest transfer operation associated with a transfer job."""
    storagetransfer = googleapiclient.discovery.build("storagetransfer", "v1")

    transferJob = (
        storagetransfer.transferJobs()
        .get(projectId=project_id, jobName=job_name)
        .execute(num_retries=retries)
    )
    print(
        "Fetched transfer job: "
        + transferJob.get("name")
        + " using {} retries".format(retries)
    )