Creating a Storage Transfer Service Client for a Google API library

Before you can make requests to Storage Transfer Service, you must make sure the Storage Transfer Service API is enabled for your project, and that your application is set up for authorization, using the OAuth 2.0 protocol. We recommend that you also implement exponential backoff to handle retry attempts in the event of request failures.

Enabling the service

Accessing Storage Transfer Service through a Google APIs Library, or directly using the Storage Transfer Service API, requires that you enable the Google Storage Transfer API. To do so: Enable the Google Storage Transfer API API.

Enable the API

Authentication flow

Because all Storage Transfer Service operations occur without the need for human intervention or user consent, the best authentication flow for a Storage Transfer Service application is server-to-server, using a service account. Google App Engine and Google Compute Engine have built-in service accounts that you can leverage, or you can create a service credential in the Google Cloud Platform Console.

When you use a service account, the Google APIs client libraries handle the authentication and authorization process for you in the following scenarios:

Google App Engine application

If your application runs on App Engine, it authenticates with the default service account of your App Engine application.

Note: The code also works with your App Engine local emulator as long as you're using Google Cloud SDK (mvn gcloud:run), by using a service account and setting the GOOGLE_APPLICATION_CREDENTIALS environment variable. See the Local or third-party host for more details.

Google Compute Engine application

If your application runs on a Google Compute Engine instance, it authenticates using the default Compute Engine service account associated with the project.

Note: Configure your instance to use service accounts and add the scope listed in the Authorization Scope section. For more information, see Setting the scope of service account access for instances in the Google Compute Engine documentation.

Local or third-party host

If you're running a local client or if you're running your program in a non–Google Cloud environment, create a new service account, download its JSON key file, and then set the environment variable GOOGLE_APPLICATION_CREDENTIALS to point to the JSON key file. For more information, see How the Application Default Credentials Work.

Authorization scope

Applications that access Storage Transfer Service using OAuth 2.0 must specify the cloud-platform authorization scope.

Scope	Meaning
https://www.googleapis.com/auth/cloud-platform	Full access.

Create a client

You can use the sample code below to create a Storage Transfer Service client in any of the environments discussed in Authentication Flow. Because the code uses Google Application Default Credentials, it selects the appropriate service account to use.

The service account must have be assigned one of the following roles:

• roles/owner
• roles/editor
• roles/storagetransfer.admin
• roles/storagetransfer.user

• A custom role that includes, at bare minimum, roles/storagetransfer.user permissions.

  For more information about adding and viewing project-level permissions, see Using IAM permissions with projects.

You can check by going to the permissions page for your project.

Java

This sample uses the Google APIs Client Library for Java. The custom RetryHttpInitializerWrapper class is described in the Retry Handling section.

storage/storage-transfer/src/main/java/com/google/cloud/storage/storagetransfer/samples/TransferClientCreator.java

View on GitHub

package com.google.cloud.storage.storagetransfer.samples;

import com.google.api.client.googleapis.auth.oauth2.GoogleCredential;
import com.google.api.client.googleapis.util.Utils;
import com.google.api.client.http.HttpRequestInitializer;
import com.google.api.client.http.HttpTransport;
import com.google.api.client.json.JsonFactory;
import com.google.api.services.storagetransfer.v1.Storagetransfer;
import com.google.api.services.storagetransfer.v1.StoragetransferScopes;
import com.google.common.base.Preconditions;
import java.io.IOException;

/**
 * Create a client to make calls to Storage Transfer API.
 */
public final class TransferClientCreator {

  /**
   * Create a Storage Transfer client using application default credentials and other default
   * settings.
   *
   * @return a Storage Transfer client
   * @throws IOException
   *           there was an error obtaining application default credentials
   */
  public static Storagetransfer createStorageTransferClient() throws IOException {
    HttpTransport httpTransport = Utils.getDefaultTransport();
    JsonFactory jsonFactory = Utils.getDefaultJsonFactory();
    GoogleCredential credential =
        GoogleCredential.getApplicationDefault(httpTransport, jsonFactory);
    return createStorageTransferClient(httpTransport, jsonFactory, credential);
  }

  /**
   * Create a Storage Transfer client using user-supplied credentials and other settings.
   *
   * @param httpTransport
   *          a user-supplied HttpTransport
   * @param jsonFactory
   *          a user-supplied JsonFactory
   * @param credential
   *          a user-supplied Google credential
   * @return a Storage Transfer client
   */
  public static Storagetransfer createStorageTransferClient(
      HttpTransport httpTransport, JsonFactory jsonFactory, GoogleCredential credential) {
    Preconditions.checkNotNull(httpTransport);
    Preconditions.checkNotNull(jsonFactory);
    Preconditions.checkNotNull(credential);

    // In some cases, you need to add the scope explicitly.
    if (credential.createScopedRequired()) {
      credential = credential.createScoped(StoragetransferScopes.all());
    }
    // Please use custom HttpRequestInitializer for automatic
    // retry upon failures. We provide a simple reference
    // implementation in the "Retry Handling" section.
    HttpRequestInitializer initializer = new RetryHttpInitializerWrapper(credential);
    return new Storagetransfer.Builder(httpTransport, jsonFactory, initializer)
        .setApplicationName("storagetransfer-sample")
        .build();
  }
}

Python

This sample uses the Google APIs Client Library for Python.

storage/transfer_service/create_client.py

View on GitHub

import googleapiclient.discovery


def create_transfer_client():
    return googleapiclient.discovery.build('storagetransfer', 'v1')

Retry handling

In this case of RPC failures, you should implement code to handle retry attempts with an exponential backoff strategy.

Java

This sample uses the Google APIs Client Library for Java. The class RetryHttpInitializerWrapper handles retry attempts for you.

storage/storage-transfer/src/main/java/com/google/cloud/storage/storagetransfer/samples/RetryHttpInitializerWrapper.java

View on GitHub

package com.google.cloud.storage.storagetransfer.samples;

import com.google.api.client.auth.oauth2.Credential;
import com.google.api.client.http.HttpBackOffIOExceptionHandler;
import com.google.api.client.http.HttpBackOffUnsuccessfulResponseHandler;
import com.google.api.client.http.HttpRequest;
import com.google.api.client.http.HttpRequestInitializer;
import com.google.api.client.http.HttpResponse;
import com.google.api.client.http.HttpUnsuccessfulResponseHandler;
import com.google.api.client.util.ExponentialBackOff;
import com.google.api.client.util.Sleeper;
import com.google.common.base.Preconditions;
import java.io.IOException;
import java.util.logging.Logger;

/**
 * RetryHttpInitializerWrapper will automatically retry upon RPC failures, preserving the
 * auto-refresh behavior of the Google Credentials.
 */
public class RetryHttpInitializerWrapper implements HttpRequestInitializer {

  private static final Logger LOG = Logger.getLogger(RetryHttpInitializerWrapper.class.getName());
  private final Credential wrappedCredential;
  private final Sleeper sleeper;
  private static final int MILLIS_PER_MINUTE = 60 * 1000;

  /**
   * A constructor using the default Sleeper.
   *
   * @param wrappedCredential
   *          the credential used to authenticate with a Google Cloud Platform project
   */
  public RetryHttpInitializerWrapper(Credential wrappedCredential) {
    this(wrappedCredential, Sleeper.DEFAULT);
  }

  /**
   * A constructor used only for testing.
   *
   * @param wrappedCredential
   *          the credential used to authenticate with a Google Cloud Platform project
   * @param sleeper
   *          a user-supplied Sleeper
   */
  RetryHttpInitializerWrapper(Credential wrappedCredential, Sleeper sleeper) {
    this.wrappedCredential = Preconditions.checkNotNull(wrappedCredential);
    this.sleeper = sleeper;
  }

  /**
   * Initialize an HttpRequest.
   *
   * @param request
   *          an HttpRequest that should be initialized
   */
  public void initialize(HttpRequest request) {
    request.setReadTimeout(2 * MILLIS_PER_MINUTE); // 2 minutes read timeout
    final HttpUnsuccessfulResponseHandler backoffHandler =
        new HttpBackOffUnsuccessfulResponseHandler(new ExponentialBackOff()).setSleeper(sleeper);
    request.setInterceptor(wrappedCredential);
    request.setUnsuccessfulResponseHandler(
        new HttpUnsuccessfulResponseHandler() {
          public boolean handleResponse(
              final HttpRequest request, final HttpResponse response, final boolean supportsRetry)
              throws IOException {
            if (wrappedCredential.handleResponse(request, response, supportsRetry)) {
              // If credential decides it can handle it, the return code or message indicated
              // something specific to authentication, and no backoff is desired.
              return true;
            } else if (backoffHandler.handleResponse(request, response, supportsRetry)) {
              // Otherwise, we defer to the judgement of our internal backoff handler.
              LOG.info("Retrying " + request.getUrl().toString());
              return true;
            } else {
              return false;
            }
          }
        });
    request.setIOExceptionHandler(
        new HttpBackOffIOExceptionHandler(new ExponentialBackOff()).setSleeper(sleeper));
  }
}

Python

You can pass the num_retries=n argument to the library execute method to retry with exponential backoff upon retryable failures.

resp = client.projects().subscriptions().pull(
subscription=subscription, body=body).execute(num_retries=3)