Boto client library plugin

This tutorial shows you how to write a simple Python program that performs basic Cloud Storage operations using the XML API. This document assumes you are familiar with Python and the Cloud Storage concepts and operations presented in the Console Quickstart.

Should you use the Cloud Storage Client Libraries instead?

Generally, you should use the Cloud Storage Client Library for Python to work with Cloud Storage buckets and objects. The Cloud Storage Client Libraries provide more flexibility, and code samples in the Cloud Storage guides use the Cloud Storage Client Libraries. Use this guide if you specifically want to interact with Cloud Storage through the XML API and the boto library.

Setting up your environment

Before starting this tutorial, you must do the following:

  1. Install a stand-alone version of gsutil on your computer.

  2. Install the boto library and gcs-oauth2-boto-plugin.

    boto is an open source Python library that is used as an interface to Cloud Storage. gcs-oauth2-boto-plugin is an authentication plugin for the boto auth plugin framework. It provides OAuth 2.0 credentials that can be used with Cloud Storage.

    Setup to use the boto library and oauth2 plugin depends on the system you are using. Use the setup examples below as guidance. These commands install pip and then use pip to install other packages. The last three commands show testing the import of the two modules to verify the installation.

    Debian and Ubuntu

    sudo python
    sudo apt-get update
    sudo apt-get upgrade
    sudo apt-get install gcc python-dev python-setuptools libffi-dev libssl-dev
    sudo pip install virtualenv
    virtualenv venv
    source ./venv/bin/activate
    (venv)pip install gcs-oauth2-boto-plugin
    >>>import boto
    >>>import gcs_oauth2_boto_plugin

    CentOS, RHEL, and Fedora

    sudo python
    sudo yum install gcc openssl-devel python-devel python-setuptools libffi-devel
    sudo pip install virtualenv
    virtualenv venv
    source ./venv/bin/activate
    (venv)pip install gcs-oauth2-boto-plugin
    >>>import boto
    >>>import gcs_oauth2_boto_plugin
  3. Set up your boto configuration file to use OAuth2.0.

    You can configure your boto configuration file to use service account or user account credentials. Service account credentials are the preferred type of credential to use when authenticating on behalf of a service or application. User account credentials are the preferred type of credentials for authenticating requests on behalf of a specific user (i.e., a human). For more information about these two credential types, see Supported Credential Types.

    Service accounts

    To use service account credentials:

    1. Use an existing service account or create a new one, and download the associated private key. Downloading the key as a .json file is the default and is preferred, but using the .p12 format is also supported.

    2. Configure the .boto file with the service account. You can do this with gsutil:

      gsutil config -e

      The command prompts you for the service account email address and the location of the service account private key (.json or .p12). Be sure to have the private key on the computer where you are running the gsutil command.

    User accounts

    To use user account credentials:

    1. If you don't already have a .boto file create one. You can do this with gsutil.

      gsutil config
    2. Use an existing client ID for an application or create a new one.

    3. Edit the .boto file. In the [OAuth2] section, specify the client_id and client_secret values with the ones you generated.

    4. Run the gsutil config again command to generate a refresh token based on the client ID and secret you entered.

      If you get an error message that indicates the .boto cannot be backed up, remove or rename the backup configuration file .boto.bak.

    5. Configure refresh token fallback logic.

      The gcs-oauth2-boto-plugin requires fallback logic for generating auth tokens when you are using application credentials. Fallback logic is not needed when you use a service account.

      You have the following options for enabling fallback:

      • Set the client_id and the client_secret in the .boto config file. This is the recommended option, and it is required for using gsutil with your new .boto config file.

      • Set environment variables OAUTH2_CLIENT_ID and OAUTH2_CLIENT_SECRET.

      • Use the SetFallbackClientIdAndSecret function as shown in the examples below.

Working with Cloud Storage

Setting up your Python source file

To start this tutorial, use your favorite text editor to create a new Python file. Then, add the following directives, import statements, configuration, and constant assignments shown.

Note that the following code uses SetFallbackClientIdAndSecret function as a fallback for generating refresh tokens. See Setting up your environment for other ways to specify a fallback. If you are using a service account to authenticate, you do not need to include the fallback logic.


import boto
import gcs_oauth2_boto_plugin
import os
import shutil
import StringIO
import tempfile
import time

# URI scheme for Cloud Storage.
# URI scheme for accessing local files.
LOCAL_FILE = 'file'

# Fallback logic. In
# under Credentials, create a new client ID for an installed application.
# Required only if you have not configured client ID/secret in
# the .boto file or as environment variables.
CLIENT_ID = 'your client id'
CLIENT_SECRET = 'your client secret'
gcs_oauth2_boto_plugin.SetFallbackClientIdAndSecret(CLIENT_ID, CLIENT_SECRET)

Creating buckets

This following code creates two buckets. Because bucket names must be globally unique (see the naming guidelines), a timestamp is appended to each bucket name to help guarantee uniqueness.

If these bucket names are already in use, you'll need to modify the code to generate unique bucket names.

now = time.time()
CATS_BUCKET = 'cats-%d' % now
DOGS_BUCKET = 'dogs-%d' % now

# Your project ID can be found at
# If there is no domain for your project, then project_id = 'YOUR_PROJECT'

  # Instantiate a BucketStorageUri object.
  uri = boto.storage_uri(name, GOOGLE_STORAGE)
  # Try to create the bucket.
    # If the default project is defined,
    # you do not need the headers.
    # Just call: uri.create_bucket()
    header_values = {"x-goog-project-id": project_id}

    print 'Successfully created bucket "%s"' % name
  except boto.exception.StorageCreateError, e:
    print 'Failed to create bucket:', e

Listing buckets

To retrieve a list of all buckets, call storage_uri() to instantiate a BucketStorageUri object, specifying the empty string as the URI. Then, call the get_all_buckets() instance method.

uri = boto.storage_uri('', GOOGLE_STORAGE)
# If the default project is defined, call get_all_buckets() without arguments.
for bucket in uri.get_all_buckets(headers=header_values):

Uploading objects

To upload objects, create a file object (opened for read) that points to your local file and a storage URI object that points to the destination object on Cloud Storage. Call the set_contents_from_file() instance method, specifying the file handle as the argument.

# Make some temporary files.
temp_dir = tempfile.mkdtemp(prefix='googlestorage')
tempfiles = {
    'labrador.txt': 'Who wants to play fetch? Me!',
    'collie.txt': 'Timmy fell down the well!'}
for filename, contents in tempfiles.iteritems():
  with open(os.path.join(temp_dir, filename), 'w') as fh:

# Upload these files to DOGS_BUCKET.
for filename in tempfiles:
  with open(os.path.join(temp_dir, filename), 'r') as localfile:

    dst_uri = boto.storage_uri(
        DOGS_BUCKET + '/' + filename, GOOGLE_STORAGE)
    # The key-related functions are a consequence of boto's
    # interoperability with Amazon S3 (which employs the
    # concept of a key mapping to localfile).
  print 'Successfully created "%s/%s"' % (
      dst_uri.bucket_name, dst_uri.object_name)

shutil.rmtree(temp_dir)  # Don't forget to clean up!

Listing objects

To list all objects in a bucket, call storage_uri() and specify the bucket's URI and the Cloud Storage URI scheme as the arguments. Then, retrieve a list of objects using the get_bucket() instance method.

uri = boto.storage_uri(DOGS_BUCKET, GOOGLE_STORAGE)
for obj in uri.get_bucket():
  print '%s://%s/%s' % (uri.scheme, uri.bucket_name,
  print '  "%s"' % obj.get_contents_as_string()

Downloading and copying objects

The following code reads objects in DOGS_BUCKET and copies them to both your home directory and CATS_BUCKET. It also demonstrates that you can use the boto library to operate against both local files and Cloud Storage objects using the same interface.

dest_dir = os.getenv('HOME')
for filename in ('collie.txt', 'labrador.txt'):
  src_uri = boto.storage_uri(
      DOGS_BUCKET + '/' + filename, GOOGLE_STORAGE)

  # Create a file-like object for holding the object contents.
  object_contents = StringIO.StringIO()

  # The unintuitively-named get_file() doesn't return the object
  # contents; instead, it actually writes the contents to
  # object_contents.

  local_dst_uri = boto.storage_uri(
      os.path.join(dest_dir, filename), LOCAL_FILE)

  bucket_dst_uri = boto.storage_uri(
      CATS_BUCKET + '/' + filename, GOOGLE_STORAGE)

  for dst_uri in (local_dst_uri, bucket_dst_uri):


Streaming Transfers

To perform a streaming upload, use the following code:

dst_uri = boto.storage_uri(<bucket> + '/' + <object>, 'gs')
dst_uri.new_key().set_contents_from_stream(<stream object>)

For example, the following code performs a streaming upload of a file named data_file to an object with the same name:

filename = 'data_file'
MY_BUCKET = 'my_app_bucket'
my_stream = open(filename, 'rb')
dst_uri = boto.storage_uri(MY_BUCKET + '/' + filename, 'gs')

To perform a streaming download, use the following code:

import sys

src_uri = boto.storage_uri(<bucket> + '/' + <object>, 'gs')

For example, the following code performs a streaming download of an object named data_file:

downloaded_file = 'saved_data_file'
MY_BUCKET = 'my_app_bucket'
object_name = 'data_file'
src_uri = boto.storage_uri(MY_BUCKET + '/' + object_name, 'gs')

Changing object ACLs

The following code grants the specified Google account FULL_CONTROL permissions for labrador.txt. Remember to replace valid-email-address with a valid Google account email address.

uri = boto.storage_uri(DOGS_BUCKET + '/labrador.txt', GOOGLE_STORAGE)
print str(uri.get_acl())
uri.add_email_grant('FULL_CONTROL', 'valid-email-address')
print str(uri.get_acl())

Reading bucket and object metadata

This code retrieves and prints the metadata associated with a bucket and an object.

# Print ACL entries for DOGS_BUCKET.
bucket_uri = boto.storage_uri(DOGS_BUCKET, GOOGLE_STORAGE)
for entry in bucket_uri.get_bucket().get_acl().entries.entry_list:
  entry_id =
  if not entry_id:
    entry_id = entry.scope.email_address
  print 'SCOPE: %s' % entry_id
  print 'PERMISSION: %s\n' % entry.permission

# Print object metadata and ACL entries.
object_uri = boto.storage_uri(DOGS_BUCKET + '/labrador.txt', GOOGLE_STORAGE)
key = object_uri.get_key()
print ' Object size:\t%s' % key.size
print ' Last mod:\t%s' % key.last_modified
print ' MIME type:\t%s' % key.content_type
print ' MD5:\t%s' % key.etag.strip('"\'') # Remove surrounding quotes
for entry in key.get_acl().entries.entry_list:
  entry_id =
  if not entry_id:
    entry_id = entry.scope.email_address
  print 'SCOPE: %s' % entry_id
  print 'PERMISSION: %s\n' % entry.permission

Deleting objects and buckets

To conclude this tutorial, this code deletes the objects and buckets that you have created. A bucket must be empty before it can be deleted, so its objects are first deleted.

for bucket in (CATS_BUCKET, DOGS_BUCKET):
  uri = boto.storage_uri(bucket, GOOGLE_STORAGE)
  for obj in uri.get_bucket():
    print 'Deleting object: %s...' %
  print 'Deleting bucket: %s...' % uri.bucket_name