Google Cloud Platform

Python Client for Google Cloud Storage

Python |Java |PHP |Go

  1. About Google Cloud Storage
  2. About the client library
  3. Downloading the client library
  4. Using the client library
    1. Alternative methods for accessing Cloud Storage
  5. Key Cloud Storage concepts
    1. Buckets, objects, and ACLs
    2. Modifying Cloud Storage objects
    3. Cloud Storage and subdirectories
    4. Retries and exponential backoff
  6. Using the client library with the development app server
  7. Pricing, quotas, and limits
  8. What to do next

About Google Cloud Storage

Google Cloud Storage is useful for storing and serving large files. Additionally, Cloud Storage offers the use of access control lists (ACLs), and the ability to resume upload operations if they're interrupted, and many other features. The Google Cloud Storage client library makes use of this resume capability automatically for your app, providing you with a robust way to stream data into Cloud Storage.

About the client library

The Google Cloud Storage client library lets your application read files from and write files to buckets in Cloud Storage. This library supports reading and writing large amounts of data to Cloud Storage, with internal error handling and retries, so you don't have to write your own code to do this. Moreover, it provides read buffering with prefetch so your app can be more efficient. The client library uses HTTPS when reading and writing to Cloud Storage so your data is encrypted during transfer.

The Google Cloud Storage client library provides the following functionality:

  • An open method that returns a file-like buffer on which you can invoke standard Python file operations for reading and writing.
  • A listbucket method for listing the contents of a Cloud Storage bucket.
  • A stat method for obtaining metadata about a specific file.
  • A delete method for deleting files from Cloud Storage.

Downloading the client library

For download instructions and distribution contents, see the downloads page.

Using the client library

Choose from one of the following options:

Alternative methods for accessing Cloud Storage

The Google Cloud Storage client library provides a way to read from and write to Cloud Storage that is closely integrated with Google App Engine, enabling App Engine apps to create objects in and serve them from Cloud Storage.

However, there are other ways to access Cloud Storage from App Engine besides using the Google Cloud Storage client library. You can use any of these methods as well:

Blobstore API

You can use the Blobstore API to upload objects to and serve objects from Cloud Storage using the BlobStore API. You'll need to use the create_gs_key() function to create a blob key representing the Cloud Storage object. This approach is useful for uploading files from a web page. When the Blobstore API is used together with the Images API, you get a powerful way to serve images, because you can serve images directly from Cloud Storage, bypassing the App Engine app, which saves on instance hour costs.

Cloud Storage REST API

You can use the Cloud Storage REST API to read and write data to Cloud Storage. The REST API can be a good choice if you're looking for a specific feature it provides that the client library doesn't provide.

However, the client library includes App Engine optimizations, so using the REST API might require extra development time. Note: The App Engine development server supports the Cloud Storage Client. It doesn't support the REST API.

Cloud Storage Browser

If you need to upload objects quickly, you can use the Cloud Storage browser.

Key Cloud Storage concepts

See the Cloud Storage documentation for a full list of concepts and techniques. The following section summarizes important concepts for the Google Cloud Storage client library.

Buckets, objects, and ACLs

A bucket is the storage location you read files from and write files to. You must always specify a bucket when using the Google Cloud Storage client library. Your project can access multiple buckets. The client library doesn't support bucket creation. You can create these manually by using the Cloud Storage browser or the gsutil tool.

Access control lists (ACLs) control access to the buckets and to the objects contained in them. Your project and your App Engine app are added to the ACL that permits bucket access during activation. The ACL that permits bucket access is distinct from the potentially many ACLs governing the objects in that bucket. Thus, your app has read and write privileges to the bucket(s) it is activated for, but it only has full rights to the objects it creates in the bucket. Your app's access to objects created by other apps or persons is limited to the rights given to your app by the objects' creator.

If an object is created in the bucket without an ACL explicitly defined for it, it uses the default object ACL assigned to the bucket by the bucket owner. If the bucket owner has not specified a default object ACL, the object default is public-read, which means that anyone allowed bucket access can read the object.

ACLs and the Google Cloud Storage client library

An app using the client library cannot change the bucket ACL, but it can specify an ACL that controls access to the objects it creates. The available ACL settings are described under documentation for the open method .

Modifying Cloud Storage objects

Once you have created an object in a bucket, it cannot be modified or appended. You must overwrite the object with a new object of the same name that contains your desired changes.

Cloud Storage and subdirectories

The Cloud Storage client library lets you supply subdirectory delimiters when you create an object, but there are no true subdirectories in Cloud Storage. Instead, a subdirectory in Cloud Storage is a part of the object filename.

For example, you might assume that creating an object somewhere/over/the/rainbow.mp3 would store the file rainbow.mp3 in the subdirectory somewhere/over/the/. Instead, the object name is set to somewhere/over/the/rainbow.mp3. This concept is important to understand when using listbucket filtering.

Note that if the above listing behavior is not optimal for you, listbucket has an optional directory emulation mode. See listbucket for more information.

Retries and exponential backoff

The Google Cloud Storage client library provides a configurable mechanism for automatic request retries in event of timeout failures when accessing Cloud Storage. The same mechanism also provides exponential backoff to determine an optimal processing rate. For a description of exponential backoff in Cloud Storage, see backoff.

To change the default values for retries and backoff, use the RetryParams class.

Using the client library with the development app server

You can use the client library with the development server from SDK version 1.8.1 and greater. This provides Cloud Storage emulation using the local disk.

Pricing, quotas, and limits

There are no bandwidth charges associated with making Google Cloud Storage client library calls to Cloud Storage. There are operations and storage charges, however,

and the calls count against your URL fetch quota, as the library uses the URL Fetch service to interact with Cloud Storage.

You'll need to create a Cloud Storage bucket, as described in Activating Google Cloud Storage. Notice that Google Cloud Storage is a pay-to-use service; you will be charged according to the Cloud Storage price sheet.

What to do next

To create, deploy, and run your app:

  1. Download the client library.
  2. Create an App Engine project and activate Cloud Storage.
  3. Go through the brief Getting Started instructions for a quick orientation in using the library.
  4. Upload and deploy your app to production App Engine.
  5. Test the app for expected behavior with Cloud Storage.