- About Google Cloud Storage
- About the client library
- Downloading the client library
- Using the client library
- Key Cloud Storage concepts
- Using the client library with the development app server
- Pricing, quotas, and limits
- What to do next
About Google Cloud Storage
Google Cloud Storage is useful for storing and serving large files. Additionally, Cloud Storage offers the use of access control lists (ACLs), and the ability to resume upload operations if they're interrupted, and many other features. The Google Cloud Storage client library makes use of this resume capability automatically for your app, providing you with a robust way to stream data into Cloud Storage.
About the client library
The Google Cloud Storage client library lets your application read files from and write files to buckets in Cloud Storage. This library supports reading and writing large amounts of data to Cloud Storage, with internal error handling and retries, so you don't have to write your own code to do this. Moreover, it provides read buffering with prefetch so your app can be more efficient. The client library uses HTTPS when reading and writing to Cloud Storage so your data is encrypted during transfer.
The Google Cloud Storage client library provides the following functionality:
openmethod that returns a file-like buffer on which you can invoke standard Python file operations for reading and writing.
listbucketmethod for listing the contents of a Cloud Storage bucket.
statmethod for obtaining metadata about a specific file.
deletemethod for deleting files from Cloud Storage.
Downloading the client library
For download instructions and distribution contents, see the downloads page.
Using the client library
Choose from one of the following options:
- Use the default Cloud Storage bucket: an already configured bucket with free quota.
- If you don't want to use the default bucket, you must first activate Cloud Storage.
Alternative methods for accessing Cloud Storage
The Google Cloud Storage client library provides a way to read from and write to Cloud Storage that is closely integrated with Google App Engine, enabling App Engine apps to create objects in and serve them from Cloud Storage.
However, there are other ways to access Cloud Storage from App Engine besides using the Google Cloud Storage client library. You can use any of these methods as well:
You can use the Blobstore API to upload objects to and serve objects from Cloud Storage using the BlobStore API. You'll need to use the
create_gs_key() function to create a blob key representing the Cloud Storage object. This approach is useful for uploading files from a web page. When the Blobstore API is used together with the Images API, you get a powerful way to serve images, because you can serve images directly from Cloud Storage, bypassing the App Engine app, which saves on instance hour costs.
Cloud Storage REST API
You can use the Cloud Storage REST API to read and write data to Cloud Storage. The REST API can be a good choice if you're looking for a specific feature it provides that the client library doesn't provide.However, the client library includes App Engine optimizations, so using the REST API might require extra development time. Note: The App Engine development server supports the Cloud Storage Client. It doesn't support the REST API.
Cloud Storage Browser
If you need to upload objects quickly, you can use the Cloud Storage browser.
Key Cloud Storage concepts
See the Cloud Storage documentation for a full list of concepts and techniques. The following section summarizes important concepts for the Google Cloud Storage client library.
Buckets, objects, and ACLs
A bucket is the storage location you read files from and write files to. You must always specify a bucket when using the Google Cloud Storage client library. Your project can access multiple buckets. The client library doesn't support bucket creation. You can create these manually by using the Cloud Storage browser or the gsutil tool.
Access control lists (ACLs) control access to the buckets and to the objects contained in them. Your project and your App Engine app are added to the ACL that permits bucket access during activation. The ACL that permits bucket access is distinct from the potentially many ACLs governing the objects in that bucket. Thus, your app has read and write privileges to the bucket(s) it is activated for, but it only has full rights to the objects it creates in the bucket. Your app's access to objects created by other apps or persons is limited to the rights given to your app by the objects' creator.
If an object is created in the bucket without an ACL explicitly defined for it, it uses the default object ACL assigned to the bucket by the bucket owner. If the bucket owner has not specified a default object ACL, the object default is
public-read, which means that anyone allowed bucket access can read the object.
ACLs and the Google Cloud Storage client libraryAn app using the client library cannot change the bucket ACL, but it can specify an ACL that controls access to the objects it creates. The available ACL settings are described under documentation for the open method .
Modifying Cloud Storage objects
Once you have created an object in a bucket, it cannot be modified or appended. You must overwrite the object with a new object of the same name that contains your desired changes.
Cloud Storage and subdirectories
The Cloud Storage client library lets you supply subdirectory delimiters when you create an object, but there are no true subdirectories in Cloud Storage. Instead, a subdirectory in Cloud Storage is a part of the object filename.
For example, you might assume that creating an object
somewhere/over/the/rainbow.mp3 would store the file
rainbow.mp3 in the subdirectory
somewhere/over/the/. Instead, the object name is set to
somewhere/over/the/rainbow.mp3. This concept is important to understand when using
listbuckethas an optional directory emulation mode. See listbucket for more information.
Retries and exponential backoff
The Google Cloud Storage client library provides a configurable mechanism for automatic request retries in event of timeout failures when accessing Cloud Storage. The same mechanism also provides exponential backoff to determine an optimal processing rate. For a description of exponential backoff in Cloud Storage, see backoff.
To change the default values for retries and backoff, use the RetryParams class.
Using the client library with the development app server
You can use the client library with the development server from SDK version 1.8.1 and greater. This provides Cloud Storage emulation using the local disk.
Pricing, quotas, and limits
There are no bandwidth charges associated with making Google Cloud Storage client library calls to Cloud Storage. There are operations and storage charges, however,
and the calls count against your URL fetch quota, as the library uses the URL Fetch service to interact with Cloud Storage.
If you don't use the default bucket, you'll need to activate Cloud Storage. If you choose this, there is no free quota and any data stored in Cloud Storage is charged the usual Cloud Storage data storage fees. Cloud Storage is a pay-to-use service; you will be charged according to the Cloud Storage price sheet.
What to do next
To create, deploy, and run your app:
- Download the client library.
- Create an App Engine project and activate Cloud Storage.
- Optionally, if you have an existing app that uses the older Cloud Storage API, migrate your app.
- Go through the brief Getting Started instructions for a quick orientation in using the library.
- Upload and deploy your app to production App Engine.
- Test the app for expected behavior with Cloud Storage.