Python 2 is no longer supported by the community. We recommend that you migrate Python 2 apps to Python 3.

Migrating to the Cloud Client Library for Storage

Cloud Storage enables your application to serve large data objects such as video or image files, and enables your users to upload large data files. In the Python 2 runtime, App Engine provides its own client library for writing and reading objects in Cloud Storage. This App Engine library is not available in newer App Engine runtimes, including the Python 3 runtime.

If your Python 2 app uses the GoogleAppEngineCloudStorageClient library, you need to migrate to the Cloud Client Library for Cloud Storage before you can run the app in the Python 3 runtime. Note that you only need to migrate your app to use a new client library. All of the data objects and permissions in your existing Cloud Storage buckets remain unchanged, and you can access your existing buckets using the new client library.

Key differences between the App Engine and Cloud Client Libraries

The following list provides a brief comparison of the two libraries:

  • The Cloud Client Library supports all of the Cloud Storage features enabled by the App Engine client library, such as reading, writing, removing, and listing objects. Migration should require only small changes to your code.

    The Cloud Client Library also supports additional features, such as creating and labeling buckets, and retrieving older versions of objects.

  • In the App Engine library, the function that retrieves a list of objects works asynchronously. The Cloud Client Library doesn't provide an asynchronous function for listing objects, though you can use paging and iterate through a small set of objects.

  • The App Engine client library requires you to use access control lists (ACL) to control access to buckets and objects. However, Cloud Storage and the Cloud Client Library support two systems for granting users permission to access your buckets and objects: ACLs and uniform bucket-level access. Uniform bucket-level access provides a simpler, consistent access control experience across all of your Cloud resources.

    All ACLs that you used with the App Engine client library remain in effect for your existing buckets after you migrate to the Cloud Client Library, and you can continue to use ACLs if needed.

    If uniform bucket-level access meets your needs, we recommend you use this simpler system for any new buckets you create. While you can convert existing buckets to use uniform bucket-level access, this may require significant changes to how your app secures access to its storage objects.

Before you start migrating

Understanding Cloud Storage permissions

By default, your app's default service account has read and write privileges to the buckets in your project, and it has full rights to the objects it creates, both before and after migration.

If you used a different service account or user account to secure access to your Cloud Storage buckets and objects, be sure you continue to use the same accounts and authentication techniques before and after migration.

Overview of the migration process

To migrate your Python app to use the Cloud Client Library for Cloud Storage instead of the App Engine client library:

  1. Install the Cloud Client Library for Cloud Storage.

  2. Update your code to use the Cloud Client Library.

  3. Test your updates.

  4. Deploy your app to App Engine.

Installing the Cloud Client Library for Cloud Storage

To make the Cloud Client Library for Cloud Storage available to your app when it runs in App Engine:

  1. Create a requirements.txt file in the same folder as your app.yaml file and add the following lines:

     google-cloud-storage==1.24.1
     googleapis_common_protos
    

    We recommend you use the 1.24.1 version of the Cloud Storage client library since it supports Python 2.7 apps.

    Python 2 apps require googleapis_common_protos to access Google Cloud services.

  2. In your app's app.yaml file, specify the Google RPC and SSL libraries in the libraries section:

    libraries:
    - name: grpcio
      version: 1.0.0
    - name: ssl
      version: latest
    
  3. Create a directory to store your third-party libraries, such as lib/. Then use pip install to install the libraries into the directory. For example:

      pip install -t lib -r requirements.txt
      

  4. Create an appengine_config.py file in the same folder as your app.yaml file. Add the following to your appengine_config.py file:

    # appengine_config.py
    import pkg_resources
    from google.appengine.ext import vendor
    
    # Set path to your libraries folder.
    path = 'lib'
    # Add libraries installed in the path folder.
    vendor.add(path)
    # Add libraries to pkg_resources working set to find the distribution.
    pkg_resources.working_set.add_entry(path)
    

    The appengine_config.py file in the preceding example assumes that the the lib folder is located in the current working directory. If you can't guarantee that lib will always be in the current working directory, specify the full path to the lib folder. For example:

    import os
    path = os.path.join(os.path.dirname(os.path.realpath(__file__)), 'lib')
    

For local development, we recommend that you install dependencies in a virtual environment such as virtualenv for Python 2.

Updating your code to use the Cloud Client Library

Creating a Cloud Storage Client

To use the Cloud Client Library for Cloud Storage, create a Client object. The client contains credentials and other data needed to connect to Cloud Storage. For example:

from google.cloud import storage

client = storage.Client()

In the default authorization scenario described previously, the Cloud Storage client contains credentials from App Engine's default service account, which is authorized to interact with your project's buckets and objects. If you aren't working in this default scenario, see Application Default Credentials (ADC) for information on how to provide credentials.

Using Cloud Client Library methods

The following table summarizes which methods from the Cloud Client Library to use when implementing specific Cloud Storage features.

Cloud Storage feature Cloud client methods
List objects in a bucket Client.list_blobs

See a simple code sample and a sample of emulating directory mode.

To page through the list of objects, use the pages property of the iterator object that Client.list_blobs returns.

Write to storage blob.upload_from_file
blob.upload_from_filename
blob.upload_as_string

See a code sample.

Specify ACLs To apply a predefined ACL, use the predefined_acl parameter when you create a bucket or create an object.

For more fine-grained control, use bucket.acl.reload() or blob.acl.reload() to retrieve any ACLs currently defined. Then use ACL methods to add or remove ACLs. See a code sample.

Read from storage blob.download_to_file
blob.download_to_filename
blob.download_as_string

See a code sample.

Delete an object blob.delete

See a code sample.

Copy an object bucket.copy_blob

See a code sample.

Read metadata for an object blob.propertyname
and
blob.metadata

See a code sample.

Testing your updates

You can test your updates to your app in a local environment, but all Cloud Storage requests must be sent over the internet to an actual Cloud Storage bucket. Neither App Engine nor Cloud Storage provides a Cloud Storage emulator.

For more information about testing Python 2 apps, see Using the local development server.

For more information about testing Python 3 apps, see Testing and deploying your application.

Deploying your app

Once you are ready to deploy your app, you should:

  1. Test the app on App Engine.

  2. If the app runs without errors, use traffic splitting to slowly ramp up traffic for your updated app. Monitor the app closely for any issues before routing more traffic to the updated app.