Using Cloud Storage with Python

This part of the Bookshelf tutorial shows how the sample app stores images in Google Cloud Storage.

This page is part of a multi-page tutorial. To start from the beginning and see instructions for setting up, go to Python Bookshelf App.

Creating a Cloud Storage bucket

Cloud Storage allows you to store and serve binary data. A bucket is a high-level container for binary objects.

The following instructions show how to create a Cloud Storage bucket. Buckets are the basic containers that hold your data in Cloud Storage.

To create a bucket:

  1. Invoke the following command in a terminal window:

    gsutil mb gs://[YOUR-BUCKET-NAME]

  2. Set the bucket's default ACL to public-read, which enables users to see their uploaded images:

    gsutil defacl set public-read gs://[YOUR-BUCKET-NAME]

Configuring settings

This section uses code in the 3-binary-data directory. Edit the files and run commands in this directory.

  1. Open config.py for editing.
  2. Set the value of PROJECT_ID to your project ID, which is visible in the GCP Console.
  3. Set DATA_BACKEND to the same value you used during the Using Structured Data tutorial.
  4. If you are using Cloud SQL or MongoDB, set the values under the Cloud SQL or Mongo section to the same values you used during the Using Structured Data step.
  5. Set the value CLOUD_STORAGE_BUCKET to your Cloud Storage bucket name.

  6. Save and close config.py.

If you are using Cloud SQL:

  1. Open app.yaml for editing.
  2. Set the value of cloud_sql_instances to the same value used for CLOUDSQL_CONNECTION_NAME in config.py . It should be in the format project:region:cloudsql-instance. Uncomment this entire line.
  3. Save and close app.yaml.

Installing dependencies

Enter these commands to create a virtual environment and install dependencies:

Linux/macOS

virtualenv -p python3 env
source env/bin/activate
pip install -r requirements.txt

Windows

virtualenv -p python3 env
env\scripts\activate
pip install -r requirements.txt

Running the app on your local machine:

  1. Start a local web server:

    python main.py
    
  2. In your web browser, enter this address:

    http://localhost:8080

Now you can browse the app's web pages, add books with cover images, edit books, and delete books.

Press Control+C to exit the local web server.

Deploying the app to the App Engine flexible environment

  1. Deploy the sample app:

    gcloud app deploy
    
  2. In your web browser, enter this address. Replace [YOUR_PROJECT_ID] with your project ID:

    https://[YOUR_PROJECT_ID].appspot.com
    

If you update your app, you can deploy the updated version by entering the same command you used to deploy the app the first time. The new deployment creates a new version of your app and promotes it to the default version. The older versions of your app remain, as do their associated VM instances. Be aware that all of these app versions and VM instances are billable resources.

You can reduce costs by deleting the non-default versions of your app.

To delete an app version:

  1. In the GCP Console, go to the App Engine Versions page.

    Go to the Versions page

  2. Click the checkbox next to the non-default app version you want to delete.
  3. Click the Delete button at the top of the page to delete the app version.

For complete information about cleaning up billable resources, see the Cleaning up section in the final step of this tutorial.

Application structure

Binary data sample structure

The application uses Cloud Storage to store binary data, pictures in this case, while continuing to use a structured database for the book information, either Cloud Datastore, Cloud SQL, or MongoDB.

Understanding the code

This section walks you through the application code and explains how it works.

Handling user uploads

To allow users to upload images, the add/edit form has been modified to allow file uploads by setting the enctype to multipart/form-data and a new field has been added for the image:

{% extends "base.html" %}

{% block content %}
<h3>{{action}} book</h3>

<form method="POST" enctype="multipart/form-data">

  <div class="form-group">
    <label for="title">Title</label>
    <input type="text" name="title" id="title" value="{{book.title}}" class="form-control"/>
  </div>

  <div class="form-group">
    <label for="author">Author</label>
    <input type="text" name="author" id="author" value="{{book.author}}" class="form-control"/>
  </div>

  <div class="form-group">
    <label for="publishedDate">Date Published</label>
    <input type="text" name="publishedDate" id="publishedDate" value="{{book.publishedDate}}" class="form-control"/>
  </div>

  <div class="form-group">
    <label for="description">Description</label>
    <textarea name="description" id="description" class="form-control">{{book.description}}</textarea>
  </div>

  <div class="form-group">
    <label for="image">Cover Image</label>
    <input type="file" name="image" id="image" class="form-control"/>
  </div>

  <div class="form-group hidden">
    <label for="imageUrl">Cover Image URL</label>
    <input type="text" name="imageUrl" id="imageUrl" value="{{book.imageUrl}}" class="form-control"/>
  </div>

  <button type="submit" class="btn btn-success">Save</button>
</form>

{% endblock %}

The Flask framework has built-in functionality to parse file uploads. Flask makes the file object available in the files field of the request object. Since the image value might not exist if the user doesn't upload a file, the get method is used to access the field instead of the usual dictionary syntax. That way, if there is no file, None is returned instead of a KeyError being thrown.

image_url = upload_image_file(request.files.get('image'))

The upload_image_file call dispatches to a helper function, also in crud.py, that calls our storage.upload_file function with the appropriate Cloud bucket and allowed extensions defined our in the config file.

def upload_image_file(file):
    """
    Upload the user-uploaded file to Google Cloud Storage and retrieve its
    publicly-accessible URL.
    """
    if not file:
        return None

    public_url = storage.upload_file(
        file.read(),
        file.filename,
        file.content_type
    )

    current_app.logger.info(
        "Uploaded file %s as %s.", file.filename, public_url)

    return public_url

Uploading to Cloud Storage

After crud.py calls storage.upload_file to handle uploading the files to Cloud Storage, the code checks for an image file extension to make sure only images are uploaded. Since the image is being hosted on Cloud Storage, there isn't any risk of a malicious file causing damage to the app, but it's still a good practice to allow only the types of files that are appropriate. Note the use of the six library to maintain support for both Python 2 and Python 3.

def upload_file(file_stream, filename, content_type):
    """
    Uploads a file to a given Cloud Storage bucket and returns the public url
    to the new object.
    """
    _check_extension(filename, current_app.config['ALLOWED_EXTENSIONS'])
    filename = _safe_filename(filename)

    client = _get_storage_client()
    bucket = client.bucket(current_app.config['CLOUD_STORAGE_BUCKET'])
    blob = bucket.blob(filename)

    blob.upload_from_string(
        file_stream,
        content_type=content_type)

    url = blob.public_url

    if isinstance(url, six.binary_type):
        url = url.decode('utf-8')

    return url

The upload_file function returns the public URL of the image returned by Cloud Storage. The URL is then saved to the database.

if image_url:
    data['imageUrl'] = image_url

Serving images from Cloud Storage

Because we have the public URL for the image, serving it is straightforward. Serving directly from Cloud Storage is helpful, because the requests leverage Google’s global serving infrastructure, and the application doesn’t have to respond to requests for images. This frees up CPU cycles for other requests.

<div class="media-left">
  {% if book.imageUrl %}
    <img class="book-image" src="{{book.imageUrl}}">
  {% else %}
    <img class="book-image" src="http://placekitten.com/g/128/192">
  {% endif %}
</div>
Was this page helpful? Let us know how we did:

Send feedback about...