Using Cloud Storage with Python

This part of the Bookshelf app tutorial shows how the sample app stores images in Cloud Storage.

This page is part of a multipage tutorial. To start from the beginning and read the setup instructions, go to Python Bookshelf app.

Creating a Cloud Storage bucket

To use Cloud Storage to store and serve binary data, you need a container, or bucket, for binary objects.

The following instructions detail how to create a Cloud Storage bucket. Buckets are the basic containers that hold your data in Cloud Storage.

  1. In your terminal window, create a Cloud Storage bucket, where YOUR_BUCKET_NAME represents the name of your bucket:

    gsutil mb gs://YOUR_BUCKET_NAME
    
  2. To view uploaded images in the Bookshelf app, set the bucket's default access control list (ACL) to public-read:

    gsutil defacl set public-read gs://YOUR_BUCKET_NAME
    

Configuring settings

This section uses code in the 3-binary-data directory. Edit the files and run commands in this directory.

  1. Open the config.py file for editing and replace the following values:
    • Set the value of [PROJECT_ID] to your project ID, which is visible in the GCP Console.
    • Set the value of [DATA_BACKEND] to the same value you used during the Using structured data tutorial.
    • If you are using Cloud SQL or MongoDB, set the values under the Cloud SQL or Mongo section to the same values you used during the Using structured data step.
    • Set the value of [CLOUD_STORAGE_BUCKET] to your Cloud Storage bucket name.
  2. Save and close the config.py file.

If you are using Cloud SQL:

  1. Open the app.yaml file for editing.
  2. Set the value of cloudsql-instance to the same value used for [CLOUDSQL_CONNECTION_NAME] in the config.py file. Use the format project:region:cloudsql-instance. Uncomment this entire line.
  3. Save and close the app.yaml file.

Installing dependencies

To create a virtual environment and install dependencies, use the following commands:

Linux/macOS

virtualenv -p python3 env
source env/bin/activate
pip install -r requirements.txt

Windows

virtualenv -p python3 env
env\scripts\activate
pip install -r requirements.txt

Running the app on your local machine

  1. Start a local web server:

    python main.py
    
  2. In your browser, enter the following address:

    http://localhost:8080
    
Now you can browse the app's web pages, add books with cover images, edit books, and delete books.

Press Control+C to exit the worker and then the local web server.

Deploying the app to the App Engine flexible environment

  1. Deploy the sample app:

    gcloud app deploy
    
  2. In your browser, enter the following address. Replace [YOUR_PROJECT_ID] with your GCP project ID:

    https://[YOUR_PROJECT_ID].appspot.com
    

If you update your app, you deploy the updated version by entering the same command that you used to deploy the app. The deployment creates a new version of your app and promotes it to the default version. The earlier versions of your app remain, as do their associated virtual machine (VM) instances. All of these app versions and VM instances are billable resources. To reduce costs, delete the non-default versions of your app.

To delete an app version:

  1. In the GCP Console, go to the Versions page for App Engine.

    Go to the Versions page

  2. Select the checkbox for the non-default app version you want to delete.
  3. Click Delete to delete the app version.

For more information about cleaning up billable resources, see the Cleaning up section in the final step of this tutorial.

App structure

Binary data sample structure

The application uses Cloud Storage to store binary data, pictures in this case, while continuing to use a structured database, either Cloud Datastore, Cloud SQL, or MongoDB, to store book information.

Understanding the code

The following sections describe the Bookshelf app's code and explain how it works.

Handling user uploads

The following code example uses multipart/form-data instead of enctype in the add/edit form to let users upload image files. Add a field for the image.

{% extends "base.html" %}

{% block content %}
<h3>{{action}} book</h3>

<form method="POST" enctype="multipart/form-data">

  <div class="form-group">
    <label for="title">Title</label>
    <input type="text" name="title" id="title" value="{{book.title}}" class="form-control"/>
  </div>

  <div class="form-group">
    <label for="author">Author</label>
    <input type="text" name="author" id="author" value="{{book.author}}" class="form-control"/>
  </div>

  <div class="form-group">
    <label for="publishedDate">Date Published</label>
    <input type="text" name="publishedDate" id="publishedDate" value="{{book.publishedDate}}" class="form-control"/>
  </div>

  <div class="form-group">
    <label for="description">Description</label>
    <textarea name="description" id="description" class="form-control">{{book.description}}</textarea>
  </div>

  <div class="form-group">
    <label for="image">Cover Image</label>
    <input type="file" name="image" id="image" class="form-control"/>
  </div>

  <div class="form-group hidden">
    <label for="imageUrl">Cover Image URL</label>
    <input type="text" name="imageUrl" id="imageUrl" value="{{book.imageUrl}}" class="form-control"/>
  </div>

  <button type="submit" class="btn btn-success">Save</button>
</form>

{% endblock %}

The Flask framework has built-in functionality to parse file uploads. Flask makes the file object available in the files field of the request object. Because the image value might not exist if the user doesn't upload a file, use the get method to access the field instead of the usual dictionary syntax. That way, if there is no file, the framework returns None instead of raising a KeyError.

image_url = upload_image_file(request.files.get('image'))

The upload_image_file call dispatches to a helper function, also in crud.py, that calls the storage.upload_file function with the appropriate Cloud Storage bucket and allowed extensions defined in the config file.

def upload_image_file(file):
    """
    Upload the user-uploaded file to Google Cloud Storage and retrieve its
    publicly-accessible URL.
    """
    if not file:
        return None

    public_url = storage.upload_file(
        file.read(),
        file.filename,
        file.content_type
    )

    current_app.logger.info(
        "Uploaded file %s as %s.", file.filename, public_url)

    return public_url

Uploading to Cloud Storage

After crud.py calls storage.upload_file to handle uploading the files to Cloud Storage, the code checks for an image file extension to make sure only appropriate file types are uploaded. Check the file type to help minimize security risk of a malicious file causing damage to the app.

The six library supports Python 2 and Python 3.

def upload_file(file_stream, filename, content_type):
    """
    Uploads a file to a given Cloud Storage bucket and returns the public url
    to the new object.
    """
    _check_extension(filename, current_app.config['ALLOWED_EXTENSIONS'])
    filename = _safe_filename(filename)

    client = _get_storage_client()
    bucket = client.bucket(current_app.config['CLOUD_STORAGE_BUCKET'])
    blob = bucket.blob(filename)

    blob.upload_from_string(
        file_stream,
        content_type=content_type)

    url = blob.public_url

    if isinstance(url, six.binary_type):
        url = url.decode('utf-8')

    return url

The upload_file function returns the public URL of the image that is returned by Cloud Storage. The URL is then saved to the database.

if image_url:
    data['imageUrl'] = image_url

Serving images from Cloud Storage

Serving the image is straightforward because the URL for the image is public. Serving images directly from Cloud Storage leverages Google's global serving infrastructure, and frees the application for other requests.

<div class="media-left">
  {% if book.imageUrl %}
    <img class="book-image" src="{{book.imageUrl}}">
  {% else %}
    <img class="book-image" src="http://placekitten.com/g/128/192">
  {% endif %}
</div>

Cleaning up

To avoid incurring charges to your Google Cloud Platform account for the resources used in this tutorial:

Delete the project

The easiest way to eliminate billing is to delete the project that you created for the tutorial.

To delete the project:

  1. In the GCP Console, go to the Manage resources page.

    Go to the Manage resources page

  2. In the project list, select the project you want to delete and click Delete .
  3. In the dialog, type the project ID, and then click Shut down to delete the project.

Delete non-default versions of your app

If you don't want to delete your project, you can reduce costs by deleting the non-default versions of your app.

To delete an app version:

  1. In the GCP Console, go to the Versions page for App Engine.

    Go to the Versions page

  2. Select the checkbox for the non-default app version you want to delete.
  3. Click Delete to delete the app version.

Delete your Cloud SQL instance

To delete a Cloud SQL instance:

  1. In the GCP Console, go to the SQL Instances page.

    Go to the SQL Instances page

  2. Click the name of the SQL instance you want to delete.
  3. Click Delete to delete the instance.

Delete your Cloud Storage bucket

To delete a Cloud Storage bucket:

  1. In the GCP Console, go to the Cloud Storage Browser page.

    Go to the Cloud Storage Browser page

  2. Click the checkbox for the bucket you want to delete.
  3. Click Delete to delete the bucket.

Was this page helpful? Let us know how we did:

Send feedback about...