Reading and writing to Cloud Storage

This document describes how to store and retrieve data using the Cloud Storage client library. It assumes that you completed the tasks described in Setting up for Cloud Storage to activate a Cloud Storage bucket and download the client libraries. It also assumes that you know how to build an App Engine application.

For additional code samples, see Cloud Storage client libraries

Required imports

Use the code snippet below for accessing Cloud Storage using the client library:

# Imports the Google Cloud client library
from google.cloud import storage

# Instantiates a client
storage_client = storage.Client()

# The name for the new bucket
bucket_name = "my-new-bucket"

# Creates the new bucket
bucket = storage_client.create_bucket(bucket_name)

print(f"Bucket {bucket.name} created.")

Specifying the Cloud Storage bucket

Before doing any operations in Cloud Storage, you need to supply the bucket name. 

# Imports the Google Cloud client library
from google.cloud import storage

# Instantiates a client
storage_client = storage.Client()

# The name for the new bucket
bucket_name = "my-new-bucket"

# Creates the new bucket
bucket = storage_client.create_bucket(bucket_name)

print(f"Bucket {bucket.name} created.")

The easiest way to do specify a bucket name is to use the default bucket for your project. The call to get_default_gcs_bucket_name succeeds only if you have created the default bucket for your project.

Writing to Cloud Storage

The following sample shows how to write to the bucket:

from google.cloud import storage


def write_read(bucket_name, blob_name):
    """Write and read a blob from GCS using file-like IO"""
    # The ID of your GCS bucket
    # bucket_name = "your-bucket-name"

    # The ID of your new GCS object
    # blob_name = "storage-object-name"

    storage_client = storage.Client()
    bucket = storage_client.bucket(bucket_name)
    blob = bucket.blob(blob_name)

    # Mode can be specified as wb/rb for bytes mode.
    # See: https://docs.python.org/3/library/io.html
    with blob.open("w") as f:
        f.write("Hello world")

    with blob.open("r") as f:
        print(f.read())

Note the following:

  • In the call to open the file for write, the sample specifies certain Cloud Storage headers that write custom metadata for the file; this metadata can be retrieved using cloudstorage.stat(). You can find the list of supported headers in the cloudstorage.open() reference.

  • The x-goog-acl header is not set. That means the default Cloud Storage ACL of public read is going to be applied to the object when it is written to the bucket.

  • Ensure you invoke the function to close the file after you finish the write. If you don't do this, the file is not written to Cloud Storage. Be aware that after you call the Python file function close(), you cannot append to the file. If you need to modify a file, you'll have to call the Python file function open() to open the file again in write mode, which does an overwrite, not an append.

Reading from Cloud Storage

The following sample shows how to read a full file from the bucket:

from google.cloud import storage


def write_read(bucket_name, blob_name):
    """Write and read a blob from GCS using file-like IO"""
    # The ID of your GCS bucket
    # bucket_name = "your-bucket-name"

    # The ID of your new GCS object
    # blob_name = "storage-object-name"

    storage_client = storage.Client()
    bucket = storage_client.bucket(bucket_name)
    blob = bucket.blob(blob_name)

    # Mode can be specified as wb/rb for bytes mode.
    # See: https://docs.python.org/3/library/io.html
    with blob.open("w") as f:
        f.write("Hello world")

    with blob.open("r") as f:
        print(f.read())

In both examples, the blob_name argument that you pass to cloudstorage.open() is the path to your file in YOUR_BUCKET_NAME/PATH_IN_GCS format. Note that the default for cloudstorage.open() is read-only mode. You do not need to specify a mode when opening a file to read it.

Listing bucket contents

The sample code shows how to page through a bucket with blob type content :

from google.cloud import storage


def list_blobs(bucket_name):
    """Lists all the blobs in the bucket."""
    # bucket_name = "your-bucket-name"

    storage_client = storage.Client()

    # Note: Client.list_blobs requires at least package version 1.17.0.
    blobs = storage_client.list_blobs(bucket_name)

    # Note: The call returns a response only when the iterator is consumed.
    for blob in blobs:
        print(blob.name)

Note that the complete file name is displayed as one string without directory delimiters. If you want to display the file with its more recognizable directory hierarchy, set the delimiter parameter to the directory delimiter you want to use.

Deleting files in Cloud Storage

The code below demonstrates how to delete a file from Cloud Storage using the cloudstorage.delete() method (imported as gcs).

from google.cloud import storage


def delete_blob(bucket_name, blob_name):
    """Deletes a blob from the bucket."""
    # bucket_name = "your-bucket-name"
    # blob_name = "your-object-name"

    storage_client = storage.Client()

    bucket = storage_client.bucket(bucket_name)
    blob = bucket.blob(blob_name)
    generation_match_precondition = None

    # Optional: set a generation-match precondition to avoid potential race conditions
    # and data corruptions. The request to delete is aborted if the object's
    # generation number does not match your precondition.
    blob.reload()  # Fetch blob metadata to use in generation_match_precondition.
    generation_match_precondition = blob.generation

    blob.delete(if_generation_match=generation_match_precondition)

    print(f"Blob {blob_name} deleted.")

This example cleans up the files that were written to the bucket in the Writing to Cloud Storage section.

What's next