Streaming Transfers

Cloud Storage supports streaming transfers, which allow you to stream data to and from your Cloud Storage account without requiring that the data first be saved to a file. Streaming transfers are useful when you want to upload data generated from a process directly into Cloud Storage, or when you want to download data from Cloud Storage into a process.

You can perform streaming transfers either by using the gsutil command-line tool or by building a Python application that uses the boto client library plugin.

Streaming uploads and downloads using gsutil

To use gsutil to perform a streaming upload from a process to a Cloud Storage object, pipe the data to the gsutil cp command and use a dash for the source URL.

The following example streams the data produced by the collect_measurements process into the data_measurements Cloud Storage object:

collect_measurements | gsutil cp - gs://my_app_bucket/data_measurements

To use gsutil to perform a streaming download from a Cloud Storage object to a process, run the gsutil cp command and use a dash for the destination URL, then pipe the data to the process.

The following example streams the data from the data_measurements Cloud Storage object to the analyze_data process:

gsutil cp gs://my_app_bucket/data_measurements - | analyze_data

You can also stream data from a Cloud Storage object to a standard Linux command like sort:

gsutil cp gs://my_app_bucket/data_measurements - | sort

Streaming uploads and downloads using Python and the boto plugin

To use the boto client library plugin in a Python application to perform a streaming upload, use the following code:

dst_uri = boto.storage_uri(<bucket> + '/' + <object>, 'gs')
dst_uri.new_key().set_contents_from_stream(<stream object>)

For example, the following code performs a streaming upload of a file named data_file to an object with the same name:

filename = 'data_file'
MY_BUCKET = 'my_app_bucket'
my_stream = open(filename, 'rb')
dst_uri = boto.storage_uri(MY_BUCKET + '/' + filename, 'gs')
dst_uri.new_key().set_contents_from_stream(my_stream)

To perform a streaming download using boto, use the following code:

import sys

src_uri = boto.storage_uri(<bucket> + '/' + <object>, 'gs')
src_uri.get_key().get_file(sys.stdout)

For example, the following code performs a streaming download of an object named data_file:

downloaded_file = 'saved_data_file'
MY_BUCKET = 'my_app_bucket'
object_name = 'data_file'
src_uri = boto.storage_uri(MY_BUCKET + '/' + object_name, 'gs')
src_uri.get_key().get_file(sys.stdout)
Was this page helpful? Let us know how we did:

Send feedback about...

Cloud Storage Documentation