Streaming transfers

Cloud Storage supports streaming transfers, which allow you to stream data to and from your Cloud Storage account without requiring that the data first be saved to a file.

Caution: You cannot specify a checksum for the upload when performing streaming transfers to or from Cloud Storage. It's recommended that you always use checksums for data validation, so you should either manually perform data validation after the transfer completes, or you should use a non-streaming transfer and specify a checksum.

Streaming uploads are useful when you want to upload data whose final size you don't know at the start of the upload, such as when generating the upload data from a process, or when compressing an object on-the-fly.

Streaming downloads are useful when you want to download data from Cloud Storage into a process.

Note: Streaming transfers are not supported for the Console, Python, or Ruby.

Streaming uploads

The following examples show how to perform a streaming upload from a process to a Cloud Storage object:

gsutil

1. Pipe the data to the gsutil cp command and use a dash for the source URL:

   PROCESS_NAME | gsutil cp - gs://BUCKET_NAME/OBJECT_NAME

   Where:

   • PROCESS_NAME is the name of the process from which you are collecting data. For example, collect_measurements.
   • BUCKET_NAME is the name of the bucket containing the object. For example, my_app_bucket.
   • OBJECT_NAME is the name of the object that is created from the data. For example, data_measurements.

Client libraries

C++

For more information, see the Cloud Storage C++ API reference documentation.

To stream object uploads using C++, see the WriteObject reference documentation.

C#

For more information, see the Cloud Storage C# API reference documentation.

To stream object uploads using C#, see the SimpleUpload reference documentation.

Go

For more information, see the Cloud Storage Go API reference documentation.

To stream object uploads using Go, see the NewWriter reference documentation.

Note that Go uses the standard io.Writer and io.Reader interfaces for uploads and downloads respectively, both of which allow streaming.

Java

For more information, see the Cloud Storage Java API reference documentation.

To stream object uploads using Java, see the writer reference documentation.

Node.js

For more information, see the Cloud Storage Node.js API reference documentation.

To stream object uploads using Node.js, see the createWriteStream reference documentation.

PHP

For more information, see the Cloud Storage PHP API reference documentation.

To stream object uploads using PHP, see the upload reference documentation.

Python

For more information, see the Cloud Storage Python API reference documentation.

You cannot currently perform streaming uploads with the Python client library.

Ruby

For more information, see the Cloud Storage Ruby API reference documentation.

You cannot currently perform streaming uploads with the Ruby client library.

REST APIs

JSON API

To perform a streaming upload, follow the instructions for performing a resumable upload with the following considerations:

• When uploading the file data itself, use a multiple chunk upload.

• Since you don't know the total file size until you get to the final chunk, use a * for the total file size in the Content-Range header of intermediate chunks.

  For example, if the first chunk you upload has a size of 512 KiB, the Content-Range header for the chunk is bytes 0-524287/*. If your upload has 64000 bytes remaining after the first chunk, you then send a final chunk that contains the remaining bytes and has a Content-Range header with the value bytes 524288-588287/588288.

XML API

To perform a streaming upload, follow the instructions for performing a resumable upload with the following considerations:

• When uploading the file data itself, use a multiple chunk upload.

• Since you don't know the total file size until you get to the final chunk, use a * for the total file size in the Content-Range header of intermediate chunks.

  For example, if the first chunk you upload has a size of 512 KiB, the Content-Range header for the chunk is bytes 0-524287/*. If your upload has 64000 bytes remaining after the first chunk, you then send a final chunk that contains the remaining bytes and has a Content-Range header with the value bytes 524288-588287/588288.

Streaming downloads

The following examples show how to perform a download from a Cloud Storage object to a process:

gsutil

1. Run the gsutil cp command using a dash for the destination URL, then pipe the data to the process:

   gsutil cp gs://BUCKET_NAME/OBJECT_NAME - | PROCESS_NAME

   Where:

   • BUCKET_NAME is the name of the bucket containing the object. For example, my_app_bucket.
   • OBJECT_NAME is the name of the object that you are streaming to the process. For example, data_measurements.
   • PROCESS_NAME is the name of the process into which you are feeding data. For example, analyze_data.

You can also stream data from a Cloud Storage object to a standard Linux command like sort:

gsutil cp gs://my_app_bucket/data_measurements - | sort

Client libraries

C++

For more information, see the Cloud Storage C++ API reference documentation.

To stream object downloads using C++, see the ReadObject reference documentation.

C#

For more information, see the Cloud Storage C# API reference documentation.

To stream object downloads using C#, see the SimpleDownload reference documentation.

Go

For more information, see the Cloud Storage Go API reference documentation.

To stream object downloads using Go, see the NewReader reference documentation.

Note that Go uses the standard io.Writer and io.Reader interfaces for uploads and downloads respectively, both of which allow streaming.

Java

For more information, see the Cloud Storage Java API reference documentation.

To stream object downloads using Java, see the reader reference documentation.

Node.js

For more information, see the Cloud Storage Node.js API reference documentation.

To stream object downloads using Node.js, see the createReadStream reference documentation.

PHP

For more information, see the Cloud Storage PHP API reference documentation.

To stream object downloads using PHP, see the downloadAsStreams reference documentation.

Python

For more information, see the Cloud Storage Python API reference documentation.

You cannot currently perform streaming downloads with the Python client library.

Ruby

For more information, see the Cloud Storage Ruby API reference documentation.

You cannot currently perform streaming downloads with the Ruby client library.