Streaming transfers

Cloud Storage supports streaming transfers, which allow you to stream data to and from your Cloud Storage account without requiring that the data first be saved to a file. Streaming transfers are useful when you want to upload data generated from a process directly into Cloud Storage, or when you want to download data from Cloud Storage into a process.

Streaming uploads

The following examples show how to perform a streaming upload from a process to a Cloud Storage object:

gsutil

  1. Pipe the data to the gsutil cp command and use a dash for the source URL:

    PROCESS_NAME | gsutil cp - gs://BUCKET_NAME/OBJECT_NAME

    Where:

    • PROCESS_NAME is the name of the process from which you are collecting data. For example, collect_measurements.
    • BUCKET_NAME is the name of the bucket containing the object. For example, my_app_bucket.
    • OBJECT_NAME is the name of the object that is created from the data. For example, data_measurements.

Client libraries

C++

For more information, see the Cloud Storage C++ API reference documentation.

To stream object uploads using C++, see the WriteObject reference documentation.

C#

For more information, see the Cloud Storage C# API reference documentation.

To stream object uploads using C#, see the SimpleUpload reference documentation.

Go

For more information, see the Cloud Storage Go API reference documentation.

To stream object uploads using Go, see the NewWriter reference documentation.

Note that Go uses the standard io.Writer and io.Reader interfaces for uploads and downloads respectively, both of which allow streaming.

Java

For more information, see the Cloud Storage Java API reference documentation.

To stream object uploads using Java, see the writer reference documentation.

Node.js

For more information, see the Cloud Storage Node.js API reference documentation.

To stream object uploads using Node.js, see the createWriteStream reference documentation.

PHP

For more information, see the Cloud Storage PHP API reference documentation.

To stream object uploads using PHP, see the upload reference documentation.

Python

For more information, see the Cloud Storage Python API reference documentation.

You cannot currently perform streaming uploads with the Python client library.

Ruby

For more information, see the Cloud Storage Ruby API reference documentation.

You cannot currently perform streaming uploads with the Ruby client library.

Streaming downloads

The following examples show how to perform a download from a Cloud Storage object to a process:

gsutil

  1. Run the gsutil cp command using a dash for the destination URL, then pipe the data to the process:

    gsutil cp gs://BUCKET_NAME/OBJECT_NAME - | PROCESS_NAME

    Where:

    • BUCKET_NAME is the name of the bucket containing the object. For example, my_app_bucket.
    • OBJECT_NAME is the name of the object that you are streaming to the process. For example, data_measurements.
    • PROCESS_NAME is the name of the process into which you are feeding data. For example, analyze_data.

You can also stream data from a Cloud Storage object to a standard Linux command like sort:

gsutil cp gs://my_app_bucket/data_measurements - | sort

Client libraries

C++

For more information, see the Cloud Storage C++ API reference documentation.

To stream object downloads using C++, see the ReadObject reference documentation.

C#

For more information, see the Cloud Storage C# API reference documentation.

To stream object downloads using C#, see the SimpleDownload reference documentation.

Go

For more information, see the Cloud Storage Go API reference documentation.

To stream object downloads using Go, see the NewReader reference documentation.

Note that Go uses the standard io.Writer and io.Reader interfaces for uploads and downloads respectively, both of which allow streaming.

Java

For more information, see the Cloud Storage Java API reference documentation.

To stream object downloads using Java, see the reader reference documentation.

Node.js

For more information, see the Cloud Storage Node.js API reference documentation.

To stream object downloads using Node.js, see the createReadStream reference documentation.

PHP

For more information, see the Cloud Storage PHP API reference documentation.

To stream object downloads using PHP, see the downloadAsStreams reference documentation.

Python

For more information, see the Cloud Storage Python API reference documentation.

You cannot currently perform streaming downloads with the Python client library.

Ruby

For more information, see the Cloud Storage Ruby API reference documentation.

You cannot currently perform streaming downloads with the Ruby client library.