Cloud Storage supports streaming transfers, which allow you to stream data to and from your Cloud Storage account without requiring that the data first be saved to a file.
Streaming uploads are useful when you want to upload data whose final size you don't know at the start of the upload, such as when generating the upload data from a process, or when compressing an object on-the-fly.
Streaming downloads are useful when you want to download data from Cloud Storage into a process.
Streaming uploads
The following examples show how to perform a streaming upload from a process to a Cloud Storage object:
gsutil
Pipe the data to the
gsutil cp
command and use a dash for the source URL:PROCESS_NAME | gsutil cp - gs://BUCKET_NAME/OBJECT_NAME
Where:
PROCESS_NAME
is the name of the process from which you are collecting data. For example,collect_measurements
.BUCKET_NAME
is the name of the bucket containing the object. For example,my_app_bucket
.OBJECT_NAME
is the name of the object that is created from the data. For example,data_measurements
.
Client libraries
For more information, see the
Cloud Storage C++ API reference documentation.
To stream object uploads using C++,
see the WriteObject
reference documentation.
For more information, see the
Cloud Storage C# API reference documentation.
To stream object uploads using C#,
see the SimpleUpload
reference documentation.
For more information, see the
Cloud Storage Go API reference documentation.
To stream object uploads using Go,
see the NewWriter reference documentation. Note that Go uses the standard
For more information, see the
Cloud Storage Java API reference documentation.
To stream object uploads using Java,
see the writer reference documentation.
For more information, see the
Cloud Storage Node.js API reference documentation.
To stream object uploads using Node.js,
see the createWriteStream reference documentation.
For more information, see the
Cloud Storage PHP API reference documentation.
To stream object uploads using PHP,
see the upload reference documentation.
For more information, see the
Cloud Storage Python API reference documentation.
You cannot currently perform streaming uploads with the Python client library.
For more information, see the
Cloud Storage Ruby API reference documentation.
You cannot currently perform streaming uploads with the Ruby client library.C++
C#
Go
io.Writer
and io.Reader
interfaces for uploads and downloads respectively, both of which allow streaming.Java
Node.js
PHP
Python
Ruby
REST APIs
JSON API
To perform a streaming upload, follow the instructions for performing a resumable upload with the following considerations:
When uploading the file data itself, use a multiple chunk upload.
Since you don't know the total file size until you get to the final chunk, use a
*
for the total file size in theContent-Range
header of intermediate chunks.For example, if the first chunk you upload has a size of 512 KiB, the
Content-Range
header for the chunk isbytes 0-524287/*
. If your upload has 64000 bytes remaining after the first chunk, you then send a final chunk that contains the remaining bytes and has aContent-Range
header with the valuebytes 524288-588287/588288
.
XML API
To perform a streaming upload, follow the instructions for performing a resumable upload with the following considerations:
When uploading the file data itself, use a multiple chunk upload.
Since you don't know the total file size until you get to the final chunk, use a
*
for the total file size in theContent-Range
header of intermediate chunks.For example, if the first chunk you upload has a size of 512 KiB, the
Content-Range
header for the chunk isbytes 0-524287/*
. If your upload has 64000 bytes remaining after the first chunk, you then send a final chunk that contains the remaining bytes and has aContent-Range
header with the valuebytes 524288-588287/588288
.
Streaming downloads
The following examples show how to perform a download from a Cloud Storage object to a process:
gsutil
Run the
gsutil cp
command using a dash for the destination URL, then pipe the data to the process:gsutil cp gs://BUCKET_NAME/OBJECT_NAME - | PROCESS_NAME
Where:
BUCKET_NAME
is the name of the bucket containing the object. For example,my_app_bucket
.OBJECT_NAME
is the name of the object that you are streaming to the process. For example,data_measurements
.PROCESS_NAME
is the name of the process into which you are feeding data. For example,analyze_data
.
You can also stream data from a Cloud Storage object to a standard Linux command like sort:
gsutil cp gs://my_app_bucket/data_measurements - | sort
Client libraries
For more information, see the
Cloud Storage C++ API reference documentation.
To stream object downloads using C++,
see the ReadObject
reference documentation.
For more information, see the
Cloud Storage C# API reference documentation.
To stream object downloads using C#,
see the SimpleDownload
reference documentation.
For more information, see the
Cloud Storage Go API reference documentation.
To stream object downloads using Go,
see the NewReader reference documentation. Note that Go uses the standard
For more information, see the
Cloud Storage Java API reference documentation.
To stream object downloads using Java,
see the reader reference documentation.
For more information, see the
Cloud Storage Node.js API reference documentation.
To stream object downloads using Node.js,
see the createReadStream reference documentation.
For more information, see the
Cloud Storage PHP API reference documentation.
To stream object downloads using PHP,
see the downloadAsStreams reference documentation.
For more information, see the
Cloud Storage Python API reference documentation.
You cannot currently perform streaming downloads with the Python client library.
For more information, see the
Cloud Storage Ruby API reference documentation.
You cannot currently perform streaming downloads with the Ruby client library.C++
C#
Go
io.Writer
and io.Reader
interfaces for uploads and downloads respectively, both of which allow streaming.Java
Node.js
PHP
Python
Ruby