Cloud Storage supports streaming transfers, which allow you to stream data to and from your Cloud Storage account without requiring that the data first be saved to a file. Streaming is useful when:
You want to upload data but don't know the final size at the start of the upload, such as when generating the upload data from a process, or when compressing an object on-the-fly.
You want to download data from Cloud Storage into a process.
Using checksum validation when streaming
Because a checksum can only be supplied in the initial request of an upload, it's often not feasible to utilize Cloud Storage's checksum validation when streaming. It's recommended that you always use checksum validation, and you can manually do so after a streaming upload completes; however, validating after the transfer completes means that any corrupted data is accessible during the time it takes to confirm the corruption and remove it.
If you require checksum validation prior to the upload completing and the data becoming accessible, then you should not use a streaming upload. You should use a different upload option that performs checksum validation prior to finalizing the object.
Similarly, you should not use a streaming download if you require checksum
validation prior to the download completing and the data becoming accessible.
This is because streaming downloads use the Range
header, and
Cloud Storage does not perform checksum validation on such requests.
Prerequisites
Prerequisites can vary based on the tool used:
Console
In order to complete this guide using the Google Cloud console, you must have the proper IAM permissions. If the bucket you want to access for streaming exists in a project that you did not create, you might need the project owner to give you a role that contains the necessary permissions.
For a list of permissions required for specific actions, see IAM permissions for the Google Cloud console.
For a list of relevant roles, see Cloud Storage roles. Alternatively, you can create a custom role that has specific, limited permissions.
Command line
In order to complete this guide using a command-line utility, you must have the proper IAM permissions. If the bucket you want to access for streaming exists in a project that you did not create, you might need the project owner to give you a role that contains the necessary permissions.
For a list of permissions required for specific actions, see IAM permissions for gsutil commands.
For a list of relevant roles, see Cloud Storage roles. Alternatively, you can create a custom role that has specific, limited permissions.
Code samples
In order to complete this guide using the Cloud Storage client libraries, you must have the proper IAM permissions. If the bucket you want to access for streaming exists in a project that you did not create, you might need the project owner to give you a role that contains the necessary permissions. Unless otherwise noted, client library requests are made through the JSON API.
For a list of permissions required for specific actions, see IAM permissions for JSON methods.
For a list of relevant roles, see Cloud Storage roles. Alternatively, you can create a custom role that has specific, limited permissions.
REST APIs
JSON API
In order to complete this guide using the JSON API, you must have the proper IAM permissions. If the bucket you want to access for streaming exists in a project that you did not create, you might need the project owner to give you a role that contains the necessary permissions.
For a list of permissions required for specific actions, see IAM permissions for JSON methods.
For a list of relevant roles, see Cloud Storage roles. Alternatively, you can create a custom role that has specific, limited permissions.
Stream an upload
The following examples show how to perform a streaming upload from a process to a Cloud Storage object:
Console
The Google Cloud console does not support streaming uploads. Use gsutil instead.
Command line
Pipe the data to the
gsutil cp
command and use a dash for the source URL:PROCESS_NAME | gsutil cp - gs://BUCKET_NAME/OBJECT_NAME
Where:
PROCESS_NAME
is the name of the process from which you are collecting data. For example,collect_measurements
.BUCKET_NAME
is the name of the bucket containing the object. For example,my_app_bucket
.OBJECT_NAME
is the name of the object that is created from the data. For example,data_measurements
.
Code samples
For more information, see the
Cloud Storage C++ API reference documentation.
For more information, see the
Cloud Storage C# API reference documentation.
For more information, see the
Cloud Storage Go API reference documentation.
For more information, see the
Cloud Storage Java API reference documentation.
For more information, see the
Cloud Storage Node.js API reference documentation.
For more information, see the
Cloud Storage PHP API reference documentation.
For more information, see the
Cloud Storage Python API reference documentation.
For more information, see the
Cloud Storage Ruby API reference documentation.
C++
C#
Go
Java
Node.js
PHP
Python
Ruby
REST APIs
JSON API
To perform a streaming upload, follow the instructions for performing a resumable upload with the following considerations:
When uploading the file data itself, use a multiple chunk upload.
Since you don't know the total file size until you get to the final chunk, use a
*
for the total file size in theContent-Range
header of intermediate chunks.For example, if the first chunk you upload has a size of 512 KiB, the
Content-Range
header for the chunk isbytes 0-524287/*
. If your upload has 64000 bytes remaining after the first chunk, you then send a final chunk that contains the remaining bytes and has aContent-Range
header with the valuebytes 524288-588287/588288
.
XML API
To perform a streaming upload, use one of the following methods:
A resumable upload, with the following adjustments:
When uploading the file data itself, use a multiple chunk upload.
Since you don't know the total file size until you get to the final chunk, use a
*
for the total file size in theContent-Range
header of intermediate chunks.For example, if the first chunk you upload has a size of 512 KiB, the
Content-Range
header for the chunk isbytes 0-524287/*
. If your upload has 64000 bytes remaining after the first chunk, you then send a final chunk that contains the remaining bytes and has aContent-Range
header with the valuebytes 524288-588287/588288
.
Stream a download
The following examples show how to perform a download from a Cloud Storage object to a process:
Console
The Google Cloud console does not support streaming downloads. Use gsutil instead.
Command line
Run the
gsutil cp
command using a dash for the destination URL, then pipe the data to the process:gsutil cp gs://BUCKET_NAME/OBJECT_NAME - | PROCESS_NAME
Where:
BUCKET_NAME
is the name of the bucket containing the object. For example,my_app_bucket
.OBJECT_NAME
is the name of the object that you are streaming to the process. For example,data_measurements
.PROCESS_NAME
is the name of the process into which you are feeding data. For example,analyze_data
.
You can also stream data from a Cloud Storage object to a standard Linux command like sort:
gsutil cp gs://my_app_bucket/data_measurements - | sort
Code samples
For more information, see the
Cloud Storage C++ API reference documentation.
For more information, see the
Cloud Storage C# API reference documentation.
For more information, see the
Cloud Storage Go API reference documentation.
For more information, see the
Cloud Storage Java API reference documentation.
For more information, see the
Cloud Storage Node.js API reference documentation.
For more information, see the
Cloud Storage PHP API reference documentation.
For more information, see the
Cloud Storage Python API reference documentation.
For more information, see the
Cloud Storage Ruby API reference documentation.
C++
C#
Go
Java
Node.js
PHP
Python
Ruby
REST APIs
JSON API
To perform a streaming download, follow the instructions for downloading an object with the following considerations:
Before beginning the download, retrieve the object's metadata and save the object's generation number. Include this generation number in each of your requests to ensure that you don't download data from two different generations in the event the original gets overwritten.
Use the
Range
header in your request to retrieve a piece of the overall object, which you can send to the desired local process.Continue making requests for successive pieces of the object, until the entire object has been retrieved.
XML API
To perform a streaming download, follow the instructions for downloading an object with the following considerations:
Before beginning the download, retrieve the object's metadata and save the object's generation number. Include this generation number in each of your requests to ensure that you don't download data from two different generations in the event the original gets overwritten.
Use the
Range
header in your request to retrieve a piece of the overall object, which you can send to the desired local process.Continue making requests for successive pieces of the object, until the entire object has been retrieved.
What's next
- Learn more about Decompressive transcoding.
- Learn more about Uploads and downloads.