Cloud Storage supports streaming data from a bucket to a process without requiring that the data first be saved to a file.
Using checksum validation when streaming
You shouldn't use a streaming download if you require checksum validation
prior to the data becoming accessible. This is because streaming downloads use
the Range
header, and Cloud Storage does not return checksums in the
response that apply to only the requested portion of object data.
It's recommended that you always use checksum validation, and you can do so after a streaming download completes; however, validating after the download completes means that any corrupted data is accessible during the time it takes to confirm the corruption and remove it.
Required roles
In order to get the required permissions for streaming downloads, ask your
administrator to grant you the Storage Object Viewer
(roles/storage.objectViewer
) role on the bucket.
This role contains the permission required to stream downloads. To see the exact permission that's required, expand the Required permissions section:
Required permissions
storage.objects.get
You might also be able to get this permission with other predefined roles or custom roles.
For instructions on granting roles on buckets, see Use IAM with buckets.
Stream a download
The following examples show how to perform a download from a Cloud Storage object to a process:
Console
The Google Cloud console does not support streaming downloads. Use the Google Cloud CLI instead.
Command line
Run the
gcloud storage cp
command using a dash for the destination URL, then pipe the data to the process:gcloud storage cp gs://BUCKET_NAME/OBJECT_NAME - | PROCESS_NAME
Where:
BUCKET_NAME
is the name of the bucket containing the object. For example,my_app_bucket
.OBJECT_NAME
is the name of the object that you are streaming to the process. For example,data_measurements
.PROCESS_NAME
is the name of the process into which you are feeding data. For example,analyze_data
.
You can also stream data from a Cloud Storage object to a standard Linux command like sort:
gcloud storage cp gs://my_app_bucket/data_measurements - | sort
Client libraries
For more information, see the
Cloud Storage C++ API
reference documentation.
To authenticate to Cloud Storage, set up Application Default Credentials.
For more information, see
Set up authentication for a local development environment.
For more information, see the
Cloud Storage C# API
reference documentation.
To authenticate to Cloud Storage, set up Application Default Credentials.
For more information, see
Set up authentication for a local development environment.
For more information, see the
Cloud Storage Go API
reference documentation.
To authenticate to Cloud Storage, set up Application Default Credentials.
For more information, see
Set up authentication for a local development environment.
For more information, see the
Cloud Storage Java API
reference documentation.
To authenticate to Cloud Storage, set up Application Default Credentials.
For more information, see
Set up authentication for a local development environment.
For more information, see the
Cloud Storage Node.js API
reference documentation.
To authenticate to Cloud Storage, set up Application Default Credentials.
For more information, see
Set up authentication for a local development environment.
For more information, see the
Cloud Storage PHP API
reference documentation.
To authenticate to Cloud Storage, set up Application Default Credentials.
For more information, see
Set up authentication for a local development environment.
For more information, see the
Cloud Storage Python API
reference documentation.
To authenticate to Cloud Storage, set up Application Default Credentials.
For more information, see
Set up authentication for a local development environment.
For more information, see the
Cloud Storage Ruby API
reference documentation.
To authenticate to Cloud Storage, set up Application Default Credentials.
For more information, see
Set up authentication for a local development environment.
C++
C#
Go
Java
Node.js
PHP
Python
Ruby
REST APIs
JSON API
To perform a streaming download, follow the instructions for downloading an object with the following considerations:
Before beginning the download, retrieve the object's metadata and save the object's generation number. Include this generation number in each of your requests to ensure that you don't download data from two different generations in the event the original gets overwritten.
Use the
Range
header in your request to retrieve a piece of the overall object, which you can send to a local process.Continue making requests for successive pieces of the object, until the entire object has been retrieved.
XML API
To perform a streaming download, follow the instructions for downloading an object with the following considerations:
Before beginning the download, retrieve the object's metadata and save the object's generation number. Include this generation number in each of your requests to ensure that you don't download data from two different generations in the event the original gets overwritten.
Use the
Range
header in your request to retrieve a piece of the overall object, which you can send to a local process.Continue making requests for successive pieces of the object, until the entire object has been retrieved.
What's next
- Stream an upload.
- Learn more about Decompressive transcoding.
- Learn more about Uploads and downloads.