Cloud Storage supports streaming data from a bucket to a process without requiring that the data first be saved to a file.
Using checksum validation when streaming
You shouldn't use a streaming download if you require checksum validation
prior to the data becoming accessible. This is because streaming downloads use
the Range
header, and Cloud Storage does not return checksums in the
response that apply to only the requested portion of object data.
It's recommended that you always use checksum validation, and you can do so after a streaming download completes; however, validating after the download completes means that any corrupted data is accessible during the time it takes to confirm the corruption and remove it.
Required roles
In order to get the required permissions for streaming downloads, ask your
administrator to grant you the Storage Object Viewer
(roles/storage.objectViewer
) role on the bucket.
This role contains the permission required to stream downloads. To see the exact permission that's required, expand the Required permissions section:
Required permissions
storage.objects.get
You might also be able to get this permission with other predefined roles or custom roles.
For instructions on granting roles on buckets, see Use IAM with buckets.
Stream a download
The following examples show how to perform a download from a Cloud Storage object to a process:
The Google Cloud console does not support streaming downloads. Use the Google Cloud CLI instead.
Run the
gcloud storage cp
command using a dash for the destination URL, then pipe the data to the process:gcloud storage cp gs://
BUCKET_NAME /OBJECT_NAME - |PROCESS_NAME Where:
BUCKET_NAME
is the name of the bucket containing the object. For example,my_app_bucket
.OBJECT_NAME
is the name of the object that you are streaming to the process. For example,data_measurements
.PROCESS_NAME
is the name of the process into which you are feeding data. For example,analyze_data
.
You can also stream data from a Cloud Storage object to a standard Linux command like sort:
gcloud storage cp gs://my_app_bucket/data_measurements - | sort
For more information, see the Cloud Storage C++ API reference documentation.
To authenticate to Cloud Storage, set up Application Default Credentials. For more information, see Set up authentication for client libraries.
For more information, see the Cloud Storage C# API reference documentation.
To authenticate to Cloud Storage, set up Application Default Credentials. For more information, see Set up authentication for client libraries.
For more information, see the Cloud Storage Go API reference documentation.
To authenticate to Cloud Storage, set up Application Default Credentials. For more information, see Set up authentication for client libraries.
For more information, see the Cloud Storage Java API reference documentation.
To authenticate to Cloud Storage, set up Application Default Credentials. For more information, see Set up authentication for client libraries.
For more information, see the Cloud Storage Node.js API reference documentation.
To authenticate to Cloud Storage, set up Application Default Credentials. For more information, see Set up authentication for client libraries.
For more information, see the Cloud Storage PHP API reference documentation.
To authenticate to Cloud Storage, set up Application Default Credentials. For more information, see Set up authentication for client libraries.
For more information, see the Cloud Storage Python API reference documentation.
To authenticate to Cloud Storage, set up Application Default Credentials. For more information, see Set up authentication for client libraries.
For more information, see the Cloud Storage Ruby API reference documentation.
To authenticate to Cloud Storage, set up Application Default Credentials. For more information, see Set up authentication for client libraries.
To perform a streaming download, follow the instructions for downloading an object with the following considerations:
Before beginning the download, retrieve the object's metadata and save the object's generation number. Include this generation number in each of your requests to ensure that you don't download data from two different generations in the event the original gets overwritten.
Use the
Range
header in your request to retrieve a piece of the overall object, which you can send to a local process.Continue making requests for successive pieces of the object, until the entire object has been retrieved.
To perform a streaming download, follow the instructions for downloading an object with the following considerations:
Before beginning the download, retrieve the object's metadata and save the object's generation number. Include this generation number in each of your requests to ensure that you don't download data from two different generations in the event the original gets overwritten.
Use the
Range
header in your request to retrieve a piece of the overall object, which you can send to a local process.Continue making requests for successive pieces of the object, until the entire object has been retrieved.
What's next
- Stream an upload.
- Learn more about Decompressive transcoding.
- Learn more about Uploads and downloads.