Streaming transfers

Cloud Storage supports streaming transfers, which allow you to stream data to and from your Cloud Storage account without requiring that the data first be saved to a file. Streaming transfers are useful when you want to upload data generated from a process directly into Cloud Storage, or when you want to download data from Cloud Storage into a process.

You can perform streaming transfers by using the gsutil command-line tool.

Caution: When performing a streaming transfer to or from Cloud Storage, neither Cloud Storage nor gsutil compute a checksum. If you require data validation, use a non-streaming transfer, which performs integrity checking automatically.

Streaming uploads and downloads using gsutil

To use gsutil to perform a streaming upload from a process to a Cloud Storage object, pipe the data to the gsutil cp command and use a dash for the source URL.

The following example streams the data produced by the collect_measurements process into the data_measurements Cloud Storage object:

collect_measurements | gsutil cp - gs://my_app_bucket/data_measurements

To use gsutil to perform a streaming download from a Cloud Storage object to a process, run the gsutil cp command and use a dash for the destination URL, then pipe the data to the process.

The following example streams the data from the data_measurements Cloud Storage object to the analyze_data process:

gsutil cp gs://my_app_bucket/data_measurements - | analyze_data

You can also stream data from a Cloud Storage object to a standard Linux command like sort:

gsutil cp gs://my_app_bucket/data_measurements - | sort