Cloud Storage subscriptions

A Cloud Storage subscription is a type of export subscription that writes messages to an existing Cloud Storage bucket as they are received. You don't need to configure a separate subscriber client. Use the Google Cloud console, the Google Cloud CLI, the client libraries, or the Pub/Sub API to create, update, list, detach, or delete a Cloud Storage subscription.

Without the Cloud Storage subscription type, you need a pull or push subscription and a subscriber (such as Dataflow) that reads messages and writes them to a Cloud Storage bucket. The overhead of running a Dataflow job is not necessary when messages don't require additional processing before storing them in a Cloud Storage bucket; you can use a Cloud Storage subscription instead.

However, a Dataflow pipeline is still recommended for Pub/Sub systems where some data transformation is required before the data is stored in a Cloud Storage bucket.

To learn how to stream data from Pub/Sub to Cloud Storage with transformation by using Dataflow, see Stream messages from Pub/Sub by using Dataflow.

Before you begin

Before reading this document, ensure that you're familiar with the following:

Cloud Storage subscription workflow

The following image shows the workflow between a Cloud Storage subscription and Cloud Storage.

Flow of messages for a Cloud Storage subscription

Figure 1. Workflow for a Cloud Storage subscription

Here is a brief description of the workflow that references Figure 1:

  1. Pub/Sub uses the Cloud Storage API to send data to the Cloud Storage bucket. After a successful completion of a write operation, the API returns an OK response.

  2. The messages are sent in batches to the Cloud Storage bucket. A single batch corresponds to an object that is stored in the Cloud Storage bucket. You can configure the batch settings based on the object size and time elapsed since the object was created.

  3. The workflow to write an object to a Cloud Storage bucket requires two OK responses. The first OK response is sent when a message is successfully written to an object. The second OK response is sent when the object containing the message is finalized. A message only becomes visible in the bucket after the Cloud Storage object containing it has been successfully finalized.

  4. If there are any failures during the write operation, the Pub/Sub message itself is negatively acknowledged. The message is then re-sent. If the message write operation fails enough times and you have a dead letter topic configured on the subscription, then the message is moved to the dead letter topic.

Properties of a Cloud Storage subscription

When you create a Cloud Storage subscription, you must specify some additional properties. These properties include common subscription properties, filename and format requirements, storage requirements and more. To learn more about these properties, see Cloud Storage subscription properties.

Pub/Sub service account permissions

To create a Cloud Storage subscription, the Pub/Sub service account must have permission to write to the specific Cloud Storage bucket and to read the bucket metadata.

For more information, see Assign Cloud Storage roles to the Pub/Sub service account.

Handle message failures

When a Pub/Sub message cannot be written to a Cloud Storage bucket, the message cannot be acknowledged. To forward such undeliverable messages, configure a dead-letter topic on the Cloud Storage subscription.

If Pub/Sub cannot write messages to Cloud Storage, then Pub/Sub backs off delivery of messages in a way similar to push backoff behavior.

Quotas

Quota limitations apply to the Cloud Storage subscriber throughput for each region. For information about the quotas for Cloud Storage subscriptions, see export subscriber quota limits in Pub/Sub quotas and limits.

Pricing

For information about the pricing for Cloud Storage subscriptions, see the Pub/Sub pricing page.

What's next