Google Cloud Pub/Sub brings the scalability, flexibility, and reliability of enterprise message-oriented middleware to the cloud. By providing many-to-many, asynchronous messaging that decouples senders and receivers, it allows for secure and highly available communication between independently written applications. Google Cloud Pub/Sub delivers low-latency, durable messaging that helps developers quickly integrate systems hosted on the Google Cloud Platform and externally.
A publisher application creates and sends messages to a topic. Subscriber applications create a subscription to a topic to receive messages from it. Communication can be one-to-many (fan-out), many-to-one (fan-in), and many-to-many.
Here are some classic use cases for Google Cloud Pub/Sub:
- Balancing workloads in network clusters. For example, a large queue of tasks can be efficiently distributed among multiple workers, such as Google Compute Engine instances.
- Implementing asynchronous workflows. For example, an order processing application can place an order on a topic, from which it can be processed by one or more workers.
- Distributing event notifications. For example, a service that accepts user signups can send notifications whenever a new user registers, and downstream services can subscribe to receive notifications of the event.
- Refreshing distributed caches. For example, an application can publish invalidation events to update the IDs of objects that have changed.
- Logging to multiple systems. For example, a Google Compute Engine instance can write logs to the monitoring system, to a database for later querying, and so on.
- Data streaming from various processes or devices. For example, a residential sensor can stream data to backend servers hosted in the cloud.
- Reliability improvement. For example, a single-zone Compute Engine service can operate in additional zones by subscribing to a common topic, to recover from failures in a zone or region.
Benefits and features
- Unified messaging: Durability and low-latency delivery in a single product
- Global presence: Connect services located anywhere in the world
- Flexible delivery options: Both push- and pull-style subscriptions supported
- Data reliability: Replicated storage and guaranteed at-least-once message delivery
- End-to-end reliability: Explicit application-level acknowledgement
- Data security and protection: Encryption of data on the wire and at rest
- Flow control: Dynamic rate limiting implemented by the Pub/Sub system
- Simplicity: Easy-to-use REST/JSON API
Pub/Sub concepts and message flow
Here is an overview of the components in the Pub/Sub system and how messages flow between them:
- A publisher application creates a topic in the Google Cloud Pub/Sub service and sends messages to the topic. A message contains a payload and optional attributes that describe the payload content.
- Messages are persisted in a message store until they are delivered and acknowledged by subscribers.
- The Pub/Sub service forwards messages from a topic to all of its subscriptions, individually. Each subscription receives messages either by Pub/Sub pushing them to the subscriber's chosen endpoint, or by the subscriber pulling them from the service.
- The subscriber receives pending messages from its subscription and acknowledges each one to the Pub/Sub service.
- When a message is acknowledged by the subscriber, it is removed from the subscription's queue of messages.
Publisher and subscriber endpoints
Publishers can be any application that can make HTTPS requests to googleapis.com: an App Engine app, a web service hosted on Google Compute Engine or any other third-party network, an installed app for desktop or mobile device, or even a browser.
Pull subscribers can also be any application that can make HTTPS requests to googleapis.com.
Currently, push subscribers must be Webhook endpoints that can accept POST requests over HTTPS.
Resource types and models
The following are the types of objects used by the Pub/Sub API. Topics and Subscriptions are the only resource types exposed as REST collections.
- A named resource to which messages are sent by publishers.
- A named resource representing the stream of messages from a single, specific topic, to be delivered to the subscribing application. For more details about subscriptions and message delivery semantics, see the Subscriber Guide.
- The combination of data and (optional) attributes that a publisher sends to a topic and is eventually delivered to subscribers.
- Message attribute
- A key-value pair that a publisher can define for a message. For example, key
encould be added to messages to mark them as readable by an English-speaking subscriber.
Pub/Sub topic and subscription names must be scoped to a project:
- The collection must be one of
- The project-identifier must be the project ID, available from the Google Cloud Platform Console. For example,
- The resource-name must start with a letter, and contain only letters (
[A-Za-z]), numbers (
[0-9]), dashes (
-), underscores (
_), periods (
.), tildes (
~), plus (
+) or percent signs (
%). It must be between 3 and 255 characters in length, and cannot begin with the string
This format lets you use URL encoding to include special characters in the resource name. For example, a topic named
mi-tópico, which is an invalid resource name for Cloud Pub/Sub, could be encoded as
mi-t%C3%B3pico, which is valid.
Note that the client libraries can take different approaches to naming resources, such as specifying the
project and the
resource-name in separate fields.