How Pub/Sub eliminates boring meetings and makes your systems scale

So, you've said goodbye to the monolithic application and refactored it into services. You are shipping faster and the code is cleaner. But the complexity of communication between the services is causing performance issues. And every time a new team needs to integrate, it's a whole lot of meetings. The time has come to consider asynchronous communication, where services publish and react to events reliably delivered by a messaging system.

In this model, a service needs only to do its computation on an event and publish it to Pub/Sub. It does not need to accommodate the different latency and availability characteristics of the downstream services. In fact, it needs to know nothing about them. Because of this, meetings about integrations can disappear and scaling becomes easier. Each service and service team can do their part on their time, whether they are doing analytics or operational data integration or building user-facing applications that must make instant decisions. 

Of course, to make this work, you have to trust that Pub/Sub—the messaging service—is available, reliable, and predictable. This article introduces you to Pub/Sub and how it works. 

Pub/Sub comes in two flavors:

Pub/Sub: 

• The default choice for most applications, this is the  “everything and the kitchen sink” option.

• It is serverless, auto-scales & auto-provisions as needed. 

• It provides the ability to publish and subscribe to events regardless of geography.

• You can create up to 10K subscriber apps per topic; independent capacity & billing for publishers and subscribers; 

• Supports push delivery model specially useful for communication between microservices 

Pub/Sub Lite: 

• Optimizes for cost over reliability. 

• It is up to 90% cheaper than Pub/Sub

• Requires you to manage your capacity and offers lower availability level

How does Pub/Sub work?

A publisher application creates and sends messages to a topic. Subscriber applications create a subscription to a topic to receive messages from it. Each message is guaranteed to be delivered at least once per subscription, as long as subscribers are sufficiently active. Communication can be one-to-many (fan-out), many-to-one (fan-in), and many-to-many.

1. A publisher application creates a topic in the Pub/Sub service and sends messages to the topic. A message contains a payload and optional attributes that describe the payload content.

2. The service ensures that published messages are retained on behalf of subscriptions. A published message is retained for a subscription until it is acknowledged by any subscriber consuming messages from that subscription.

3. Pub/Sub forwards messages from a topic to all of its subscriptions, individually.

4. A subscriber receives messages either by Pub/Sub pushing them to the subscriber's chosen endpoint, or by the subscriber pulling them from the service.

5. The subscriber sends an acknowledgement to the Pub/Sub service for each received message.

6. The service removes acknowledged messages from the subscription's message queue.

Pub/Sub features

• Global routing:  you can publish messages to a topic anywhere in the world. It is persisted in the nearest region for low latency. On the other end, subscribers, deployed in any region, receive messages from all publish locations without doing anything special. 

• Partition-less in-order delivery: If messages have the same ordering key and are in the same region, you can enable message ordering and receive the messages in the order that the Pub/Sub service receives them.

• Dead letter topics: If the Pub/Sub service attempts to deliver a message but the subscriber can't acknowledge it, Pub/Sub can forward the undeliverable message to a dead-letter topic to attempt delivery at a later time. 

• Seek and replay: There could be times when you need to alter the acknowledgement stage of messages in bulk, seek and replay feature allows you to do that. 

• Filtering: You can filter messages by message attributes and when you receive messages from a subscription with a filter, you only receive the messages that match the filter. Pub/Sub automatically acknowledges the messages that don't match the filter

Pub/Sub use cases

Stream analytics: Data ingestion is the foundation for analytics and machine learning, whether you are building stream, batch, or unified pipelines. Cloud Pub/Sub provides a simple and reliable staging location for your event data on its journey towards processing, storage, and analysis. Use Cloud Dataflow with Cloud Pub/Sub to enrich, deduplicate, order, aggregate, and land events. Mix real-time and batch processing via Cloud Pub/Sub’s durable storage.

Asynchronous service integration: Pub/Sub works as a messaging middleware for traditional service integration or a simple communication medium for modern microservices. Push subscriptions deliver events to serverless webhooks on Cloud Functions, App Engine, Cloud Run, or custom environments on Google Kubernetes Engine or Compute Engine. Low-latency pull delivery is available when exposing webhooks is not an option or for efficient handling of higher throughput streams.

For a more in-depth look into Pub/Sub check out the documentation or checkout my Pub/Sub Made Easy video series.

For more #GCPSketchnote, follow the GitHub repo. For similar cloud content follow me on Twitter @pvergadia and keep an eye out on thecloudgirl.dev.