Pub/Sub makes scalable real-time analytics more accessible than ever

These days, real-time analytics has become critical for business. Automated, real-time decisions based on up-to-the-second data are no longer just for advanced, tech-first companies. It is becoming a basic way of doing business. According to IDC, more than a quarter of data created will be real-time in the next five years. The factors we see driving this growth are the competitive pressure to improve service and user experience quality. Another factor is the consumerization of many traditional businesses where many functions that used to be performed by agents are now done by consumers themselves. Now, every bank, retailer, and service provider needs to have a number of user interfaces, from internal apps, to mobile apps, and web apps. These interfaces not only require fresh data to operate but also produce transaction and interaction data at unprecedented scale. 

Real-time data is not just about application features. It is fundamentally about scaling operations to deliver great user experiences: up-to-date systems monitoring, alerts, customer service dashboards, and automated controls for anything from industrial machinery to customer service operations to consumer devices. It can accelerate data insights to action and in turn increase operational responsiveness.

“With Google Cloud, we’ve been able to build a truly real-time engagement platform,” says Levente Otti, Head of Data, Emarsys. “The norm used to be daily batch processing of data. Now, if an event happens, marketing actions can be executed within seconds, and customers can react immediately. That makes us very competitive in our market.” 

Real-time analytics all starts with messaging 

At Google, we’ve contended with the challenge of creating real-time user experiences at a vast scale from the early days of the company. A key component of our solution for this is Pub/Sub, a global, horizontally scalable messaging system. For over a decade, Google products, including Ads, Search and Gmail, have been using this infrastructure to handle hundreds of millions of events per second. 

Several years ago, we made this system available to the world as Cloud Pub/Sub. Pub/Sub is uniquely easy to use. Traditional messaging middleware offered many of the same features, but were not designed to scale horizontally or were offered as services. Apache Kafka, the open-source stream processing platform, has solved the scalability problem by creating a distributed, partitioned log that supported horizontally scalable streaming writes and reads. Managed services inspired by the same idea have sprung up. Because these services are generally based on the notion of a fixed, local resource, such as a partition or a cluster, these services still left the users to solve the problem of global distribution of data and managing capacity.

Pub/Sub took automated capacity management to an extreme: Data producers need not worry about the capacity required to deliver data to subscribers, with up to 10,000 subscriptions per topic supported. In fact, consumers even pay for the capacity needed to read the data independently from the data producers. The global nature of Pub/Sub is unique, with a single endpoint resolving to nearby regions for fast persistence of data. On the other side, the subscribers can be anywhere and receive a single stream of data aggregated from across all regions. At the same time, users retain precise control over where the data is stored and how it makes it there. This makes Pub/Sub a convenient way to make data available to a broad range of applications on Google Cloud and elsewhere, from ingestion into BigQuery to automated, real-time AI-assisted decision making with Dataflow. This provides data practitioners with the choice of creating an integrated feedback loop easily. "Our clients around the world increasingly are looking for quality real-time data within the cloud," said Trey Berre, CME Group Global Head of Data Services. "This innovative collaboration with Google Cloud will not only make it easier for our clients to access the data they need from anywhere with an internet connection, but will also make it easier than ever to integrate our market data into new cloud-based technologies." 

Making messaging more accessible

In 2020, we have focused on making Pub/Sub even simpler. We observed that some of our users had to adapt their application design to the guarantees made by the service. Others were left building their own cost-optimized Apache Kafka clusters to achieve ultra low-cost targets. To address these pain points, we have made Pub/Sub much easier to use for several use cases and introduced an offering that achieves an order of magnitude lower total cost of ownership (TCO) for our customers. 

The cost-efficient ingestion option

We set out to build a version of Pub/Sub for customers who needed a horizontally scalable messaging service at a cost typical of cost-optimized, self-managed single-zone Apache Kafka or similar OSS systems. The result is Pub/Sub Lite, which can match or even improve upon the TCO of running your own OSS solution. In comparison to Pub/Sub itself, Pub/Sub Lite is as much as ten times cheaper, as long as the single-zone availability and capacity management models work for your use case. This managed service is suitable for a number of use cases, including:

• Security log analysis, where it is often a cost center and not every event must be scanned to detect threats 

• Search indexes and serving cache updates, which are commonly “best effort” cost-saving measures and don’t require a highly reliable messaging service

• Gaming and media behavior analytics, where low price is often key to getting startups off the ground

This guide to choosing between Pub/Sub and Pub/Sub Lite and the pricing comparisons can help you decide if Lite is for you. 

Comprehensive and enterprise-ready messaging that scales

This year, Pub/Sub added a number of features that will allow our users to simplify their code significantly. These features include: 

• Scalable message ordering: Scalable message delivery in order is a tough problem and critical for many applications, from general change data capture (CDC) to airplane operations. We were able to make this work with only minimal changes to our APIs and without sacrificing scalability and the on-demand capacity. Your applications that require ordering can now be much less stateful, and thus simpler to write and operate. There are no shards or partitions and every message for a key, such as a customer ID, arrives in order reliably. 

• Dead-letter topics automatically detect messages that repeatedly cause applications to fail and put them aside for manual, off-line debugging. This saves on processing time and keeps processing pipeline latency low. 

• Filters automatically drop messages your application does not care to receive, saving on processing and egress costs. Filters are configuration, so there is no need to write code or deploy an application. It’s that simple. 

• Data residency controls: In addition to Pub/Sub’s resource location constraints, which allows organizations to dictate where Pub/Sub stores message data regardless of where it is published, we have launched regional endpoints to give you a way of connecting to Pub/Sub servers in a specific region. 

• Publisher flow control (Java, Python) is perhaps the most notable of many updates to our client libraries. Flow control is another surprisingly tough problem, as many applications require multiple threads to publish data concurrently, which can overwhelm the client machine’s network stack and lose data unless the threads coordinate. With flow control, you can achieve very high, sustainable publish rates safely. 

• Also of note are configurable retry policy and subscription detachment. 

As one of our users recently said: “I’m going to go and use this right now.”

What’s next