Data Analytics

Streaming analytics 101: Making modern data decisions with ease

December 10, 2020

Evren Eryurek PhD

Director Product Management, Google Cloud

Within the realm of modern data collection, streaming analytics is just what it sounds like: a flow of constantly moving data called event streams. These streams comprise events that occur as the result of an action at any moment, such as a web click, a transaction, or a fraudulent activity. Streaming analytics provides the ability to constantly monitor these operational events and automatically perform an action as soon as these event streams are generated. When streaming analytics is working well, you can ask questions of your data and get answers that help you make better decisions: What are your customers buying online at any given moment? What error messages are they seeing, and how often? To get streaming analytics right, it helps to think about what you want to get out of it. Think about where you want to focus your time and resources, and which data can provide you the most relevant insights.

Why should you consider streaming analytics?

According to IDC, by 2025, more than a quarter of data created in the global datasphere will be real time in nature. What’s driving the growth? Well, what aspects of your life or business aren’t creating a digital trail today? There are a lot of forces at work creating all this real-time data: digital and connected devices and connected devices innovations such as industrial sensors, smartphones, wearables, and car navigation; online interactions such as purchase histories, clickstreams, advertising, inventories, and ledgers; and digital communication services such as social media posts, photos, email, and collaboration platforms. The volume, velocity and variety of data is exponentially increasing and businesses have to be able to charter this new brave world in order to remain competitive.

The de facto approach of organizing data for analytics has been in batch, where new data may only be processed hourly, daily or even weekly. This approach focuses on historical information, which limits businesses to only be able to react to past events. In today's business environment, where data often has become a strategic differentiator, if data is not processed in near-real time, decisions may be made too late. Real-time data from event sources provides a high-value opportunity to act on a perishable insight within a tight window. That means businesses need to act fast. To do so, we need analysis to arrive at the point of action in real time. That’s the difference between preventing fraud and discovering fraud, a customer making a purchase or abandoning a cart, and proactive/effective and reactive/ineffective customer service.

There are plenty more places where real-time data can make a difference on a business’s bottom line:

Creating targeted pricing strategies. If your business runs promotions on items, testing the right pricing is paramount to ensuring that customers buy your products. Streaming data can allow for more precise actions on price elasticity for each customer, timing of discounts, customized offering and sales channel
Detecting fraud in real time. Access to real-time streaming data means you can respond quickly to any financial irregularities—so instead of writing off the costs of a fraudulent transaction, a company can flag it immediately.
Building customer loyalty and capturing market share. Building more responsive relationships helps to gain customer trust and capture revenue. So companies that can propose and interact with their prospects in close to real time with a customized offering of content, pricing, and solution will lead to loyal and happy customers.
Finding operational efficiencies. Real-time data analytics can continually monitor data integrity and let you respond automatically. Adoption of streaming can help eliminate manual processes that are susceptible to error, enable better data interoperability with other organizations, and increase speed-to-market by making data more actionable.

How to assess if streaming analytics is right for your business

Not all your problems will benefit from streaming analytics equally and getting started with real-time data can be overwhelming. There are plenty of ways to capture, ingest, and process data, and plenty of information to be gleaned from analyzing your company’s data. Which data is the right data to gather and analyze? What’s the right way to prioritize the data you want to capture in real time, and which data can wait? To decide if streaming analytics is right for you, it helps to consider the following:

Assess your current environment: identify which applications generate data in your organization and rank those data streams based on their importance. For example, in retail, the need for real-time applications would probably rank higher for website clickstreams compared to back office payroll, given direct revenue generation opportunity.
Map real-time analysis use cases to the data streams: decide which are your critical activities to improve top and bottom line, whether it be responding to customers, detecting faulty products or enhancing security.
Evaluate buy vs. build: Do you have staff with the right skill sets to capture the maximum value from the technology? Do you have the resources to hire these experts? This will have an implication on time to value as you choose between an open source technology vs. a fully managed service.

At Google Cloud, our fully managed, real-time streaming platform includes Cloud Pub/Sub for durable message storage and real-time message delivery, Cloud Dataflow, our data processing engine for real-time and batch pipelines, and BigQuery, our serverless data warehouse. We design for flexibility and scalability, so we also support and integrate with familiar open-source tools, plus other Google Cloud tools like Cloud Storage and our databases. The result is you don’t have to make compromises as streaming and batch sources are pulled into one place for easy access and powerful analytics.

We offer reference patterns to help you get started with an architecture for your high-value use cases.