Smart analytics reference patterns

This page provides links to sample code and technical reference guides for common analytics use cases. Use these resources to learn, identify best practices, and leverage sample code to build the analytics features that you need.

The reference patterns listed here are code-oriented and meant to get you quickly to implementation. To see a broader range of analytics solutions, review the list of Big data technical reference guides.

Anomaly detection

Solution Description Products Links
Building a Telecom network anomaly detection application using k-means clustering

This solution shows you how to build an ML-based network anomaly detection application for telecom networks to identify cyber security threats by using Dataflow, BigQuery ML and Cloud Data Loss Prevention.

Technical reference guide: Building a secure anomaly detection solution using Dataflow, BigQuery ML, and Cloud Data Loss Prevention

Sample code: Anomaly Detection in Netflow logs

Blog post: Anomaly detection using streaming analytics and AI

Overview video: Building a Secure Anomaly Detection Solution

Finding anomalies in financial transactions in real time using BoostedTrees

Use this reference implementation to learn how to identify fraudulent transactions by using a TensorFlow boosted tree model with Dataflow and AI Platform.

Technical reference guide: Detecting anomalies in financial transactions by using AI Platform, Dataflow, and BigQuery

Sample code: Anomaly Detection in Financial Transactions

Finding anomalies in time series data by using an LSTM autoencoder

Use this reference implementation to learn how to pre-process time series data to fill gaps in the source data, then run the data through an LSTM autoencoder to identify anomalies. The autoencoder is built as a Keras model that implements an LSTM neural network.

Sample code: Processing time-series data

General analytics

Solution Description Products Links
Building a real-time website analytics dashboard

Learn how to build a dashboard that provides real-time metrics you can use to understand the performance of incentives or experiments on your website.

Sample code: Realtime Analytics using Dataflow and Memorystore

Overview video: Level Up - Real-time analytics using Dataflow and Memorystore

Building a pipeline to transcribe and analyze speech files

Learn how to transcribe and analyze uploaded speech files, then save that data to BigQuery for use in visualizations.

Sample code: Speech Analysis Framework

Processing audio clips into a transcript by using Dataflow and the Speech-to-Text API

Learn how to process audio clips in real time to create a transcript in WebVTT format.

Sample code: Automatic WebVTT Caption From Streaming STT API By Using Dataflow

Log analytics

Solution Description Products Links
Building a pipeline to capture Dialogflow interactions

Learn how to build a pipeline to capture and store Dialogflow interactions for further analysis.

Sample code: Dialogflow log parser

Processing logs at scale using Dataflow

Learn to build analytical pipelines that process log entries from multiple sources, then combine the log data in ways that help you extract meaningful information.

Technical reference guide: Processing Logs at Scale Using Dataflow

Sample code: Processing Logs at Scale Using Dataflow

Pattern recognition

Solution Description Products Links
Detecting objects in video clips

This solution shows you how to build a real-time video clip analytics solution for object tracking by using Dataflow and the Video Intelligence API, allowing you to analyze large volumes of unstructured data in near real time.

Sample code: Video Analytics Solution Using Dataflow and the Video Intelligence API

Apache Beam Ptransform for calling Video Intelligence API: apache_beam.ml.gcp.videointelligenceml module

Processing user-generated content using the Video Intelligence API and the Cloud Vision API This set of solutions describes the architecture for deploying a scalable system to filter image and video submissions by using Cloud Vision API and Video Intelligence API.

Architecture: Processing User-Generated Content Using the Video Intelligence API and the Cloud Vision API

Tutorial: Processing User-Generated Content Using the Video Intelligence API and the Cloud Vision API

Sample code: Processing User-generated Content Using the Video Intelligence API and the Cloud Vision API

Apache Beam Ptransform for calling Cloud Vision API: apache_beam.ml.gcp.visionml module

Anonymize (de-identify) and re-identify PII data in your smart analytics pipeline This series of solutions shows you how to use Dataflow, Cloud Data Loss Prevention, BigQuery, and Pub/Sub to de-identify and re-identify personally identifiable information (PII) in a sample dataset.

Technical reference guides:

Sample code: Migrate Sensitive Data in BigQuery Using Dataflow and Cloud Data Loss Prevention

Predictive forecasting

Solution Description Products Links
Building an e-commerce recommendation system by using BigQuery ML

Learn how to build a recommendation system by using BigQuery ML to generate product or service recommendations from customer data in BigQuery. Then, learn how to make that data available to other production systems by exporting it to Google Analytics 360 or Cloud Storage, or programmatically reading it from the BigQuery table.

Technical reference guide: Building an e-commerce recommendation system by using BigQuery ML

Notebook: bqml_retail_recommendation_system.ipynb

Building a propensity model for financial services on Google Cloud

This solution shows how to explore data and build a scikit-learn machine learning (ML) model on Google Cloud. The use case for this solution is a predictive, propensity-to-buy model for financial services. Propensity models are widely used in the financial industry to analyze a prospective customer's inclination to make a purchase, but the best practices described in this solution can be applied to a broad range of ML use cases.

Technical reference guide: Building a propensity model for financial services on Google Cloud

Sample code: Professional Services

Building a propensity to purchase solution

Learn how to build and deploy a propensity to purchase model, use it to get predictions about customer purchasing behavior, and then build a pipeline to automate the workflow.

Technical reference guide: Predicting customer propensity to buy by using BigQuery ML and AI Platform

Sample code: How to build an end-to-end propensity to purchase solution using BigQuery ML and Kubeflow Pipelines

Blog post: How to build an end-to-end propensity to purchase solution using BigQuery ML and Kubeflow Pipelines

Building new audiences based on current customer lifetime value

Learn how to identify your most valuable current customers and then use them to develop similar audiences in Google Ads.

Sample code: Activate on LTV predictions

Predict mechanical failures using a vision analytics pipeline

This solution guides you through building a Dataflow pipeline to derive insights from large-scale image files stored in a Cloud Storage bucket. Automated visual inspection can help meet manufacturing goals, such as improving quality control processes or monitoring worker safety, while reducing costs.

Sample code: Vision Analytics Solution Using Dataflow and Cloud Vision API

Predicting customer lifetime value

This series shows you how to predict customer lifetime value (CLV) by using AI Platform and BigQuery.

Technical reference guides:

Sample code: Customer Lifetime Value Prediction on Google Cloud

Real-time clickstream analytics

Solution Description Products Links
E-commerce sample application using streaming analytics and real-time AI

The e-commerce sample application illustrates common use cases and best practices for implementing streaming data analytics and real-time AI. Use it to learn how to dynamically respond to customer actions by analyzing and responding to events in real time, and also how to store, analyze and visualize that event data for longer-term insights.

Technical overview: E-commerce sample application using streaming analytics and real-time AI

Sample code: E-commerce sample application for Java

Interactive demo: Explore Google's Stream Analytics

Overview video: Activate real-time web experiences with Stream Analytics