Lifesight: Building a unified consumer graph with the support of Google Cloud
About Lifesight
Lifesight is a customer intelligence platform that helps brands and enterprises leverage identity resolution and data enrichment to power their customer data strategies. Its industry-leading solution enables the transformation of customer data into actionable insights that help drive business decisions, optimize marketing spend, and improve customer experiences.
Tell us your challenge. We're here to help.
Contact us
About Searce
Searce is a global Google Cloud Premier Partner that helps businesses with end-to-end migration and integration of cloud solutions.
Lifesight leverages Dataproc and Cloud Bigtable to crunch petabyte-scale data, providing accurate and holistic customer insights for brands and enterprises so they can make better business decisions.
Google Cloud results
- Reduces query processing time by 70% with BigQuery
- Analyzes petabytes of customer data in minutes for better business insights
- Easily experiment with graph databases on Cloud Bigtable
- Leverages Bigtable for 150,000 writes per second (with 10 nodes)
- Fast turnaround time for ad hoc queries from large streams of data
Organizes 50 TB of data monthly with Dataproc
In the world of digital marketing, every piece of data is valuable in driving the success of a campaign. A brand that really knows its customers can reach out to them more effectively, whether it’s understanding their purchasing habits or how they feel about the brand.
"We aim to give companies a 360-degree view of their customers, and as a data-heavy company, we felt that Google Cloud, and its suite of products, was most suited to our business."
—Amit Nandan, Chief Strategy Officer, LifesightTo help businesses understand their customers better, Lifesight pieces together a range of customer datasets from various sources, before combining them on its platform to analyze them. "Understanding customers better shouldn't be difficult. Our vision is to become the go-to self-serve platform to help businesses easily leverage big datasets to deliver better customer experiences, optimize marketing spend, and make better data-driven decisions," says Amit Nandan, Chief Strategy Officer at Lifesight. "We aim to give companies a 360-degree view of their customers, and as a data-heavy company, we felt that Google Cloud, and its suite of products, was most suited to our business."
Lifesight’s customer intelligence platform processes more than 4.6 billion non-personally identifiable information data points daily. That is why having a reliable platform that can manage the load is paramount. Lifesight needed a technology solution that could support large streams of data and run powerful queries on an ad hoc basis in the most cost-effective way.
Today, Lifesight runs its entire operations on Google Cloud. BigQuery and Cloud Bigtable support its use cases in terms of improving query speed and reducing turnaround time for the ad hoc projects, while Dataproc ensures a seamless and secure end-to-end data pipeline.
Migrating to Google Cloud with BigQuery
Lifesight worked with Searce to manage the migration to Google Cloud. After a four-week proof of concept (POC), the team managed to complete a phase-by-phase migration in just six weeks.
"Our setup was not simple because of the scale and complexity of the infrastructure," says Amit. "The Searce team really made sure they understood our exact requirements and went above and beyond to enable a seamless migration."
The POC replicated two of Lifesight's pipelines with three months worth of data and found that the average query processing time was reduced by 75% with BigQuery. The robust data platform also meant that Lifesight could now analyze petabytes of data within just a few minutes. With that in mind, the Searce team used BigQuery to build a data warehouse and migrated its data processing frameworks. For scalability, Lifesight parks its different application stacks on Google Kubernetes Engine (GKE).
Even after the migration was complete, the Searce team continued to propose ways to increase optimization. "The team has never been hesitant to pull in external resources to make sure that they have everything they need to support us. As an example, even after this migration, they are working with us on ways to improve the optimization of our JanusGraph cluster, which is currently stored on Bigtable," Amit says.
"Having a reliable fully managed cloud service like Dataproc is the most important factor for us, where we are able to ingest and transform data from various sources into a format where it can be consumed by some of our downstream applications."
—Amit Nandan, Chief Strategy Officer, LifesightIngesting and analyzing terabytes of data at scale
As a company that ingests 40 TB of new data every month, Lifesight relies heavily on Dataproc as a managed service to run all of its Spark jobs, transforming batch and streaming data that come from a wide range of data sources. This includes information such as lat-long data linked to certain consumers or other user-related data like web browsing behavior and purchase habits.
"Having a reliable fully managed cloud service like Dataproc is the most important factor for us, where we are able to ingest and transform data from various sources into a format where it can be consumed by some of our downstream applications," says Amit. As a data company, Lifesight has other pipelines of different streaming jobs running concurrently. It moved its existing Kafka pipelines to Pub/Sub when it migrated to Google Cloud, which powers one of its applications in near real time.
Recognizing the fast read and write speed of Bigtable, along with its ability to scale up and down seamlessly, Lifesight frequently stores and experiments on it with graph databases from its holistic consumer profiles. As Lifesight continues to grow its JanusGraph cluster, the storage continuously increases as more data is added to power the graph. "When creating graph databases, you sometimes create millions or billions of vertices and edges to form the relationship of different attributes. When you use JanusGraph and Bigtable as the underlying storage layer, you can both execute fast queries and scale your storage layer independently for the size needed," explains Amit. The team can continue to add nodes as needed without worrying about storage, despite the complexity of the graph implementation.
Data scientists can model their data on the NoSQL database so that all the different attributes are able to be queried against each other, and the team can get more specific results. "With Bigtable, we are able to slice and dice the data that is available to generate very specific datasets quickly, to get a holistic and enriched view of consumers' first party data in almost real time," adds Amit.
“Demand for services like ours is very much on the horizon, so one thing I can say for sure is that the scale of data we handle will grow manyfold, if not exponentially, and this is where we will look to tap more into the different Google Cloud services.”
—Amit Nandan, Chief Strategy Officer, LifesightMaking holistic customer insights available globally
With the right platforms in place to support growth, Lifesight has its eyes set on the global arena. From APAC to its recent expansion into Europe and the US, it plans to further scale Cloud Bigtable and JanusGraph to support more than 2.3 billion users' data and generate even deeper insights.
"Demand for services like ours is very much on the horizon, so one thing I can say for sure is that the scale of data we handle will grow manyfold, if not exponentially, and this is where we will look to tap more into the different Google Cloud services," says Amit.
Tell us your challenge. We're here to help.
Contact us
About Lifesight
Lifesight is a customer intelligence platform that helps brands and enterprises leverage identity resolution and data enrichment to power their customer data strategies. Its industry-leading solution enables the transformation of customer data into actionable insights that help drive business decisions, optimize marketing spend, and improve customer experiences.
About Searce
Searce is a global Google Cloud Premier Partner that helps businesses with end-to-end migration and integration of cloud solutions.