Applift: Optimizing with BigQuery for 300M mobile ad events a day

About Applift

Applift is a leading mobile ad tech company that empowers businesses to connect and activate customers in a mobile-first world by creating the next-generation platform for advertisers to reach and convert their audiences on mobile.

Industries: Technology
Location: Germany

Applift uses BigQuery with Cloud Storage, Compute Engine, and Google Data Studio to store, analyze, and visualize mobile advertising data, dealing with one million real-time bidding requests a second.

Google Cloud Results

  • Enables data access at an aggregated or granular level with BigQuery, providing insights into app user behavior
  • Produces data visualizations using Google Data Studio to support quick data analysis and speed up optimizations
  • Controls client access and keeps costs down by deploying partitions on BigQuery and alerts

Data matching takes minutes, down from seven hours

By leveraging data to find the best users for mobile app developers and advertisers, Applift has grown from a dynamic start-up in 2012 to having offices in nine locations globally. With its unified DataLift 360 platform, it enables its end users to integrate all relevant data, meaning advertisers can tap into various mobile supply channels through one access point and view real-time analytics to enhance their campaign's performance.

"At Applift, we don't want to let tech challenges slow down business. If that's happening, then we're not doing what we're supposed to do. Our tech team is very connected to the business side of things, we think about it every day."

Uzi Blum, VP Technology, Applift

To stay ahead of its competitors, Applift needs its data analytics to deliver complex calculations in as close to real time as possible and enable access at both an aggregated and granular level. Following significant growth in traffic in 2016, its previous open source solution was failing to provide this functionality and required constant maintenance. Applift found that moving to Google Cloud Platform was the answer.

"At Applift, we don't want to let tech challenges slow down business," explains Uzi Blum, VP Technology at Applift. "If that's happening, then we're not doing what we're supposed to do. Our tech team is very connected to the business side of things, we think about it every day."

Running big data analytics

For mobile advertisers promoting an app, securing return on advertiser spend (ROAS) relies on fast analytics and fraud detection. Applift processes 300M events daily on the publisher network side and one million requests a second on the real-time bidding side, but its previous data analytics system was struggling to deliver the necessary recommendations and reports in close to real time. It looked for a platform that would deliver processing power while requiring minimal maintenance.

To do that, Applift moved its database to Google Cloud Platform. "At the beginning of 2017, we decided to build a Python framework based on BigQuery, which allows us to do all the data transformations we need between external data sources to Cloud Storage, and to or from BigQuery. We store requests, bids, wins, impressions, installs, and post-install events, as well as third-party data. It took us about six weeks to set up an integration framework (Python based) and start getting data in BigQuery," says Uzi. "Now, all the data feeds into BigQuery and can be accessed at either an aggregated or a granular level. We can optimize campaigns by seeing an aggregated level of data or a particular record. We learn a lot from seeing the journey of the user, and can store historic data for longer. Six months of data in one table can be around 1.5TB but depending on the complexity, querying this table can take just a few seconds. Some of our matching operations used to take up to seven hours, now the processes take minutes, and we have no limitations on the range we want to query."

“Having BigQuery for the analytics part allows the campaign managers to do better optimizations on the advertisements, on the exchange level and placement levels, and so on. Instead of focusing on infrastructure maintenance, BigQuery allows us to focus on the business logic.”

Uzi Blum, VP Technology, Applift

"After the data comes to BigQuery, we analyze it in close to real time for fraud prevention," says Uzi. "We're able to take action on whether we approve a conversion or reject it. We also run mid-level once-an-hour optimizations that look into statistical KPI and distributions customized to our advertisers' needs. The BigQuery data can be consumed via several means, either directly from BigQuery by publishing it to files stored on Cloud Storage or extracting it to Cloud SQL."

"Having BigQuery for the analytics part allows the campaign managers to do better optimizations on the advertisements, on the exchange level and placement levels, and so on," says Uzi. "Instead of focusing on infrastructure maintenance, BigQuery allows us to focus on the business logic."

Visualizing data usage

In order to provide access for its internal business users, Applift uses Google Data Studio to visualize the data. "Using Google Data Studio, we can control exactly what kind of table and dataset our business user is looking at," says Uzi. "We have made them partition-dynamic, meaning a search only scans the relevant tables, not the entire dataset. That helps to keep costs down."

Applift also uses Google Data Studio to keep track of its overall spending by connecting it to BigQuery cost estimations. "That way, every day we are able to see which users run how many queries and how much data they consume," says Uzi. As the team uses G Suite, integration across the ecosystem is straightforward, and using separate accounts for each type of user also helps Applift keep track of usage and spending. "The other advantage of Google Data Studio is that it is so intuitive. Anyone from an account manager to top management can create a report, with minimal input from the BI team," says Uzi.

“Thanks to BigQuery, we have more time for innovation, which is very important to us. The digital advertising world is very fast-paced, so we need to always be moving forward. We now have the resources we need to investigate new business and data points.”

Uzi Blum, VP Technology, Applift

Harnessing AI

Switching to Google Cloud Platform has significantly improved the performance of Applift's analytics infrastructure. "It has speeded up delivery, and the amount of features we are able to deliver and business impact we can make," says Uzi. "Thanks to BigQuery, we have more time for innovation, which is very important to us. The digital advertising world is very fast-paced, so we need to always be moving forward. We now have the resources we need to investigate new business and data points."

In the last two years, Applift has worked on several smart applications based on recommendation engines and statistical optimization. The company's focus is to build data-based applications, which are revenue-driven. "The amount of investment by Google into AI is impressive. We are adapting Cloud Machine Learning Engine where we think we can leverage their knowledge and experience into our very specific domain," explains Uzi. "We want to make sure Applift is staying innovative for our engineers, developers, and analysts; we want to keep them hungry and passionate about what they're doing."

About Applift

Applift is a leading mobile ad tech company that empowers businesses to connect and activate customers in a mobile-first world by creating the next-generation platform for advertisers to reach and convert their audiences on mobile.

Industries: Technology
Location: Germany