Viant: Creating a self-service operations team through easy-to-use data exploration and automation tools
About Viant Technology LLC
Viant Technology LLC enables marketers to plan, execute, and measure their digital media investments. The Viant Advertising Cloud provides access to over 250 million registered users in the United States, infusing accuracy, reach, and accountability into cross-device advertising.
Tell us your challenge. We're here to help.
Contact us
About Potens.io
Google Cloud Partner Potens.io develops Potens, a suite of powerful, efficient big data tools built on top of BigQuery.
Viant stays ahead of competitors by democratizing data across the organization and dramatically speeding up query results with BigQuery and Google Cloud Partner Potens.
Google Cloud results
- Provides BI team with full, self-service access to streaming ad logs
- Scales as needed without adding expensive, hard-to-manage hardware
- Enables teams to rapidly answer customers' complex data queries
- 10x increase in questions BI team receives and responds to from clients
Operations team answers 250K database queries monthly
Most companies deal with big data. Viant deals with massive data. More than 25 billion ad requests are made daily on its cloud-based platform for marketers, ballooning up to 100 billion requests during busy periods. Billions more bids of ad impressions move across the platform every day. Each ad request, bid, or impression contains more than 700 data points.
In 2011, Viant's growing volume of data presented the company's Operations team with two major challenges: gaining full access to all data and the need to eliminate repetitive, manual tasks through self-service automation.
At the time, the Business Intelligence (BI) team within Operations had only a single user interface available to access extremely limited, aggregated data. As a result, the BI team couldn't easily or efficiently answer basic questions Viant's clients asked, such as how many users a client's campaign had reached.
"There was no comparison between BigQuery and any other tool we'd used. BigQuery was exponentially faster and opened the floodgates to new reporting and insight possibilities."
—Steven Ohrnstein, SVP, Platform Automation and Analytics, ViantThis type of basic analysis required going back to the log-level data to deduplicate unique users across multiple days and campaigns. "A simple question required the team to fill out a 20-question request form, written in engineering lingo that required a dictionary to translate, to then submit to the Data Warehouse team," explains Steven Ohrnstein, SVP, Platform Automation and Analytics for Viant. "From there, you'd have to wait three to five days for a reply. The odds were strong that an Operations team member hadn't properly written the request, or the engineer hadn't accurately translated the data, which would then create additional back-and-forth. It could take two or three weeks to answer a simple question."
The second challenge — the need to automate tasks — was just as formidable. Back in 2011, all the manual, repetitive tasks required to gather data and share insights routinely bogged down the Operations team, cutting into the time team members needed to focus on thorough data analysis and support. For example, once the team accessed the limited data, that data was then exported to a desktop spreadsheet application for analysis and distribution. If an updated version of the same data was needed the following day, or if the data query hadn't been interpreted correctly, a team member had to repeat the process all over again.
"We were stuck, effectively reinventing the wheel, day after day," says Steven.
In 2013, Viant began migrating its data and processing away from running on Apache Hadoop, Netezza, and their own hardware infrastructure in favor of Google Cloud, especially BigQuery. "BigQuery was a huge leap forward for us in terms of performance and efficiency," Steven says. "The BI team went from having limited aggregated data to full access of streaming ad logs. Overnight, the Data Warehouse team eliminated themselves as the barrier between Operations and their data needs. There was no comparison between BigQuery and any other tool we'd used. BigQuery was exponentially faster and opened the floodgates to new reporting and insight possibilities."
Speeding up data access
The complicated, expensive infrastructure supporting Viant's data warehouse operations was a big factor hindering the ability to streamline and speed up data access.
"Because BigQuery is serverless and built to be easily scalable on demand, we don't worry about scaling at all. And we don't have to involve the IT department to help us scale."
—Fabrizio Blanco, CTO, ViantBefore Google Cloud, Viant couldn't expand its data warehouse and supporting hardware fast enough. "Our business can be seasonal," says Fabrizio Blanco, CTO at Viant. "Sometimes, we have a lot of data to process for our customers, other times we have much less. We didn't usually have much warning when we'd need to add more hardware or storage."
Beyond seasonal fluctuations, the company used to buy more hardware and storage to keep up with the growth of its customer base. "It was expensive, difficult to manage, and not always easy to predict," Fabrizio says. "Because BigQuery is serverless and built to be easily scalable on demand, we don't worry about scaling at all. And we don't have to involve the IT department to help us scale."
The elastic capacity scaling of BigQuery has also helped boost database query performance. Long query and data extract load times were the norm for SQL and the data warehouse solution Viant had been using. "With BigQuery, we can run queries much faster, without the need to buy more hardware to increase the speed," Fabrizio says. "Running a database query before BigQuery could take two days to fully execute. With BigQuery, it takes only a few minutes."
With access to log-level data in BigQuery, the BI team can now easily and quickly answer basic ad-hoc queries from Viant customers about such things as their ad campaign's reach, instead of having to put in a formal request that could take days to fulfill. The ability to answer basic questions so much faster has resulted in a 10x increase in the number of questions Viant's BI team receives and responds to from clients.
"Suddenly the BI team could answer basic questions all day long, in a self-service fashion," Steven adds. "But with the increasing volume and complexity of requests, we needed a way to automate and turn those requests around faster."
Building on top of BigQuery
The majority of customer requests are too complicated to be solved in a single query. For example, a typical production request might require multiple steps, including accessing data from Cloud Storage, performing some transformations on the data, running aggregations for certain conditional steps and sending notifications. In fact, the most complex requests could require hundreds of steps. In addition, the sequence of steps needs to be triggered by external events, such as the presence of new data in Cloud Storage.
"Our business is all about data, and when we moved to BigQuery and Potens, suddenly we were miles ahead of the competition in terms of delivering great data to clients in a timely fashion. And by providing even more value to our clients, they stick with us."
—Fabrizio Blanco, CTO, ViantTo meet the challenge of automating respective tasks, in 2015 Viant built Potens, which is now a commercially available platform and a Google Cloud Partner. Potens offers two tools built on top of BigQuery: Magnus, for automating data query workflows, and Goliath, a data exploration tool for managing and querying BigQuery data.
Using Potens, the Operations team has built an algorithm that automates that value of each ad request, eliminating the need for the team to spend time on manually managing bid prices. "Potens enabled the automation of over 300 new and custom reports and processes that enabled the Operations team to concentrate more of their time driving results for their clients through performance optimizations and less of their time collecting and compiling data." Steven says.
With Potens running on BigQuery, the BI team for the first time has the access and means to blend multiple data sources together, on their own, to provide Viant customers with more sophisticated insights into their ad campaign data. For example, because of the automations that Potens enabled, nearly 3,000 custom requests for data intelligence from Viant customers were answered in 2018 via email within minutes. In the past, customers had to fill out a request form, then wait 24 to 48 hours for the BI team to manually pull the queries and respond. "The ability to automate so many repetitive steps through the power of BigQuery and Potens has been a huge leap forward for us," Steven says.
The entire Operations team has since been provided full access to data and Potens with weekly training to help ensure all team members are self-sufficient in their data needs. "In the end, the combination of BigQuery and Potens enabled the Operations team to go from answering a single business question in two or three weeks to answering over 250,000 queries every month," Steven says.
"In addition, the Data Warehouse team was freed from managing infrastructure and handling repetitive database query tasks which gave them the opportunity to focus on developing new features and capabilities, such as building Potens.," Fabrizio says. "Potens and BigQuery make it much easier for everyone in our organization — from Operations to Data Scientists and Product Managers — to schedule complex data queries without having to be an engineer.
Providing more value to clients
The combination of BigQuery and Potens gives Viant a competitive advantage. "Our business is all about data, and when we moved to BigQuery and Potens, suddenly we were miles ahead of the competition in terms of delivering great data to clients in a timely fashion," says Fabrizio. "And by providing even more value to our clients, they stick with us."
Tell us your challenge. We're here to help.
Contact us
About Viant Technology LLC
Viant Technology LLC enables marketers to plan, execute, and measure their digital media investments. The Viant Advertising Cloud provides access to over 250 million registered users in the United States, infusing accuracy, reach, and accountability into cross-device advertising.
About Potens.io
Google Cloud Partner Potens.io develops Potens, a suite of powerful, efficient big data tools built on top of BigQuery.