Query fresh Google Ads data in BigQuery, via Apache Beam and Dataflow

Modern marketers know that in the digital ad marketplace, speed is the key to maximizing the return on investment on your ad spend. The challenge is that most advertisers rely on data that is days, weeks, or even months old — and in an environment that moves as quickly as the time to return a Google search result, advertisers stand to gain a massive advantage by leveraging real-time data.

Today we are excited to announce native support for writing Google Ads data to BigQuery powered by Dataflow, enabling organizations to make data-driven decisions on campaign strategies in real-time. Users can now query their own campaign, conversion, impression, and bidding data from Google Ads to BigQuery using our new Dataflow template, which makes connecting these systems as simple as filling out a webform. If you’re looking to build your own pipeline for custom transformations, analytics, or writing to multiple data stores, GoogleAdsIO is available in the open-source Apache Beam Java SDK. Customers that would have to wait at least 24 hours for daily data loads can now analyze events as they happen with a ~10 min buffer. Together, these two new features will help marketers adopt intraday bidding strategies, modifying bidding strategies within minutes of serving ads.

There are two key pieces of this launch: the Google Ads I/O connector and the Dataflow template. We’ll explore each of these in the following sections.

Google Ads I/O

Our new GoogleAdsIO is available in the Apache Beam Java SDK. Apache Beam is an open-source SDK that is a unified framework for expressing batch and stream data processing pipelines. Apache Beam gives developers the flexibility to write pipelines in the programming language of their choice and run those pipelines using the execution engine they prefer. Apache Beam already supports 60+ connectors and offers a comprehensive framework to develop custom I/O connectors if your system is not supported.

GoogleAdsIO calls the Ads Reporting gRPC-based API to fetch data from your Google Ads account. Google’s Professional Services Organization (PSO) architects worked closely with the Ads Developer Relations team to implement best practices with the I/O connector, adding features like transparent handling of OAuth credentials and flexible rate-limiting policies. These features ensure the control of calls made to the API does not violate per account or developer token limits, so that you can rapidly prototype your own custom pipeline safely. In summary, GoogleAdsIO utilizes the flexibility of the Apache Beam I/O framework to make sure you can query your creative, bidding strategy, and campaign from Google Ads with safety and reliability first.

Dataflow template

Dataflow templates connect data stores across your Google Cloud project, simplifying common data ingestion and replication tasks to filling out a short web form. Under the hood, Dataflow templates prepackage Apache Beam code and separate the development environment and execution environment, which then enables users to extend these pipelines to users with no programming skills. Companies use Dataflow templates to define common data processing workflows, and surface them to their internal users who don’t have any knowledge of Beam. All users need to do is fill out user-defined fields relevant to the template, and they are off and running.

To extend the magic of our new GoogleAdsIO connector, our PSO team created a Dataflow Flex template that reads from Google Ads and writes to BigQuery. Instead of users having launch a program on their local machines, this template allows them to pass in a Google Ads query through the Google Cloud console and return their results to a BigQuery table for further analysis. Another powerful feature of this template is the ability to pass in a list of Google Ads account IDs to execute the query, as opposed to manually going into each account ID in the Google Ads interface and pulling individual reports. This alone will save teams dozens of toilsome hours per month and will help divert that saved time towards experimentation of new bidding strategies like Maximize Clicks, Target CPA, or Target ROAS, to drive better outcomes. Users can also use Dataflow Data Pipelines to automate customized reporting with the help of our integrated scheduling tool.

Customer impact

Customers have already benefited tremendously with this template. Just Eat Takeaway.com, a leading global online food delivery marketplace, shares their experience:

“What this pipeline is really good at is the heavy lifting of connections, processing that connection to the Ads API, handling the downloads, cleaning the file, loading it to BigQuery. Normally, it would take us a couple of weeks to develop a bespoke report from Google Ads from our engineering team. With this, it was really quick — I opened it up, put in my query, specified the columns, and off you go.” - Irfan Rafiq, Manager of Performance Technology at Just Eat Takeaway.com

You can start on your journey by heading to Dataflow in the console, selecting the Create Job from the Template button at the top of your screen, and selecting the Google Ads to BigQuery template in the drop down. We’re excited to see how you unlock new bidding strategies with this template!

Learn more

• Google Ads to BigQuery documentation, reference for launching the template
• GoogleAdsIO Apache Beam documentation, reference for building your own Beam pipeline with GoogleAdsIO
• Booking.com’s case study on using Apache Beam and the Ads connector

_{Special thanks to: Steven van Rossum, Google Cloud PSO, Natalija Najdova, Google Cloud Customer Engineering, Cory Liseno, Google Ads Developer Relations, David Stevens, Google Ads Product Management}