Twitter takes data activation to new heights with Google Cloud

Twitter is an open, social platform that’s home to a world of diverse people, perspectives, ideas, and information. We aim to foster free and global conversations that allow people to consume, create, distribute, and discover information about the topics they care about the most.

Founded in 2006, Twitter keeps a watchful eye on emerging technologies to maintain a modern platform that can meet the needs of the changing times. These early investments helped accelerate Twitter's product but predated modern open source equivalents. As a result of its desire to leverage more open source technologies to keep up with the changing times, Twitter wanted to use the data it collected to maximize the user experience. However, its past generation of operational tools highlighted a need to create less time-consuming and more reliable data processing techniques that allowed Twitter developers to automate complex, manual tasks to relieve developer burden. This presented an opportunity for Twitter to modernize its tools and glean valuable insights that would be transformative for the evolution of its products and partnerships with advertisers. With the plan to standardize and simplify its approach to data processing across its operations, Twitter progressively migrated its operations to BigQuery on Google Cloud.

In the complex, competitive world of programmatic advertising, the relevance, quality, and interpretation of data insights are critical in a company’s ability to stay ahead of ever-changing needs. The ability to streamline its approach to large-scale data processing quickly became an anchor in Twitter’s plan to better align its goals with those of its advertisers and customers. With the recent migration of its advertising data from on-premises to Google Cloud, Twitter has leveraged several Google Cloud solutions, notably BigQuery and Dataflow, to facilitate this greater alignment.

Leveraging BigQuery for improved advertising partnerships and data extraction

Aligning the goals of advertisers and customers with those of a company is a considerable challenge, but for a company with hundreds of millions of avid users like Twitter, developing and executing an approach that balanced the needs of all parties was proving to be a complex task. Pradip Thachile, a senior data scientist responsible for Twitter’s revenue team's adoption of Google Cloud, likened the process to a kind of flywheel that allows the Twitter team to work in collaboration with advertising partners to develop and test hypothetical approaches that center its goals and those of advertising partners. He explained the essential role of the BigQuery solution in the synthesis of these goals with an eye on the optimization of business growth for all involved. “Mating all this is a nontrivial problem at scale. The only way we can accomplish it is by being able to build this kind of scientific learning flywheel. BigQuery is a critical component, because the velocity with which we can go from hypothesizing to actual action through BigQuery is huge.”

As the anchoring service for the ingestion, movement, and the extraction of valuable insights from all data at Twitter, BigQuery is the engine of Twitter’s recent optimization of internal productivity and revenue growth.

Data modeling for optimized productivity and value extraction with Dataflow

As a fully managed streaming analytics service, Dataflow has proven to be a time-saving solution that contributes significantly to the enhancement of productivity at Twitter. Through the reduction of the time invested in manual tasks for scaling, Dataflow facilitates the seamless and effortless organization and templatization of the movement of the archetypal data sets at Twitter. With less time devoted to the calibration of operational tools, Twitter’s team can focus on the higher-value tasks related to the discovery and development of innovative ways to further leverage its data insights. 

Reliable support with data expertise from Google

Notable for its expertise in data, Google Cloud contributed substantial technical support to Twitter. The Twitter team routinely accessed the Google Cloud product team for guidance on ingestion velocity as they leveraged the sizable ingestion capabilities of BigQuery for its data. At a higher level, the Google Cloud support team supplied valuable resources including white papers and use cases that could enhance Twitter’s performance. Thachile describes the value of Google Cloud’s support, “Google Cloud provides a very effective stratified layer of support. They can be as close to the problem as you'd like them to be.”

For more of the story about how Twitter is using BigQuery, read this blog from Twitter.