Lowered TCO and reduced hours spent on cluster management
Increased engineering velocity by removing infrastructure management
Moved from weekly planning to a twice-daily cadence
Achieved resilience for critical cost calculations
Freed senior engineers to focus on proactive innovation
Delivery Hero scales Q-commerce with Managed Service for Apache Spark, cutting overhead and enabling real-time decisions.
Delivery Hero scales Q-commerce with Managed Service for Apache Spark, cutting overhead and enabling real-time decisions.
In the hyper-competitive world of on-demand delivery, a data platform must be both massively scalable and operationally resilient. For a global leader like Delivery Hero, achieving this balance is critical to their mission. Their solution lies in a sophisticated strategy for running Apache Spark on Google's Agentic Data Cloud, a unified platform that allows them to optimize for both cost at scale and resilience for their most mission-critical workloads.
The company is a pioneer in Quick Commerce (Q-commerce)—the next generation of ecommerce aiming to bring groceries and household goods to customers in under one hour and often in 20 to 30 minutes. For the data platform team supporting this vertical, this high-speed business model demands a data architecture that is equally agile and robust.
The foundation: A unified and open Agentic Data Cloud
A history of rapid growth and acquisitions meant Delivery Hero’s data was spread across hundreds of disparate sources. To break down these data silos, the company built its data cloud on a modern lakehouse architecture centered on BigQuery. A core principle driving this decision was a deep commitment to differentiated openness—embracing open-source technologies and standards to avoid vendor lock-in and ensure future flexibility.
The proof point: A pragmatic, two-pronged strategy for Spark
This commitment to open source led the Q-Commerce data engineering team to build their processing capabilities around Apache Spark. On Google Cloud, they found a flexible platform that allowed them to run these open-source workloads in two distinct ways.
Method 1: Managed clusters for foundational scale
For their large-scale, high-volume raw data import pipelines, the team runs Spark on provisioned Managed Service for Apache Spark clusters. These jobs are the workhorse of their data platform, providing the ideal balance of control, throughput, and cost-efficiency needed to reliably ingest petabytes of data.
Method 2: Serverless Spark for mission-critical resilience
While managed clusters handle scale, the team faced a different challenge with cost calculation workloads running on a self-managed Spark cluster. A routine but complex runtime upgrade caused an unexpected failure, triggering a strategic decision to use serverless Managed Service for Apache Spark and offload all infrastructure management to Google.
Making Spark easier: From firefighting to innovation
This strategic shift from a self-managed environment to serverless Spark perfectly illustrates the "Easier" pillar of running Spark on Google Cloud. It fundamentally changed the "jobs to be done" for Delivery Hero's senior data engineers.
Before: The engineering team was responsible for the operational health of the Spark environment, manually managing runtime versions and configurations. When the critical P1 job failed, it became an all-hands-on-deck "firefighting" exercise.
After: The job of the engineer has been elevated. By migrating the Weighted Average Cost (WAC) calculation to serverless Managed Spark, the team no longer has to think about the underlying infrastructure. As staff data engineer Hasan Cosan put it, "We don't want to care about the infra related issues there." This has freed up their most valuable engineering talent to focus on proactive, forward-looking initiatives.
Making the business faster: From weekly reports to real-time decisions
The reliability and performance of Spark on Google Cloud directly translate into making the business "Faster." The demand for high-frequency, near real-time insights at Delivery Hero is growing rapidly to keep pace with the speed of their operations. The stability of their data platform provides the fresh, accurate data needed to support this evolution. This allows critical functions like supply chain and inventory management to move from weekly or bi-weekly planning to a twice-daily cadence, enabling them to react instantly to market changes and optimize inventory with much greater precision.
Building a smarter foundation for AI
While the data platform team focuses on making Spark easier and faster, their work is the critical prerequisite for making the entire business "Smarter." A successful AI or machine learning model is only as good as the data it's trained on. By providing a robust, scalable, and reliable platform that delivers high-quality, timely data into BigQuery, the team creates the foundational layer that enables downstream Data Science and ML teams. This clean, governed data is the fuel for the intelligent models that power everything from demand forecasting to personalization, allowing the business to derive smarter insights and build smarter products.
Business outcomes: The power of a unified platform
By building on Google's Agentic Data Cloud and strategically leveraging Spark, the Q-Commerce data platform team achieved several key business outcomes:
- Operational efficiency and cost reduction: By choosing the right tool for each job—cost-effective Managed Spark clusters for scale and resilient serverless Managed Spark for critical workloads—they lowered their total cost of ownership (TCO) and reduced the operational hours spent on cluster management.
- Accelerated innovation and time to market: Freeing senior engineers from reactive infrastructure management directly increased the velocity of the data team, allowing them to develop and deploy new data products and pipelines faster.
- Fulfilling the mission: Ultimately, this stable and efficient data foundation directly supports the core mission to "deliver anything fast." The timely insights generated enable a more reliable and efficient delivery experience, driving the customer growth and retention that powers the business.
The story of Delivery Hero's Q-Commerce data team is a masterclass in modern data platform strategy. By building on a unified data cloud and making pragmatic choices about how to run their Spark workloads—using Managed Spark clusters for cost-effective scale and serverless Managed Spark for non-negotiable resilience—they have built a platform that is as efficient as it is robust, empowering them to fulfill their mission.
Delivery Hero's success demonstrates the power and flexibility of running Apache Spark on Google Cloud. Whether you need to manage large-scale clusters with fine-grained control or de-risk critical jobs with a serverless approach, Google Cloud provides the tools to make your Spark workloads easier, faster, and smarter.
Learn more about how you can get started with Managed Service for Apache Spark, BigQuery on Google’s Agentic Data Cloud today.
Delivery Hero is the world’s leading local delivery platform, managing over 11 million daily orders in around 65 countries with a workforce of 50,000 employees. The company connects over one million restaurants and one million riders globally to provide seamless food and quick-commerce delivery services.
Industry: Food Delivery
Location: Germany
Products: Managed Service for Apache Spark, BigQuery
