Migrated two petabytes of data to a managed lakehouse
Accelerated data joining by up to 60%
Completed setup in weeks versus months
By migrating to Google Cloud’s Lakehouse, Etsy accelerated ML pipelines and improved search relevance to match unique items.
By migrating to Google Cloud’s Lakehouse, Etsy accelerated ML pipelines and improved search relevance to match unique items.
Overcoming file-path dependency and data silos
The global ecommerce marketplace Etsy has a clear mission: keep commerce human. Connecting more than five million sellers with over 86 million buyers seeking unique, handmade, and vintage goods, the heart of the platform is human connection. Shoppers get to know the person who created their purchase, discovering meaningful items and the unique stories behind them.
Fostering this human-centric marketplace across millions of interactions requires a massive, data-driven backbone to successfully match the right item with the right buyer at the right time. Behind the scenes, Etsy’s data engineering team acts as an enablement layer that bridges deep infrastructure and product-facing groups. The group builds middleware and services to provide reusable building blocks, freeing product teams from the complexity of general-purpose cloud management so they can focus on delivering valuable features to users.
To serve its massive community, Etsy relies on multiple compute engines to process vast amounts of structured and unstructured items and search data. Previously, users read and wrote to Google Cloud Storage using Apache Spark or similar processing engines by referencing direct file paths or internal modules. Because this setup lacked a unified metadata layer and schema awareness, developers had to manually open and inspect the raw data files simply to understand their underlying structure, columns, and data types before they could properly write code to analyze them.
Making this data accessible to analytics teams in BigQuery required complex data-movement patterns. Teams had to repeatedly create external tables or export raw files into BigQuery, work on them, and move them right back out to Cloud Storage. Etsy initially attempted to build an internal, hand-rolled lakehouse catalog to solve these visibility problems, but the team quickly ran into heavy operational overhead trying to manage hosting, infrastructure configuration, and complex security permissions.
To maintain its experimentation velocity, Etsy needed a managed lakehouse architecture that brought data pipelines and analytics engines under a single metadata umbrella.
Iceberg and Lakehouse technology allow us to join datasets in a much more efficient way that wasn't possible before. Instead of copying a ton of duplicate data or building entirely new datasets, we can keep one main dataset and use highly efficient joins based on the data format and storage layer.
Matthew Hall
Senior Engineering Manager, Etsy
An automated lakehouse built on
open-source standards
Building on its existing infrastructure, Etsy turned to Google Cloud for its Lakehouse for Apache Iceberg technology to remove management complexity while retaining the flexibility of open standards like Apache Iceberg. Deployed automatically through Terraform, Etsy’s new architecture centers around a custom catalog that connects disparate processing environments.
"Iceberg and Lakehouse technology allow us to join datasets in a much more efficient way that wasn't possible before," says Matthew Hall, senior engineering manager at Etsy. "Instead of copying a ton of duplicate data or building entirely new datasets, we can keep one main dataset and use highly efficient joins based on the data format and storage layer."
This efficiency is largely due to the ability to now use storage-partitioned joins (SPJ) in Spark, a capability unlocked by the Iceberg format. By implementing SPJ, Etsy has accelerated data joining by up to 60%.
By using Google Cloud’s Managed Service for Apache Spark serverless solution, Etsy established a streamlined, recommended path for all big data compute jobs. The Lakehouse runtime catalog, based on the Iceberg REST catalog, eliminates the need for developers to track raw Cloud Storage file paths. Now, data written through Spark jobs becomes instantly and automatically readable in BigQuery for analytics teams, allowing seamless interoperability across the organization's favored engines. With a self-hosted solution, the setup would’ve taken months. With Google’s Lakehouse solution, the team completed setup in weeks.
We're always going to be a company that spans multiple compute engines. Being able to use the data with Google’s Managed Spark, create it in Cloud Storage, and know that the right people can access it in BigQuery has been huge.
Laura Richards
Data Engineer, Etsy
“We're always going to be a company that spans multiple compute engines,” says Laura Richards, data engineer at Etsy. “Being able to use the data with Google’s Managed Spark, create it in Cloud Storage, and know that the right people can access it in BigQuery has been huge.”
Accelerating machine learning and search relevance
By taking a strategic, use-case-driven approach rather than a single large-scale migration, Etsy seamlessly moved more than two petabytes of data to the new architecture. Workloads that previously struggled under old configurations—such as data engineering pipelines that pulled large tables out of BigQuery into standalone Spark applications—saw immediate runtime performance gains and cost optimizations.
“Our cost savings go hand-in-hand with our experimentation velocity and the ability for teams to iterate quickly,” Hall explains. “If you can move both of those things in a positive direction at the same time, that’s a massive win.”
These cost and speed efficiencies directly improved Etsy's machine learning pipelines. Since massive data joins are no longer cost-prohibitive, machine learning models can process significantly larger datasets and train on many more feature variations. This supports a top company objective at Etsy: to match shoppers with the perfect items at the right time.
Our cost savings go hand-in-hand with our experimentation velocity and the ability for teams to iterate quickly. If you can move both of those things in a positive direction at the same time, that’s a massive win.
Matthew Hall
Senior Engineering Manager, Etsy
Ultimately, its lakehouse architecture built on Google Cloud allows Etsy's engineering teams to offload infrastructure maintenance to a trusted environment. Backed by a close, collaborative relationship with Google Cloud product owners and technical account managers, Etsy continues to provide fast feedback and help shape the future product roadmap.
“With Google Cloud, we get the best of both worlds,” Hall says. “We get all the muscle and reputation of an industry leader together with the direct access that usually only comes with a smaller vendor. That’s a rarity, and we’re excited to see what we can do next.”
Etsy is a global ecommerce marketplace connecting millions of buyers and sellers of unique, handmade, and vintage goods. Its mission is to keep commerce human by fostering personal connections.
Industry: Technology
Location: Global
Products: Cloud Storage, BigQuery, Lakehouse for Apache Iceberg, Managed Service for Apache Spark
