SIGOPS Hall of Fame goes to Spanner paper — here’s why that matters

Earlier this month, SIGOPS announced that it had selected the paper, “Spanner: Google’s Globally-Distributed Database" for the 2022 SIGOPS Hall of Fame Award, an honor bestowed on the most influential Operating Systems papers published by the organization.

In giving this award, the award committee stated:

“Spanner showed how to balance the requirements of consistency, availability, and partition tolerance for a globally replicated transactional database system. In doing so, Spanner simplified the implementation of a range of higher level services which could then scale across the planet through a familiar and powerful transactional API. The key insights of leveraging clock synchronization using TrueTime and a dedicated, redundant, high-capacity datacenter WAN distinct from the public Internet as the basis for scaling remain fundamental design patterns today.”

As one of the original authors of the paper, to say that I am humbled by this award is an understatement. It’s also taken me on a trip down memory lane, and gotten me thinking about all the ways — expected and unexpected — that Spanner has influenced not only how Google builds its applications, but also how it’s simplified application design and operations for organizations that aren’t running at hyperscale. 

From very early on — think early 2000s — one of Google’s biggest challenges was scaling its software to keep up with the rapid growth of the internet. We needed a database that scaled, that could be built with a small team. Those needs led to Bigtable and the NoSQL movement: a feature-lean database that performed and scaled.

But as Google grew, and entered other lines of business, our priorities evolved. We needed to simplify the jobs of our ever-larger engineering staff and support business-critical services like the AdWords and Payments platforms, as well as then-new consumer products like Gmail. To do that, we knew we had to give our product teams database features like ACID transactions, multi-region consistency, schema-declared invariants, SQL, and more — without compromising on cost or scale. At the time it was an outrageous goal, but with our experience building Bigtable and Megastore, as well as new innovations like TrueTime, we knew it could be done – and it was clear talking to our internal customers that it needed to be done.

Like most good software, our plans for Spanner really came together when we found our first serious customer: AdWords. At the time, Google’s entire advertiser-facing interface was sharded across MySQL instances on specially acquired hardware. Ads had just completed a very challenging resharding project to accommodate Google’s growing business, and was resolute in its desire to never do it again. So the Spanner team and Ads partnered on a project, called F1, to transparently move the entire AdWords stack from MySQL to Spanner.

By 2012, when we published the Spanner paper, we had just completed the migration of AdWords frontend traffic to the new Spanner backend, and we knew we had the seed of something really interesting. Since then, we have been rapidly improving Spanner in every dimension. Google, like most businesses, needs databases that are fully managed, easy to deploy, highly available, and robust in the face of any workload. Google services set a high bar: Spanner stores a copy of the Internet (actually several, at different stages of the indexing pipeline) and is the bedrock of availability and durability for billions of users that depend on Google in their daily lives.

Then, in 2017, Google Cloud launched Cloud Spanner, a fully managed database service that brought the unique capabilities of Spanner to every organization, allowing them to deliver always-on experiences at any scale from thousands to millions of active users across the world without sacrificing consistency.

That took us down another road: innovating to make it easy for all enterprises to build data-driven applications using Cloud Spanner. This meant adding enterprise features such as CMEK and access approval, fine-grained access control, backup and restore, and point-in-time recovery (PITR). We also added VPC Service Controls support and compliance certifications and necessary approvals so that Spanner can be used for workloads requiring ISO 27001, 27017, 27018, PCI DSS, SOC1|2|3, HIPAA and FedRamp. More recently, we added granular instance sizing, a PostgreSQL interface, and free trial instances to lower the barrier to entry for Spanner, and make it accessible for any developer and for any application, big or small. 

Speaking of small, one of the interesting things we discovered along the way is that Spanner isn’t only a great fit for huge mission-critical applications – we also use it internally for smaller applications and internal tools, secure in the knowledge that the data is safe, available, and nowhere near any limits. That’s the thing about infrastructure innovations: at their best, they aren’t only about scale, they’re about extracting complexity from applications and presenting the solution in an easy-to-use package. 

As we have incorporated solutions to those problems into Spanner, customers have dramatically simplified their applications. For example, application teams often created a dedicated “storage” tier to provide higher level features like indexes, transactions, or data synchronization needed to support their business logic. Spanner enables application teams to hollow out or even completely eliminate these tiers for significant cost and maintenance savings. We have also seen big reliability dividends: rather than each application discovering the corner cases of transactions, consistency or high availability architectures one at a time, Spanner’s customers are all able to benefit from a rigorously designed and battle-tested implementation out of the gate.

The tradeoff between the scale limitations of relational databases and the functionality limitations of NoSQL systems is really familiar, and the software world has learned to build great applications within those confines. But we never tire of seeing great applications transform as Spanner melts the distinction away.

Get started with Spanner 

It has never been easier to try out Spanner. With a free trial instance, you can try out Spanner at no cost for 90 days. You can even prototype an entire application for free on Google Cloud by using the Spanner free trial along with the free tier offered by other Google Cloud products such as Compute Engine and BigQuery.

• Create a 90-day Spanner free trial instance. Try Spanner free. 

• Take a deep dive into the new trial experience and learn more about Spanner.