Expanding our Public Datasets for geospatial and ML-based analytics

The Google Cloud Public Datasets Program, launched in 2016, works with public data providers to store copies of high-value, high-demand public datasets in Google Cloud to make them more accessible and discoverable. We currently host more than 100 of these public datasets in BigQuery. They stretch across a wide variety of domains to create the only ML-ready public dataset program. Additionally, we have made more than 3 petabytes (PBs) of data, such as Landsat data from the United States Geological Survey (USGS), available in Cloud Storage so users can access and analyze huge volumes of meteorological satellite data faster than ever before.

Public Datasets provide an experimentation toolbox for those new to data analysis, and they offer a powerful repository for skilled researchers and organizations looking to join and augment their own datasets. Users can query up to 1 terabyte (TB) of data per month at no charge, and so far more than 95 petabytes (PBs) of public data have been queried in BigQuery since the program launched.

We have worked hard over the past three years to continually listen to our users’ needs so we can host datasets that help improve their workflows and maximize their ability to realize the value of this data. And we have developed collaborative relationships with data providers, like National Oceanic and Atmospheric Association (NOAA), to ensure that these datasets are stored and described according to appropriate best practices as defined by subject-matter experts. After receiving positive feedback from users, we decided to expand the program.

We also continuing to curate and host datasets in BigQuery so users can leverage BigQuery Machine Learning to analyze data with machine learning using standard SQL queries. We are continually working to minimize the effort required to use geospatial datasets and applying machine learning, so that our users can JOIN their private data and the world’s public data with as little time and effort as possible.

In order to do all this at a bigger scale, we needed more help from the public data providers with whom we are working. And while Google Cloud has made recently significant investments in expanding the Public Dataset Program, we were continually asked one question when we talked to our collaborators: how can we provide our users with a consistent data source or repository?

That is why, in addition to announcing the additional storage volume, we made a second announcement at Google Cloud Next ‘18: this additional storage will be available for the next five years. We believe this is truly an unparalleled commitment to public data. It excites us, our users, and our collaborators because it provides everyone with an assurance that their effort applied to our public datasets will be matched by our commitment going forward. This means that data users can spend their time and focus on analysis instead of searching for data.