How to query public data sets using BigQuery
BigQuery is a fully managed data warehouse and analytics platform. Public datasets are available for you to analyze using SQL queries. You can access BigQuery public data sets using the web UI, the command-line tool, or by making calls to the BigQuery REST API using a variety of client libraries such as Java, .NET, or Python.
Currently, BigQuery public datasets are stored in the
location. When you query a public dataset, supply
--location=US flag on the command line, choose
US as the
processing location in the BigQuery web UI, or specify the
location property in the
jobReference section of the job resource
when you use the API. Because the public datasets are stored in the US, you cannot write public
data query results to a table in another region, and you cannot join tables in public datasets
with tables in another region.
To get started using a BigQuery public dataset, create or select a project. The first terabyte of data processed per month is free, so you can start querying public datasets without enabling billing. If you intend to go beyond the free tier, you should also enable billing.
Sign in to your Google Account.
If you don't already have one, sign up for a new account.
Select or create a GCP project.
Make sure that billing is enabled for your project.
- BigQuery is automatically enabled in new projects. To activate BigQuery in a pre-existing project, Enable the BigQuery API.
This public data is published by the US Department of Health and Human Services and includes all weekly surveillance reports of nationally notifiable diseases for all U.S. cities and states published between 1888 and 2013. The data set consists of eight important vaccine-preventable contagious diseases: diphtheria, hepatitis A, measles, mumps, pertussis, polio, rubella and smallpox.
You can start exploring this data in the BigQuery console:
Here are some examples of SQL queries you can run on this data in BigQuery.
These samples use BigQuery’s legacy SQL by setting the
prefix. For more information, see Setting a query prefix.
Diseases by year
#legacySQL SELECT * FROM ( SELECT *, MIN(z___rank) OVER (PARTITION BY cdc_reports_epi_year) AS z___min_rank FROM ( SELECT *, RANK() OVER (PARTITION BY cdc_reports_disease ORDER BY cdc_reports_epi_year ) AS z___rank FROM ( SELECT FLOOR(cdc_reports.epi_week/100) AS cdc_reports_epi_year, cdc_reports.disease AS cdc_reports_disease, COALESCE(CAST(SUM((FLOAT(cdc_reports.cases))) AS FLOAT),0) AS cdc_reports_total_cases FROM [lookerdata:cdc.project_tycho_reports] AS cdc_reports GROUP EACH BY 1, 2) ww ) aa ) xx WHERE z___min_rank <= 500 LIMIT 30000
Comparing Mumps outbreak in California and Connecticut
#legacySQL SELECT * FROM ( SELECT *, MIN(z___rank) OVER (PARTITION BY cdc_reports_epi_week) AS z___min_rank FROM ( SELECT *, RANK() OVER (PARTITION BY cdc_reports_state ORDER BY cdc_reports_epi_week ) AS z___rank FROM ( SELECT cdc_reports.epi_week AS cdc_reports_epi_week, cdc_reports.state AS cdc_reports_state, COALESCE(CAST(SUM((FLOAT(cdc_reports.cases))) AS FLOAT),0) AS cdc_reports_total_cases FROM [lookerdata:cdc.project_tycho_reports] AS cdc_reports WHERE (cdc_reports.disease = 'MUMPS') AND (FLOOR(cdc_reports.epi_week/100) = 1970) AND (cdc_reports.state = 'CA' OR cdc_reports.state = 'CT') GROUP EACH BY 1, 2) ww ) aa ) xx WHERE z___min_rank <= 500 LIMIT 30000
Mumps shows the same seasonal pattern coast to coast, in both California and Connecticut, during the 1970 outbreak.
About the data
Dataset Source: Data.gov
Use: This dataset is publicly available for anyone to use under the following terms provided by the Dataset Source — http://www.data.gov/privacy-policy#data_policy — and is provided "AS IS" without any warranty, express or implied, from Google. Google disclaims all liability for any damages, direct or indirect, resulting from the use of the dataset.
View in BigQuery: Go to the USA Contagious Diseasae Dataset