Hacker News Data

How to query public data sets using BigQuery

BigQuery is a fully managed data warehouse and analytics platform. Public datasets are available for you to analyze using SQL queries. You can access BigQuery public data sets using the web UI, the command-line tool, or by making calls to the BigQuery REST API using a variety of client libraries such as Java, .NET, or Python.

Currently, BigQuery public datasets are stored in the US multi-region location. When you query a public dataset, supply the --location=US flag on the command line, choose US as the processing location in the BigQuery web UI, or specify the location property in the jobReference section of the job resource when you use the API. Because the public datasets are stored in the US, you cannot write public data query results to a table in another region, and you cannot join tables in public datasets with tables in another region.

To get started using a BigQuery public dataset, create or select a project. The first terabyte of data processed per month is free, so you can start querying public datasets without enabling billing. If you intend to go beyond the free tier, you should also enable billing.

  1. Sign in to your Google Account.

    If you don't already have one, sign up for a new account.

  2. Select or create a GCP project.

    Go to the Manage resources page

  3. Assurez-vous que la facturation est activée pour votre projet.

    En savoir plus sur l'activation de la facturation

  4. BigQuery is automatically enabled in new projects. To activate BigQuery in a pre-existing project, Enable the BigQuery API.

    Enable the API

Dataset overview

This dataset contains all stories and comments from Hacker News from its launch in 2006. Each story contains a story ID, the author that made the post, when it was written, and the number of points the story received.

You can start exploring this data in the BigQuery console:

Go to the Hacker News Dataset

Sample queries

Here are some examples of SQL queries you can run on this data in BigQuery.

These samples use BigQuery’s legacy SQL by setting the #legacySQL prefix. For more information, see Setting a query prefix.

How are Hacker News story points distributed?

If you use the score as a dimension (group by score, in SQL) and count the number of posts with each score, you can get an idea about how likely a story is to get a given score.

Where do the stories live?

By parsing out the host from the URL you can see where Hacker News stories originate.

About the data

Dataset Source: Hacker News

Category: Media, Social

Use: This dataset is publicly available for anyone to use under the following terms provided by the Dataset Source — https://github.com/HackerNews/API — and is provided "AS IS" without any warranty, express or implied, from Google. Google disclaims all liability for any damages, direct or indirect, resulting from the use of the dataset.

View in BigQuery: Go to the Hacker News dataset

Cette page vous a-t-elle été utile ? Évaluez-la :

Envoyer des commentaires concernant…