Project exceeds quota getting BigQuery data using API

Problem

When using BigQuery REST API you get the following message: Exceeded rate limits: Your project:123456 exceeded quota for tabledata.list bytes per second per project.

Environment

  • BigQuery tables
  • REST API

Solution

  1. Use the BigQuery Storage API, which is designed to be an improved alternative to the record-based paginated access used by tabledata.list, it should provide a significant performance boost over the tabledata.list API and it is not subject to quota limitations on bytes per second. However, if it is not feasible, users would need to implement a retry strategy for tabledata.list requests.

Cause

The users are using tabledata.list API method which has the per-minute limit on the bytes read (currently of 3.6 GB). This limit cannot be adjusted.