Maximum row size for streaming inserts limit error

Problem

While trying to store streaming data from multiple pub/sub subscriptions data in BigQuery, job fails and you get the following error:

Maximum allowed row size exceeded. Table: <dataset_name>. Allowed: 5242880 Row size: 6517236

 

Environment

  • BigQuery

Solution

  1. Split the message into messages less than 5 MB and issue two inserts.

Note: If the limit is increased to 10 MB, a 10 MB request for BigQuery does not constitute 10 MB of data. This is the size of the whole request with any overhead it might include. The data size larger than 8 MB might result in a request larger than 10 MB.

Cause

BigQuery's row size limit is 5 MB for streaming inserts.