Loading Avro files from Cloud Storage
Avro is an open source data format that bundles serialized data with the data's schema in the same file.
When you load Avro data from Cloud Storage, you can load the data into a new table or partition, or you can append to or overwrite an existing table or partition. When your data is loaded into BigQuery, it is converted into columnar format for Capacitor (BigQuery's storage format).
When you load data from Cloud Storage into a BigQuery table, the dataset that contains the table must be in the same regional or multi- regional location as the Cloud Storage bucket.
For information about loading Avro data from a local file, see Loading data into BigQuery from a local data source.
Advantages of Avro
Avro is the preferred format for loading data into BigQuery. Loading Avro files has the following advantages over CSV and JSON (newline delimited):
- The Avro binary format:
- Is faster to load. The data can be read in parallel, even if the data blocks are compressed.
- Doesn't require typing or serialization.
- Is easier to parse because there are no encoding issues found in other formats such as ASCII.
- When you load Avro files into BigQuery, the table schema is automatically retrieved from the self-describing source data.
Avro schemas
When you load Avro files into BigQuery, the table schema is automatically retrieved using the source data. When BigQuery retrieves the schema from the source data, the alphabetically last file is used.
For example, you have the following Avro files in Cloud Storage:
gs://mybucket/00/ a.avro z.avro gs://mybucket/01/ b.avro
Running this command in the bq
command-line tool loads all of the files (as a
comma-separated list), and the schema is derived from mybucket/01/b.avro
:
bq load \ --source_format=AVRO \ dataset.table \ "gs://mybucket/00/*.avro","gs://mybucket/01/*.avro"
When importing multiple Avro files with different Avro schemas, all schemas must be compatible with Avro's schema resolution.
When BigQuery detects the schema, some Avro data types are converted to BigQuery data types to make them compatible with BigQuery SQL syntax. For more information, see Avro conversions.
Avro compression
BigQuery supports the following compression codecs for data blocks in Avro files:
Snappy
DEFLATE
Required permissions
When you load data into BigQuery, you need permissions to run a load job and permissions that let you load data into new or existing BigQuery tables and partitions. If you are loading data from Cloud Storage, you also need permissions to access to the bucket that contains your data.
BigQuery permissions
At a minimum, the following permissions are required to load data into BigQuery. These permissions are required if you are loading data into a new table or partition, or if you are appending or overwriting a table or partition.
bigquery.tables.create
bigquery.tables.updateData
bigquery.jobs.create
The following predefined IAM roles include both
bigquery.tables.create
and bigquery.tables.updateData
permissions:
bigquery.dataEditor
bigquery.dataOwner
bigquery.admin
The following predefined IAM roles include bigquery.jobs.create
permissions:
bigquery.user
bigquery.jobUser
bigquery.admin
In addition, if a user has bigquery.datasets.create
permissions, when that
user creates a dataset, they are granted bigquery.dataOwner
access to it.
bigquery.dataOwner
access lets the user create and
update tables in the dataset by using a load job.
For more information on IAM roles and permissions in BigQuery, see Access control.
Cloud Storage permissions
To load data from a Cloud Storage bucket, you must be granted
storage.objects.get
permissions. If you are using a URI wildcard,
you must also have storage.objects.list
permissions.
The predefined IAM role storage.objectViewer
can be granted to provide both storage.objects.get
and storage.objects.list
permissions.
Loading Avro data into a new table
You can load Avro data into a new table in the following ways:
- Using the Cloud Console.
- Using the
bq load
command in thebq
command-line tool. - Calling the
jobs.insert
API method and configuring aload
job. - Using the client libraries.
To load Avro data from Cloud Storage into a new BigQuery table:
Console
Open the BigQuery page in the Cloud Console.
In the Explorer panel, expand your project and select a dataset.
In the details panel, click Create table.
On the Create table page, in the Source section:
For Create table from, select Cloud Storage.
In the source field, browse to or enter the Cloud Storage URI. Note that you cannot include multiple URIs in the Cloud Console, but wildcards are supported. The Cloud Storage bucket must be in the same location as the dataset that contains the table you're creating.
For File format, select Avro.
On the Create table page, in the Destination section:
For Dataset name, choose the appropriate dataset.
Verify that Table type is set to Native table.
In the Table name field, enter the name of the table you're creating in BigQuery.
In the Schema section, no action is necessary. The schema is self-described in Avro files.
(Optional) To partition the table, choose your options in the Partition and cluster settings:
- To create a partitioned table,
click No partitioning, select Partition by field and choose a
DATE
orTIMESTAMP
column. This option is unavailable if your schema does not include aDATE
orTIMESTAMP
column. - To create an ingestion-time partitioned table, click No partitioning and select Partition by ingestion time.
- To create a partitioned table,
click No partitioning, select Partition by field and choose a
(Optional) For Partitioning filter, click the Require partition filter box to require users to include a
WHERE
clause that specifies the partitions to query. Requiring a partition filter may reduce cost and improve performance. For more information, see Querying partitioned tables. This option is unavailable if No partitioning is selected.(Optional) To cluster the table, in the Clustering order box, enter between one and four field names.
(Optional) Click Advanced options.
- For Write preference, leave Write if empty selected. This option creates a new table and loads your data into it.
- For Number of errors allowed, accept the default value of
0
or enter the maximum number of rows containing errors that can be ignored. If the number of rows with errors exceeds this value, the job will result in aninvalid
message and fail. - For Unknown values, leave Ignore unknown values unchecked. This option applies only to CSV and JSON files.
- For Encryption, click Customer-managed key to use a Cloud Key Management Service key. If you leave the Google-managed key setting, BigQuery encrypts the data at rest.
Click Create table.
bq
Use the bq load
command, specify AVRO
using the --source_format
flag, and include a Cloud Storage URI.
You can include a single URI, a comma-separated list of URIs, or a URI
containing a wildcard.
(Optional) Supply the --location
flag and set the value to your
location.
Other optional flags include:
--time_partitioning_type
: Enables time-based partitioning on a table and sets the partition type. Possible values areHOUR
,DAY
,MONTH
, andYEAR
. This flag is optional when you create a table partitioned on aDATE
,DATETIME
, orTIMESTAMP
column. The default partition type for time-based partitioning isDAY
.--time_partitioning_expiration
: An integer that specifies (in seconds) when a time-based partition should be deleted. The expiration time evaluates to the partition's UTC date plus the integer value.--time_partitioning_field
: TheDATE
orTIMESTAMP
column used to create a partitioned table. If time-based partitioning is enabled without this value, an ingestion-time partitioned table is created.--require_partition_filter
: When enabled, this option requires users to include aWHERE
clause that specifies the partitions to query. Requiring a partition filter may reduce cost and improve performance. For more information, see Querying partitioned tables.--clustering_fields
: A comma-separated list of up to four column names used to create a clustered table.--destination_kms_key
: The Cloud KMS key for encryption of the table data.For more information on partitioned tables, see:
For more information on clustered tables, see:
For more information on table encryption, see:
To load Avro data into BigQuery, enter the following command:
bq --location=location load \ --source_format=format \ dataset.table \ path_to_source
Replace the following:
- location is your location. The
--location
flag is optional. For example, if you are using BigQuery in the Tokyo region, you can set the flag's value toasia-northeast1
. You can set a default value for the location using the .bigqueryrc file. - format is
AVRO
. - dataset is an existing dataset.
- table is the name of the table into which you're loading data.
- path_to_source is a fully-qualified Cloud Storage URI or a comma-separated list of URIs. Wildcards are also supported.
Examples:
The following command loads data from gs://mybucket/mydata.avro
into a
table named mytable
in mydataset
.
bq load \
--source_format=AVRO \
mydataset.mytable \
gs://mybucket/mydata.avro
The following command loads data from gs://mybucket/mydata.avro
into an
ingestion-time partitioned table named mytable
in mydataset
.
bq load \
--source_format=AVRO \
--time_partitioning_type=DAY \
mydataset.mytable \
gs://mybucket/mydata.avro
The following command loads data from gs://mybucket/mydata.avro
into a
partitioned table named mytable
in mydataset
. The table is partitioned
on the mytimestamp
column.
bq load \
--source_format=AVRO \
--time_partitioning_field mytimestamp \
mydataset.mytable \
gs://mybucket/mydata.avro
The following command loads data from multiple files in gs://mybucket/
into a table named mytable
in mydataset
. The Cloud Storage URI uses a
wildcard.
bq load \
--source_format=AVRO \
mydataset.mytable \
gs://mybucket/mydata*.avro
The following command loads data from multiple files in gs://mybucket/
into a table named mytable
in mydataset
. The command includes a comma-
separated list of Cloud Storage URIs with wildcards.
bq load \
--source_format=AVRO \
mydataset.mytable \
"gs://mybucket/00/*.avro","gs://mybucket/01/*.avro"
API
Create a
load
job that points to the source data in Cloud Storage.(Optional) Specify your location in the
location
property in thejobReference
section of the job resource.The
source URIs
property must be fully-qualified, in the formatgs://bucket/object
. Each URI can contain one '*' wildcard character.Specify the Avro data format by setting the
sourceFormat
property toAVRO
.To check the job status, call
jobs.get(job_id*)
, where job_id is the ID of the job returned by the initial request.- If
status.state = DONE
, the job completed successfully. - If the
status.errorResult
property is present, the request failed, and that object will include information describing what went wrong. When a request fails, no table is created and no data is loaded. - If
status.errorResult
is absent, the job finished successfully, although there might have been some non-fatal errors, such as problems importing a few rows. Non-fatal errors are listed in the returned job object'sstatus.errors
property.
- If
API notes:
Load jobs are atomic and consistent; if a load job fails, none of the data is available, and if a load job succeeds, all of the data is available.
As a best practice, generate a unique ID and pass it as
jobReference.jobId
when callingjobs.insert
to create a load job. This approach is more robust to network failure because the client can poll or retry on the known job ID.Calling
jobs.insert
on a given job ID is idempotent. You can retry as many times as you like on the same job ID, and at most one of those operations will succeed.
Java
Node.js
Before trying this sample, follow the Node.js setup instructions in the BigQuery Quickstart Using Client Libraries. For more information, see the BigQuery Node.js API reference documentation.
Python
Before trying this sample, follow the Python setup instructions in the BigQuery Quickstart Using Client Libraries. For more information, see the BigQuery Python API reference documentation.
Appending to or overwriting a table with Avro data
You can load additional data into a table either from source files or by appending query results.
In the Cloud Console, use the Write preference option to specify what action to take when you load data from a source file or from a query result.
You have the following options when you load additional data into a table:
Console option | bq tool flag |
BigQuery API property | Description |
---|---|---|---|
Write if empty | None | WRITE_EMPTY |
Writes the data only if the table is empty. |
Append to table | --noreplace or --replace=false ; if
--[no]replace is unspecified, the default is append |
WRITE_APPEND |
(Default) Appends the data to the end of the table. |
Overwrite table | --replace or --replace=true |
WRITE_TRUNCATE |
Erases all existing data in a table before writing the new data. This action also deletes the table schema and removes any Cloud KMS key. |
If you load data into an existing table, the load job can append the data or overwrite the table.
You can append or overwrite a table in the following ways:
- Using the Cloud Console.
- Using the
bq load
command in thebq
command-line tool. - Calling the
jobs.insert
API method and configuring aload
job. - Using the client libraries.
To append or overwrite a table with Avro data:
Console
Open the BigQuery page in the Cloud Console.
In the Explorer panel, expand your project and select a dataset.
In the details panel, click Create table.
On the Create table page, in the Source section:
For Create table from, select Cloud Storage.
In the source field, browse to or enter the Cloud Storage URI. Note that you cannot include multiple URIs in the Cloud Console, but wildcards are supported. The Cloud Storage bucket must be in the same location as the dataset that contains the table you're appending or overwriting.
For File format, select Avro.
On the Create table page, in the Destination section:
For Dataset name, choose the appropriate dataset.
In the Table name field, enter the name of the table you're appending or overwriting in BigQuery.
Verify that Table type is set to Native table.
In the Schema section, no action is necessary. The schema is self-described in Avro files.
For Partition and cluster settings, leave the default values. You cannot convert a table to a partitioned or clustered table by appending or overwriting it, and the Cloud Console does not support appending to or overwriting partitioned or clustered tables in a load job.
Click Advanced options.
- For Write preference, choose Append to table or Overwrite table.
- For Number of errors allowed, accept the default value of
0
or enter the maximum number of rows containing errors that can be ignored. If the number of rows with errors exceeds this value, the job will result in aninvalid
message and fail. - For Unknown values, leave Ignore unknown values unchecked. This option applies only to CSV and JSON files.
For Encryption, click Customer-managed key to use a Cloud Key Management Service key. If you leave the Google-managed key setting, BigQuery encrypts the data at rest.
Click Create table.
bq
Enter the bq load
command with the --replace
flag to overwrite the
table. Use the --noreplace
flag to append data to the table. If no flag is
specified, the default is to append data. Supply the --source_format
flag
and set it to AVRO
. Because Avro schemas are automatically retrieved
from the self-describing source data, you do not need to provide a schema
definition.
(Optional) Supply the --location
flag and set the value to your
location.
Other optional flags include:
--destination_kms_key
: The Cloud KMS key for encryption of the table data.
bq --location=location load \ --[no]replace \ --source_format=format \ dataset.table \ path_to_source
Replace the following:
- location is your location.
The
--location
flag is optional. You can set a default value for the location by using the .bigqueryrc file. - format is
AVRO
. - dataset is an existing dataset.
- table is the name of the table into which you're loading data.
- path_to_source is a fully-qualified Cloud Storage URI or a comma-separated list of URIs. Wildcards are also supported.
Examples:
The following command loads data from gs://mybucket/mydata.avro
and
overwrites a table named mytable
in mydataset
.
bq load \
--replace \
--source_format=AVRO \
mydataset.mytable \
gs://mybucket/mydata.avro
The following command loads data from gs://mybucket/mydata.avro
and
appends data to a table named mytable
in mydataset
.
bq load \
--noreplace \
--source_format=AVRO \
mydataset.mytable \
gs://mybucket/mydata.avro
For information on appending and overwriting partitioned tables using the
bq
command-line tool, see:
Appending to and overwriting partitioned table data.
API
Create a
load
job that points to the source data in Cloud Storage.(Optional) Specify your location in the
location
property in thejobReference
section of the job resource.The
source URIs
property must be fully-qualified, in the formatgs://bucket/object
. You can include multiple URIs as a comma-separated list. Note that wildcards are also supported.Specify the data format by setting the
configuration.load.sourceFormat
property toAVRO
.Specify the write preference by setting the
configuration.load.writeDisposition
property toWRITE_TRUNCATE
orWRITE_APPEND
.
Java
Node.js
Before trying this sample, follow the Node.js setup instructions in the BigQuery Quickstart Using Client Libraries. For more information, see the BigQuery Node.js API reference documentation.
Python
Before trying this sample, follow the Python setup instructions in the BigQuery Quickstart Using Client Libraries. For more information, see the BigQuery Python API reference documentation.
Loading hive-partitioned Avro data
BigQuery supports loading hive partitioned Avro data stored on Cloud Storage and will populate the hive partitioning columns as columns in the destination BigQuery managed table. For more information, see Loading Externally Partitioned Data from Cloud Storage.
Avro conversions
BigQuery converts Avro data types to the following BigQuery data types:
Primitive types
BigQuery data type | Notes | |
---|---|---|
null | BigQuery ignores these values | |
boolean | BOOLEAN | |
int | INTEGER | |
long | INTEGER | |
float | FLOAT | |
double | FLOAT | |
bytes | BYTES | |
string | STRING | UTF-8 only |
Logical types
By default, BigQuery ignores the logicalType
attribute for most
of the types and uses the underlying Avro type instead. To convert
Avro logical types to their corresponding BigQuery data types,
set the --use_avro_logical_types
flag to true
using
the bq
command-line tool, or set the useAvroLogicalTypes
property in the
job resource
when you call the
jobs.insert
method to create a load job.
The table below shows the conversion of Avro logical types to BigQuery data types.
BigQuery data type: Logical type disabled | BigQuery data type: Logical type enabled | |
---|---|---|
date | INTEGER | DATE |
time-millis | INTEGER | TIME |
time-micros | INTEGER (converted from LONG) | TIME |
timestamp-millis | INTEGER (converted from LONG) | TIMESTAMP |
timestamp-micros | INTEGER (converted from LONG) | TIMESTAMP |
duration | BYTES (converted from fixed type of size 12) |
BYTES (converted from fixed type of size 12) |
decimal | NUMERIC, BIGNUMERIC (Preview), or STRING (see Decimal logical type) | NUMERIC, BIGNUMERIC (Preview), or STRING (see Decimal logical type) |
For more information on Avro data types, see the Apache Avro™ 1.8.2 Specification.
Decimal logical type
Decimal
logical types can be converted to NUMERIC
, BIGNUMERIC
(Preview), or STRING
types. The converted type depends
on the precision and scale parameters of the decimal
logical type and the
specified decimal target types. Specify the decimal target type as follows:
- For a load job using the
jobs.insert
API: use theJobConfigurationLoad.decimalTargetTypes
field. - For a load job using the
bq load
command in thebq
command-line tool: use the--decimal_target_types
flag. - For a query against a table with external sources:
use the
ExternalDataConfiguration.decimalTargetTypes
field. - For a persistent external table created with DDL:
use the
decimal_target_types
option.
For backward compatibility, if the decimal target types are not specified, you can
load an Avro file containing a bytes
column with
the decimal
logical type into a BYTES
column of an existing table. In this
case, the decimal
logical type on the column in the Avro file is ignored. This
conversion mode is deprecated and might be removed in the future.
For more information on the Avro decimal
logical type, see the
Apache Avro™ 1.8.2 Specification.
Complex types
BigQuery data type | Notes | |
---|---|---|
record | RECORD |
|
enum | STRING |
|
array | repeated fields | Arrays of arrays are not supported. Arrays containing only NULL types are ignored. |
map<T> | RECORD | BigQuery converts an Avro map<T> field to a repeated RECORD that contains two fields: a key and a value. BigQuery stores the key as a STRING, and converts the value to its corresponding data type in BigQuery. |
union |
|
|
fixed | BYTES |
|