The BigQuery Data Transfer Service for Cloud Storage allows you to schedule recurring data loads from Cloud Storage to BigQuery. The Cloud Storage path and the destination table can both be parameterized, allowing you to load data from Cloud Storage buckets organized by date.
Supported file formats
The BigQuery Data Transfer Service currently supports loading data from Cloud Storage in one of the following formats:
- Comma-separated values (CSV)
- JSON (newline-delimited)
Supported compression types
The BigQuery Data Transfer Service for Cloud Storage supports loading compressed data. The compression types supported by BigQuery Data Transfer Service are the same as the compression types supported by BigQuery load jobs. For more information, see Loading compressed and uncompressed data.
Cloud Storage URI
To load data from a Cloud Storage data source, you must provide the Cloud Storage URI.
The Cloud Storage URI comprises your bucket name and your object (filename).
For example, if the Cloud Storage bucket is named
mybucket and the data
file is named
myfile.csv, the bucket URI would be
If your data is separated into multiple files you can use a
wildcard in the URI. For more information, see Cloud Storage
BigQuery Data Transfer Service does not support source URIs that include multiple
consecutive slashes after the initial double slash. Cloud Storage object
names can contain multiple consecutive slash ("/") characters. However,
BigQuery Data Transfer Service converts multiple consecutive slashes into a single
slash. For example, the following source URI, though valid in Cloud Storage,
does not work in BigQuery Data Transfer Service:
To retrieve the Cloud Storage URI:
Open the Cloud Storage console.
Browse to the location of the object (file) that contains the source data.
At the top of the Cloud Storage console, note the path to the object. To compose the URI, replace
gs://bucket/filewith the appropriate path, for example,
gs://mybucket/myfile.json. bucket is the Cloud Storage bucket name and file is the name of the object (file) containing the data.
Wildcard support for Cloud Storage URIs
If your Cloud Storage data is separated into multiple files that share a common base-name, you can use a wildcard in the URI when you load the data.
To add a wildcard to the Cloud Storage URI, you append an asterisk (*) to
the base-name. For example, if you have two files named
fed-sample000002.csv, the bucket URI would be
This wildcard URI can then be used in the web UI or CLI.
You can use only one wildcard for objects (filenames) within your bucket. The wildcard can appear inside the object name or at the end of the object name. Appending a wildcard to the bucket name is unsupported.
When you choose a location for your data, consider the following:
- Colocate your Cloud Storage buckets for loading data.
- If your BigQuery Data Transfer Service dataset is in a multi-regional location, the Cloud Storage bucket containing the data you're loading must be in a regional or multi-regional bucket in the same location. For example, if your BigQuery Data Transfer Service dataset is in the EU, the Cloud Storage bucket must be in a regional or multi-regional bucket in the EU.
- If your dataset is in a regional location, your Cloud Storage bucket must be a regional bucket in the same location. For example, if your dataset is in the Tokyo region, your Cloud Storage bucket must be a regional bucket in Tokyo.
- Exception: If your dataset is in the US multi-regional location, you can load data from a Cloud Storage bucket in any regional or multi-regional location.
- Develop a data management plan.
- If you choose a regional storage resource such as a BigQuery Data Transfer Service dataset or a Cloud Storage bucket, develop a plan for geographically managing your data.
For more information on Cloud Storage locations, see Bucket locations in the Cloud Storage documentation.
To manually move a dataset from one location to another, follow this process:
Export the data from your BigQuery Data Transfer Service tables to a regional or multi-region Cloud Storage bucket in the same location as your dataset. For example, if your dataset is in the EU multi-region location, export your data into a regional or multi-region bucket in the EU.
There are no charges for exporting data from BigQuery Data Transfer Service, but you do incur charges for storing the exported data in Cloud Storage. BigQuery Data Transfer Service exports are subject to the limits on export jobs.
Copy or move the data from your Cloud Storage bucket to a regional or multi-region bucket in the new location. For example, if you are moving your data from the US multi-region location to the Tokyo regional location, you would transfer the data to a regional bucket in Tokyo. For information on transferring Cloud Storage objects, see Renaming, copying, and moving objects in the Cloud Storage documentation.
Note that transferring data between regions incurs network egress charges in Cloud Storage.
After you transfer the data to a Cloud Storage bucket in the new location, create a new BigQuery Data Transfer Service dataset (in the new location). Then, load your data from the Cloud Storage bucket into BigQuery Data Transfer Service.
You are not charged for loading the data into BigQuery Data Transfer Service, but you will incur charges for storing the data in Cloud Storage until you delete the data or the bucket. You are also charged for storing the data in BigQuery Data Transfer Service after it is loaded. Loading data into BigQuery Data Transfer Service is subject to the limits on load jobs.
For more information on using Cloud Storage to store and move large datasets, see Using Cloud Storage with big data.
For information on BigQuery Data Transfer Service pricing, see the Pricing page.
Quotas and limits
The BigQuery Data Transfer Service uses load jobs to load Cloud Storage data into BigQuery. All BigQuery Quotas and limits on load jobs apply to recurring Cloud Storage load jobs.
- Learn about setting up a Cloud Storage transfer
- Learn about Using runtime parameters in Cloud Storage transfers
- Learn more about the BigQuery Data Transfer Service