This page documents production updates to BigQuery. We recommend that BigQuery developers periodically check this list for any new announcements. BigQuery automatically updates to the latest release and cannot be downgraded to a previous version.
You can see the latest product updates for all of Google Cloud on the Google Cloud page, browse and filter all release notes in the Google Cloud console, or you can programmatically access release notes in BigQuery.
To get the latest product updates delivered to you, add the URL of this page to your
feed
reader, or add the feed URL directly: https://cloud.google.com/feeds/bigquery-release-notes.xml
March 22, 2023
BigQuery now supports Unicode column naming using international character sets, alphanumeric and special characters. Existing columns can use these new capabilities using the RENAME
command. This feature is now in preview.
March 20, 2023
A weekly digest of client library updates from across the Cloud SDK.
Go
Changes for bigquery/storage/apiv1beta1
1.49.0 (2023-03-16)
Features
Python
Changes for google-cloud-bigquery
3.7.0 (2023-03-06)
Features
- Add
connection_properties
andcreate_session
toLoadJobConfig
(#1509) (cd0aaa1) - Add default_query_job_config property and property setter to BigQuery client (#1511) (a23092c)
Documentation
March 14, 2023
The Lineage tab in the table properties page lets you track how your data moves and transforms through BigQuery. This feature is now generally available (GA).
March 13, 2023
A weekly digest of client library updates from across the Cloud SDK.
You can now specify translation configurations in the BigQuery Interactive SQL Translator and use it to debug Batch SQL translator jobs. This feature is now in preview.
March 10, 2023
The CREATE TABLE AS SELECT
statement now lets you filter data from files in Amazon S3 and Azure Blob Storage before transferring results into BigQuery tables This feature is in preview.
March 07, 2023
Case-insensitive collation support is now generally available (GA). In addition to features available in the preview, the GA release includes:
- MIN, MAX, COUNT with DISTINCT, and PERCENTILE_DISC windows functions
- ORDER BY and PARTITION BY in the WINDOWS clause
- LIKE operator with limitations
- Views
- Materialized views with limitations
- Table functions with limitations
- BigQuery BI engine
March 06, 2023
A weekly digest of client library updates from across the Cloud SDK.
Go
Changes for bigquery/storage/apiv1beta1
1.48.0 (2023-03-01)
Features
Java
Changes for google-cloud-bigquery
2.23.1 (2023-03-02)
Dependencies
March 02, 2023
The WITH RECURSIVE
clause is now generally available (GA). This clause lets you include one or more recursive common table expressions (CTEs) in a query.
February 27, 2023
A weekly digest of client library updates from across the Cloud SDK.
Go
Changes for bigquery/storage/apiv1beta1
1.47.0 (2023-02-22)
Features
- bigquery/storage: Add default_value_expression to TableFieldSchema (#7400) (1244b3f)
- bigquery: Add support for session in load jobs (#7418) (f9ff2ca)
Bug Fixes
Java
Changes for google-cloud-bigquery
2.23.0 (2023-02-22)
Features
Dependencies
- Update cloud client dependencies (#2526) (4d88ccc)
- Update dependency com.google.api.grpc:proto-google-cloud-bigqueryconnection-v1 to v2.13.0 (#2533) (ed2cb74)
- Update dependency com.google.apis:google-api-services-bigquery to v2-rev20230210-2.0.0 (#2530) (62ff092)
- Update dependency com.google.cloud:google-cloud-shared-dependencies to v3.3.0 (#2534) (f1bcc33)
- Update dependency org.graalvm.buildtools:junit-platform-native to v0.9.20 (#2527) (5fe5e74)
- Update dependency org.graalvm.buildtools:native-maven-plugin to v0.9.20 (#2528) (554e75d)
Python
Changes for google-cloud-bigquery
3.6.0 (2023-02-22)
Features
Bug Fixes
- Annotate optional integer parameters with optional type (#1487) (a190aaa)
- Loosen ipywidget dependency (#1504) (20d3276)
- Removes scope to avoid unnecessary duplication (#1503) (665d7ba)
Dependencies
- Update minimum google-cloud-core to 1.6.0 (a190aaa)
You can set default values on columns in your BigQuery tables. This feature is now generally available (GA).
February 23, 2023
Authorized stored procedures are now in preview. This feature lets you share stored procedures with users or groups without giving them direct access to the underlying tables.
February 22, 2023
Fixed linked datasets querying shared dataset that has data ingested through streaming inserts or the BigQuery Storage Write API.
Primary and foreign key table constraints are now available in preview. You can define table constraints using the CREATE TABLE
statement, the ALTER TABLE ADD PRIMARY KEY
statement, or the ALTER TABLE ADD CONSTRAINT
statement.
February 21, 2023
The ALTER TABLE RENAME COLUMN statement and the ALTER TABLE DROP COLUMN statement are now generally available (GA).
February 15, 2023
You can now make a dataset and the tables in that dataset case-insensitive when you create a dataset or alter a dataset. This feature is generally available (GA).
You can now run bq
commands using service account impersonation. This feature is generally available (GA).
In the Explorer pane, the resource corresponding to the focused tab is now selected. This feature is generally available (GA).
In the Explorer pane, you can now see all the resources in the searched resource's level by clicking Show more. This feature is generally available (GA).
February 13, 2023
A weekly digest of client library updates from across the Cloud SDK.
Go
Changes for bigquery/storage/apiv1beta1
1.46.0 (2023-02-06)
Features
- bigquery: Add dataset/table collation (#7235) (9f7bbeb)
- bigquery: Use storage api for query jobs (#6822) (26c04f4)
Bug Fixes
Java
Changes for google-cloud-bigquery
2.22.0 (2023-02-08)
Features
Dependencies
- Update arrow.version to v11 (major) (#2495) (94ed060)
- Update dependency com.google.api.grpc:proto-google-cloud-bigqueryconnection-v1 to v2.11.0 (#2482) (e6ffb9b)
- Update dependency com.google.api.grpc:proto-google-cloud-bigqueryconnection-v1 to v2.12.0 (#2512) (09f280d)
- Update dependency com.google.cloud:google-cloud-bigquerystorage-bom to v2.31.0 (#2499) (c0a393c)
- Update dependency com.google.cloud:google-cloud-datacatalog-bom to v1.15.0 (#2483) (5c2bf69)
- Update dependency com.google.cloud:google-cloud-shared-dependencies to v3.2.0 (#2513) (02832dd)
You can now create materialized views over BigLake metadata cache-enabled tables to reference structured data stored in Cloud Storage. This feature is in preview.
February 09, 2023
You can now apply four new types of dynamic data masking to table columns in BigQuery. These new data masking types include date year, email, first four characters, and last four characters masks. This feature is generally available (GA).
Autoscaling slot reservations are now available in preview. You can create autoscaling reservations and associated commitments using slots autoscaling.
Cloud console updates: In the Explorer pane, you can now refresh the contents of a resource (project or dataset). To refresh the contents of a resource, click
View actions, and then click Refresh contents.February 06, 2023
The HAVING MAX
and HAVING MIN
clause for the ANY_VALUE
function is now in preview.
You can now view information related to query processing to monitor and optimize queries with the query_info
column in INFORMATION_SCHEMA.JOBS
, JOBS_BY_FOLDER
and JOBS_BY_ORGANIZATION
views. This feature is generally available (GA).
A weekly digest of client library updates from across the Cloud SDK.
February 01, 2023
The BigQuery Data Transfer Service can now transfer data from Azure Blob Storage into BigQuery. This feature is now in preview.
January 31, 2023
Azure workload identity federation is now generally available (GA) for BigQuery Omni connections. You can now create a connection for federated identity using Google Cloud console.
Cloud console updates: When you create datasets, select locations to run specific queries, or create exchanges in Analytics Hub, you now see separate options for multi-region and specific regions. Based on your selection, you see a list with more options.
January 30, 2023
You can search for BigQuery partners in the BigQuery Partner Center. This feature is in Preview.
A weekly digest of client library updates from across the Cloud SDK.
Node.js
Changes for @google-cloud/bigquery
6.1.0 (2023-01-25)
Features
Java
Changes for google-cloud-bigquery
2.21.0 (2023-01-23)
Features
Bug Fixes
Dependencies
- Update com.google.cloud:google-cloud-bigquerystorage-bom to v2.28.3 (5a20c32)
- Update com.google.cloud:google-cloud-datacatalog-bom to v1.14.0 (606fc1e)
- Update com.google.cloud:google-cloud-storage to v2.17.1 (#2465) (606fc1e)
- Update com.google.cloud:google-cloud-storage to v2.17.2 (#2479) (5a20c32)
- Update dependency com.google.api.grpc:proto-google-cloud-bigqueryconnection-v1 to v2.10.0 (#2463) (5bfa7ae)
- Update dependency com.google.apis:google-api-services-bigquery to v2-rev20230114-2.0.0 (#2477) (b6409d5)
- Update dependency com.google.cloud:google-cloud-shared-dependencies to v3.1.2 (#2476) (688b6a6)
January 23, 2023
A weekly digest of client library updates from across the Cloud SDK.
Python
Changes for google-cloud-bigquery
3.4.2 (2023-01-13)
Bug Fixes
Dependencies
Documentation
- Create sample to write schema file from table (#1439) (093cc68)
- Created samples for load table and create table from schema file (#1436) (8ad2e5b)
- Revise create table cmek sample (#1452) (57740e4)
- Revise get table labels code sample, add TODO to clean up snipp… (#1464) (b5ccbfe)
- Revise label table code samples (#1451) (14ae1f2)
- Revise sample for nested schema (#1446) (a097631)
January 19, 2023
You can now attach Resource Manager tags to datasets, which let you conditionally apply Identity and Access Management (IAM) policies to your resources. This feature is generally available (GA).
January 16, 2023
A weekly digest of client library updates from across the Cloud SDK.
Java
Changes for google-cloud-bigquery
2.20.2 (2023-01-12)
Dependencies
- Update com.google.cloud:google-cloud-bigquerystorage-bom to 2.28.1 (b62391d)
- Update dependency com.google.apis:google-api-services-bigquery to v2-rev20221209-2.0.0 (#2449) (9d0a107)
- Update dependency com.google.apis:google-api-services-bigquery to v2-rev20221217-2.0.0 (#2459) (08e2927)
- Update dependency com.google.cloud:google-cloud-shared-dependencies to v3.1.1 (#2455) (412710c)
- Update dependency org.junit.vintage:junit-vintage-engine to v5.9.2 (#2456) (04d47ad)
- Update dependency org.threeten:threeten-extra to v1.7.2 (#2450) (485be56)
January 10, 2023
The ALTER CAPACITY SET OPTIONS
statement and ALTER RESERVATION SET OPTIONS
statement are now generally available (GA). Additionally, the CREATE CAPACITY
, CREATE RESERVATION
, and CREATE ASSIGNMENT
statements now support the OPTIONS
clause.
January 09, 2023
The following generally available (GA) features have been added for sessions:
In a session, temporary functions are now maintained until the session ends.
In a session, statements that include the
TEMP
keyword can also include theOR REPLACE
andIF NOT EXISTS
keywords.
A weekly digest of client library updates from across the Cloud SDK.
Go
Changes for bigquery/storage/apiv1beta1
1.45.0 (2023-01-05)
Features
Java
Changes for google-cloud-bigquery
2.20.1 (2023-01-04)
Dependencies
December 22, 2022
The Lineage tab in the table properties page lets you track how your data moves and transforms through BigQuery. This feature is now in preview.
BigQuery now blocks saving query results to Google Drive from projects inside a VPC Service Controls protected perimeter.
December 19, 2022
A weekly digest of client library updates from across the Cloud SDK.
Java
Changes for google-cloud-bigquery
2.20.0 (2022-12-13)
Features
- Add fast query path support when empty jobId object is passed (#2349) (42c083a)
- Next release from main branch is 2.20.0 (#2405) (9297a43)
Dependencies
- Update arrow.version to v10.0.1 (#2426) (aff9019)
- Update cloud client dependencies (#2444) (7255357)
- Update dependency com.google.api.grpc:proto-google-cloud-bigqueryconnection-v1 to v2.8.0 (#2418) (1ac1653)
- Update dependency com.google.api.grpc:proto-google-cloud-bigqueryconnection-v1 to v2.9.0 (#2441) (01cc3c3)
- Update dependency com.google.apis:google-api-services-bigquery to v2-rev20221127-2.0.0 (#2437) (eb52002)
- Update dependency com.google.cloud:google-cloud-datacatalog-bom to v1.12.0 (#2419) (c449031)
- Update dependency com.google.cloud:google-cloud-shared-dependencies to v3.1.0 (#2435) (c99b215)
- Update dependency com.google.cloud:google-cloud-storage to v2.15.1 (#2420) (baf337a)
- Update dependency org.graalvm.buildtools:junit-platform-native to v0.9.18 (#2424) (63b5196)
- Update dependency org.graalvm.buildtools:junit-platform-native to v0.9.19 (#2432) (396c6dc)
- Update dependency org.graalvm.buildtools:native-maven-plugin to v0.9.18 (#2425) (cd2ae9f)
- Update dependency org.graalvm.buildtools:native-maven-plugin to v0.9.19 (#2433) (bf94087)
- Update jmh.version to v1.36 (#2415) (0676586)
Python
Changes for google-cloud-bigquery
3.4.1 (2022-12-09)
Documentation
Dependencies
- make
pyarrow
andBQ Storage
optional dependencies (e1aa921)
December 15, 2022
You can now access and query Cloud SQL data over a private connection. This feature is generally available (GA).
December 08, 2022
The demo query guide helps you query a public dataset from Google Trends and is now in preview.
December 05, 2022
A weekly digest of client library updates from across the Cloud SDK.
Go
Changes for bigquery/storage/apiv1beta1
1.44.0 (2022-11-30)
Features
- bigquery/datatransfer: Add Location API methods docs: updated comments (22ec3e3)
- bigquery/storage: Add missing_value_interpretations to AppendRowsRequest (2a0b1ae)
- bigquery: Add default partition expiration to Dataset (#7096) (601c77a), refs #7021
- bigquery: Remove code for reservation/apiv1beta1 (#7010) (451acc1)
- bigquery: Start generating proto stubs (#7026) (debc4c7)
- bigquery: Widen retry predicate (#6976) (753b751)
Bug Fixes
Documentation
- bigquery/storage: Remove stale header guidance for AppendRows (9c5d6c8)
December 01, 2022
BigQuery now supports querying Apache Iceberg tables that are created by open source engines. This feature is in preview.
November 28, 2022
BigQuery now supports the following features when you load data:
- ASCII control characters for CSV files.
- Reference file with the expected table schema for creating external tables with Avro, ORC, and Parquet files.
These features are generally available (GA).
November 21, 2022
A weekly digest of client library updates from across the Cloud SDK.
November 17, 2022
Object tables are now in preview. Object tables are read-only tables containing metadata for unstructured data stored in Cloud Storage. These tables enable you to analyze and perform inference on images, audio files, documents, and other file types by using BigQuery ML and BigQuery remote functions. Object tables extend structured data features such as data security and governance best practices to unstructured data.
Metadata caching is now in preview. Using cached metadata might improve query performance for BigLake tables and object tables that reference large numbers of objects, by allowing the query to avoid listing objects from Cloud Storage.
November 14, 2022
A weekly digest of client library updates from across the Cloud SDK.
Java
Changes for google-cloud-bigquery
2.19.1 (2022-11-08)
Dependencies
2.19.0 (2022-11-07)
Features
Dependencies
- Update dependency com.google.apis:google-api-services-bigquery to v2-rev20221028-2.0.0 (#2393) (d3f6a6b)
- Update dependency com.google.cloud:google-cloud-shared-dependencies to v3.0.6 (#2399) (9de9aa8)
- Update dependency org.graalvm.buildtools:junit-platform-native to v0.9.17 (#2396) (87f8cdd)
- Update dependency org.graalvm.buildtools:native-maven-plugin to v0.9.17 (#2397) (7927350)
The slot estimator helps you manage slot capacity based on historical performance metrics. This feature is now generally available (GA).
November 09, 2022
You can now transfer data from Amazon S3 and Azure Blob Storage to BigQuery using the LOAD DATA
statement. This feature is generally available (GA) and includes support for the following features:
- Transfer files that are hive partitioned.
- Load semi-structured JSON source data into BigQuery without providing a schema by using JSON columns in the destination table.
- Encrypt destination tables using customer managed encryption keys.
- Transfer data to
US
multi-region andUS-EAST-4
regions.
November 07, 2022
A weekly digest of client library updates from across the Cloud SDK.
Python
Changes for google-cloud-bigquery
3.3.6 (2022-11-02)
Features
Bug Fixes
- Corrects test for non-existent attribute (#1395) (a80f436)
- deps: Allow protobuf 3.19.5 (#1379) (3e4a074)
- deps: Allow pyarrow < 11 (#1393) (c898546)
- deps: Require requests>=2.21.0 (#1388) (e398336)
- Refactor to adapt to changes to shapely dependency (#1376) (2afd278)
Documentation
Miscellaneous Chores
- release 3.3.6 (4fce1d9)
In the Explorer pane, you can now star your projects, datasets, and tables. This feature replaces the pin feature, which formerly allowed you to pin projects to the Explorer pane. This feature is now generally available (GA).
In the Cloud console, the Add data feature lets you access popular ways to search for and ingest data sources that work with BigQuery. For an example, see viewing listings in Analytics Hub. This feature is now generally available (GA).
November 03, 2022
SQL functions for managing wrapped keysets are generally available (GA). You can now perform the following actions natively in BigQuery with fewer risks and steps:
- Create a wrapped keyset
- Rotate a wrapped keyset
- Rewrap a wrapped keyset
- Encrypt and decrypt a column with a wrapped keyset
Included with this release are the following new key management functions:
November 02, 2022
The query execution graph is now in preview. You can use the query execution graph to diagnose query performance issues, and to receive query performance insights.
November 01, 2022
The BigQuery migration assessment is now available for Amazon Redshift in preview. You can use this feature to assess the complexity of migrating from your Amazon Redshift data warehouse to BigQuery.
October 31, 2022
A weekly digest of client library updates from across the Cloud SDK.
Go
Changes for bigquery/storage/apiv1beta1
1.43.0 (2022-10-24)
Features
- bigquery/analyticshub: rename nodejs analyticshub library package name (52dddd1)
- bigquery/storage/managedwriter: Enable field name indirection (#6247) (1969273)
- bigquery/storage/managedwriter: Retry on FailedPrecondition (#6761) (d1a444d)
- bigquery/storage/managedwriter: Support append retries (#6695) (6ae9c67)
- bigquery/storage/managedwriter: Switch to opt-in retry (#6765) (a3e97a6)
- bigquery: Add remote function options to routine metadata (#6702) (d9a437d)
Bug Fixes
- bigquery/storage/managedwriter: Address possible resource leak (#6775) (979440b)
- bigquery: Avoid stack overflow on query param with recursive types (#6890) (854ccfc), refs #6884
- bigquery: Bq connection auth scopes (#6752) (8e09288), refs #6744
Documentation
Java
Changes for google-cloud-bigquery
2.18.2 (2022-10-28)
Dependencies
2.18.1 (2022-10-28)
Dependencies
2.18.0 (2022-10-27)
Features
Bug Fixes
- Add --add-opens arg to native-image command (#2369) (8e8b6d7)
- Properly handle external table schema on table update (#2236) (460ef31)
Dependencies
- Update arrow.version to v10 (major) (#2371) (b7873db)
- Update cloud client dependencies (#2362) (0936699)
- Update dependency com.google.api.grpc:proto-google-cloud-bigqueryconnection-v1 to v2.6.0 (#2355) (7bc59a7)
- Update dependency com.google.api.grpc:proto-google-cloud-bigqueryconnection-v1 to v2.7.0 (#2366) (02102d3)
- Update dependency com.google.apis:google-api-services-bigquery to v2-rev20221015-2.0.0 (#2370) (9b796cf)
- Update dependency com.google.cloud:google-cloud-datacatalog-bom to v1.10.0 (#2356) (edb2ca0)
- Update dependency com.google.cloud:google-cloud-shared-dependencies to v3.0.5 (#2361) (51b2258)
- Update dependency com.google.code.gson:gson to v2.10 (#2367) (82e3de5)
- Update dependency org.graalvm.buildtools:junit-platform-native to v0.9.15 (#2352) (b0f172c)
- Update dependency org.graalvm.buildtools:junit-platform-native to v0.9.16 (#2358) (f4e5fc5)
- Update dependency org.graalvm.buildtools:native-maven-plugin to v0.9.15 (#2353) (ac9226c)
- Update dependency org.graalvm.buildtools:native-maven-plugin to v0.9.16 (#2359) (52ec31a)
The max_staleness
materialized view option helps you achieve consistently high performance with controlled costs when processing large, frequently changing datasets. This feature is now in preview.
Column-level data masking is now generally available (GA). You can use data masking to selectively obscure column data for groups of users, while still allowing access to the column.
October 27, 2022
Search indexes and the SEARCH() function are now generally available (GA). These enable you to use Google Standard SQL to efficiently pinpoint specific data elements in unstructured text and semi-structured data.
October 26, 2022
The following geography functions are now generally available (GA):
ST_ISCLOSED
: ReturnsTRUE
for a non-empty geography, where each element in the geography has an empty boundary.ST_ISRING
: Checks if a geography is a linestring and if the linestring is both closed and simple.
October 24, 2022
You can now view BI Engine Top Tables Cached Bytes, BI Engine Query Fallback Count, and Query Execution Count as dashboard metrics for BigQuery. This feature is now in preview.
October 18, 2022
Remote functions, which let you invoke functions from Cloud Functions or Cloud Run in your Google Standard SQL queries, are now generally available (GA).
October 17, 2022
A weekly digest of client library updates from across the Cloud SDK.
Java
Changes for google-cloud-bigquery
2.17.1 (2022-10-10)
Dependencies
Python
Changes for google-cloud-bigquery
3.3.5 (2022-10-10)
Bug Fixes
3.3.5 (2022-10-10)
Bug Fixes
October 12, 2022
The reporting process for the tabledata.list bytes per minute
quota has been updated to more accurately reflect the enforced limit. The limit has not changed.
October 11, 2022
Analytics Hub is now generally available. As an Analytics Hub publisher, you can now view all subscriptions to your listing and remove a subscription from your listing.
You can now use stored procedures for Apache Spark. This feature is in preview.
October 10, 2022
Multi-statement transactions are now generally available (GA).
A weekly digest of client library updates from across the Cloud SDK.
Java
Changes for google-cloud-bigquery
2.17.0 (2022-10-03)
Features
Dependencies
- Update dependency com.google.api.grpc:proto-google-cloud-bigqueryconnection-v1 to v2.5.5 (#2328) (6e48ec2)
- Update dependency com.google.apis:google-api-services-bigquery to v2-rev20220913-2.0.0 (#2287) (fa33184)
- Update dependency com.google.apis:google-api-services-bigquery to v2-rev20220924-2.0.0 (#2325) (82c2097)
- Update dependency com.google.cloud:google-cloud-shared-dependencies to v3.0.4 (#2327) (188c779)
- Update dependency gcp-releasetool to v1.8.9 (#2326) (52dfd13)
- Update dependency importlib-metadata to v4.13.0 (#2323) (4c7e089)
- Update dependency importlib-metadata to v5 (#2324) (bd43cf4)
- Update dependency org.graalvm.buildtools:junit-platform-native to v0.9.14 (#2288) (959519c)
- Update dependency org.graalvm.buildtools:native-maven-plugin to v0.9.14 (#2289) (3cf7ef8)
- Update dependency org.junit.vintage:junit-vintage-engine to v5.9.1 (#2285) (65fac18)
Python
Changes for google-cloud-bigquery
3.3.4 (2022-09-29)
Bug Fixes
The ability to use physical bytes for storage billing is now in Preview. For more information, see Dataset storage billing models.
October 05, 2022
You can now explore query results in Colab using Python libraries. This feature is now in preview.
Concurrent connections quotas are now based on the project that initiates the Storage Write API request, not the project containing the BigQuery dataset resource.
October 03, 2022
A weekly digest of client library updates from across the Cloud SDK.
Python
Changes for google-cloud-bigquery
3.3.3 (2022-09-28)
Bug Fixes
- Refactors code to account for a tdqm code deprecation (#1357) (1369a9d)
- Validate opentelemetry span job attributes have values (#1327) (8287af1)
Documentation
- samples: uses function (create_job) more appropriate to the described sample intent (5aeedaa)
September 29, 2022
In addition to standard rounding, BigQuery now supports the rounding mode ROUND_HALF_EVEN
for parameterized NUMERIC
or BIGNUMERIC
columns. The ROUND()
function also accepts the rounding mode as an optional argument. This feature is now in preview.
September 28, 2022
With Datastream for BigQuery, you can now replicate data and schema updates from operational databases directly into BigQuery. This feature is now in preview.
September 26, 2022
The totalItems
field returned by the projects.list
API method now returns the number of items per page, rather than an approximate total number of projects across all pages.
A weekly digest of client library updates from across the Cloud SDK.
Go
Changes for bigquery/storage/apiv1beta1
1.42.0 (2022-09-21)
Features
- bigquery/analyticshub: Start generating apiv1 (#6707) (feb7d7d)
- bigquery/datapolicies: Start generating apiv1beta1 (#6697) (f5443e8)
- bigquery/reservation/apiv1beta1: add REST transport (f7b0822)
- bigquery/storage/managedwriter: Define append retry predicate (#6650) (478b8dd)
- bigquery/storage: add proto annotation for non-ascii field mapping (ec1a190)
- bigquery: Add reference file schema option for federated formats (#6693) (3d26091)
- bigquery: Add support for explicit query parameter type (#6596) (d59b5b2), refs #4704
Bug Fixes
- bigquery/connection: integrate gapic-generator-python-1.4.1 and enable more py_test targets (ec1a190)
In the Explorer pane, you can now open tables in Connected Sheets. This feature is now generally available (GA).
September 21, 2022
BigQuery Omni has introduced support for on-demand pricing model (GA) for a limited duration. For more information, see BigQuery Omni Pricing.
You can now view shuffle usage ratios in the admin resource charts. This feature is now in preview.
September 19, 2022
A weekly digest of client library updates from across the Cloud SDK.
Go
Changes for bigquery/storage/apiv1beta1
1.41.0 (2022-09-14)
Features
- bigquery/storage: add location to WriteStream and add WriteStreamView support (6a0080a)
Bug Fixes
Java
Changes for google-cloud-bigquery
2.16.1 (2022-09-15)
Dependencies
2.16.0 (2022-09-12)
Features
- Add preserveAsciiControlCharacters to CsvOptions (#2143) (856893f)
- Add reference file schema option for federated formats (#2269) (8c488e6)
Bug Fixes
Dependencies
September 16, 2022
BigQuery Omni now supports the following quota and limit:
- The quota for total query result sizes for a project is now 1 TB per day. For more information, see Query jobs.
- The limit for maximum result size for a query has been increased from 2 MB to 10 GB (preview).
For more information, see Limitations.
September 15, 2022
The BigQuery Data Transfer Service for Google Ads now supports the new Google Ads API. This feature is now in preview.
BigQuery is now available in the Madrid (europe-southwest1), Milan (europe-west8), and Paris (europe-southwest1) regions. The Madrid and Paris regions have the lowest carbon impact.
September 14, 2022
The Merge is coming! You may experience disruptions in the Ethereum public datasets in BigQuery.
The is_case_insensitive
schema option, which allows you to make a dataset and its table names case-insensitive, is now in preview.
JDBC driver update, release 1.3.0 1001
- You can now configure the connector to authenticate the connection using an external account (workforce or workload identity federation).
- You can now configure the connector to use Private Service Connect URLs.
- The connector now supports JDBC transaction APIs. BigQuery supports multi-statement transactions inside a single query, or across multiple queries, when using sessions.
- The connector is now verified to use a default project for datasets. To do this, set the
dataset_project_id
property inQueryProperties
of the connection string to the desired project. MATERIALIZED_VIEW
has been added to the list of table types when using thegetTableTypes
function.- The connector now supports the JSON data type.
ODBC driver update, release 2.5.0 1001
- You can now configure the connector to authenticate the connection using an external account (workforce or workload identity federation), with limited support, using Azure AD and Okta identity providers.
- You can now configure the connector to use Private Service Connect URLs.
- The connector now supports ODBC transaction APIs. BigQuery supports multi-statement transactions inside a single query, or across multiple queries, when using sessions.
- The connector is now verified to use a default project for datasets. To do this, set the
dataset_project_id
property inQueryProperties
of the connection string to the desired project. MATERIALIZED_VIEW
has been added to the list of table types. To retrieve these table types, configureSQLTables
toTABLE_TYPES_ONLY
.- The connector now supports the JSON data type.
September 13, 2022
In Cloud Monitoring, you can view metrics for quota usage and limits of the Storage Write API's concurrent connections and throughput quotas. This feature is now generally available (GA).
September 12, 2022
A weekly digest of client library updates from across the Cloud SDK.
Node.js
Changes for @google-cloud/bigquery
6.0.3 (2022-08-23)
Bug Fixes
Go
Changes for bigquery/storage/apiv1beta1
1.40.0 (2022-09-08)
⚠ BREAKING CHANGES
- bigquery/dataexchange: update BigQuery Analytics Hub API v1beta1 client BREAKING CHANGE: refresh current dataexchange/v1beta1/* directory to include recent change in protos. Removed common directory and use local enum Category fix!: refactor references to Category message docs: improve proto documentation.
Features
- bigquery/dataexchange: update BigQuery Analytics Hub API v1beta1 client BREAKING CHANGE: refresh current dataexchange/v1beta1/* directory to include recent change in protos. Removed common directory and use local enum Category fix!: refactor references to Category message docs: improve proto documentation. (e45ad9a)
- bigquery/storage/managedwriter: Augment reconnection logic (#6609) (6b0ac0c)
- bigquery: Add trace instrumentation support for individual rpcs (#6493) (eedc632)
- bigquery: Improve error when reading null values (#6566) (e9a94c2), refs #2612
Documentation
Miscellaneous Chores
September 06, 2022
Cloud console updates: Improvements that are related to query execution include the following:
For long-running queries, the Execution details tab is automatically displayed with the timing details of each stage of the query.
In the query editor, you can now see the query validation message when your query is completed or canceled.
September 01, 2022
Customer-managed encryption keys are now integrated with CMEK organization policies. This feature is generally available (GA).
The slot recommender creates recommendations for customers using on-demand billing and is now generally available (GA).
August 29, 2022
A weekly digest of client library updates from across the Cloud SDK.
Go
Changes for bigquery/storage/apiv1beta1
1.39.0 (2022-08-23)
Features
- bigquery/storage: allow users to set Apache Avro output format options through avro_serialization_options param in TableReadOptions message Through AvroSerializationOptions, users can set enable_display_name_attribute, which populates displayName for every avro field with the original column name Improved documentation for selected_fields, added example for clarity. (41ab4ec)
- bigquery: add PreserveAsciiControlCharacters support for CSV (#6448) (b7bac2f)
- bigquery: add preview support for default values (#6464) (edc3be5)
Java
Changes for google-cloud-bigquery
2.14.7 (2022-08-23)
Bug Fixes
2.15.0 (2022-08-25)
Features
Community contributed UDFs are now generally available in the bigquery-utils
GitHub repository and the bigquery-public-data.persistent_udfs
public dataset.
Cloud console updates: In the query editor, when you select a function signature from the autocomplete list, you can remove the parameter names quickly by pressing the Backspace or Delete key.
August 22, 2022
A weekly digest of client library updates from across the Cloud SDK.
Go
Changes for bigquery/storage/apiv1beta1
1.38.0 (2022-08-12)
Features
- bigquery/migration: Add MySQL dialect to bigquerymigration v2 client library (370e23e)
- bigquery/storage/managedwriter: improve error communication (#6360) (b30d89d)
Bug Fixes
Python
Changes for google-cloud-bigquery
3.3.2 (2022-08-16)
Bug Fixes
August 19, 2022
The ALTER TABLE RENAME COLUMN
DDL statement, which allows you to rename the columns of a table, is now in preview.
August 17, 2022
You can now set default values on columns in your BigQuery tables. This feature is now in preview.
Cloud console updates: You can now copy BigQuery metadata to your clipboard by using the following options:
In the Schema view, to copy a table's schema, select any fields, and then click
Copy.In the Explorer pane, to copy the ID of a resource, click
View actions, and then click Copy ID.
Cloud console updates: Improvements include the following:
Query results are now displayed in resizable columns.
Tab titles now expand when space is available for longer names.
Tooltips no longer display text immediately when you hold the pointer over them, avoiding unnecessary distraction.
In the Explorer pane, you can now access saved queries by expanding your project. The Saved Queries pane is no longer at the bottom of the console.
In the Explorer pane, you can now find a table by searching for
mydataset.mytable
.In the query editor, you can now press the F1 shortcut key to view more editor shortcuts.
August 16, 2022
Workforce identity federation lets you authenticate and authorize users from external identity providers to access supported Google Cloud products, including BigQuery resources. This feature is now in preview.
August 15, 2022
A weekly digest of client library updates from across the Cloud SDK.
Python
Changes for google-cloud-bigquery
3.3.1 (2022-08-09)
Bug Fixes
An issue was identified in the max_staleness
materialized view option. This feature is not available.
BigQuery Omni now supports reservation DDL and access control DCL. This feature is now generally available (GA).
Previously, you could commit up to 100 GB in streaming bytes for every Storage Write API pending mode commit that you triggered in regions other than the US and EU multi-regions. This limit is now 1 TB. For more information, see Storage Write API quotas.
A weekly digest of client library updates from across the Cloud SDK.
August 10, 2022
You can now set default configurations at a project or organization level. This feature is now generally available (GA).
You can now manage query execution priority for Cloud Spanner federated queries. This feature is now generally available (GA).
August 09, 2022
August 08, 2022
A weekly digest of client library updates from across the Cloud SDK.
Go
Changes for bigquery/storage/apiv1beta1
1.37.0 (2022-08-04)
Features
- bigquery/connection: Add service_account_id output field to CloudSQL properties (1d6fbcc)
- bigquery/storage/managedwriter: refactor AppendResponse (#6402) (c07bca2)
- bigquery: support JSON as a data type (#5986) (835fe4f)
Bug Fixes
Documentation
- bigquery/storage: clarify size limitations for AppendRowsRequest chore: add preferred_min_stream_count to CreateReadSessionRequest chore: add write_stream to AppendRowsResponse (1d6fbcc)
A weekly digest of client library updates from across the Cloud SDK.
Java
Changes for google-cloud-bigquery
2.14.3 (2022-08-05)
Dependencies
- update dependency com.google.apis:google-api-services-bigquery to v2-rev20220730-2.0.0 (#2208) (5165e2b)
- update dependency com.google.cloud:google-cloud-storage to v2.11.2 (#2207) (da5389d)
2.14.2 (2022-08-04)
Dependencies
- update arrow.version to v9 (major) (#2201) (3ec5ef9)
- update dependency com.google.apis:google-api-services-bigquery to v2-rev20220716-2.0.0 (#2202) (c1ca09e)
- update dependency com.google.cloud:google-cloud-bigquerystorage-bom to 2.18.0 (c1ca09e)
- update dependency com.google.cloud:google-cloud-datacatalog-bom to 1.9.1 (c1ca09e)
- update dependency com.google.cloud:google-cloud-shared-dependencies to v3 (c1ca09e)
- update dependency com.google.cloud:google-cloud-storage to 2.11.0 (c1ca09e)
- update dependency com.google.cloud:google-cloud-storage to v2.11.1 (#2194) (45be001)
- update dependency com.google.code.gson:gson to v2.9.1 (#2190) (4bd4539)
- update dependency org.threeten:threeten-extra to v1.7.1 (c1ca09e)
Documentation
August 03, 2022
The max_staleness
materialized view option helps you achieve consistently high performance with controlled costs when processing large, frequently changing datasets. This feature is now in preview.
August 01, 2022
A weekly digest of client library updates from across the Cloud SDK.
Python
Changes for google-cloud-bigquery
3.3.0 (2022-07-25)
Features
Bug Fixes
Documentation
The trigonometric SQL function CBRT is now generally available (GA). With this function, you can compute the cube root of a value.
The LOAD DATA
statement
is now available for Preview in Google Standard SQL for BigQuery.
You can use the LOAD DATA
statement to load data from one or more files into a table.
A weekly digest of client library updates from across the Cloud SDK.
Java
Changes for google-cloud-bigquery
2.14.1 (2022-07-27)
Dependencies
July 28, 2022
You can now create BigQuery subscriptions in Pub/Sub to write messages directly to an existing BigQuery table.
July 27, 2022
Inverse trigonometric SQL functions are now generally available (GA). These functions include:
July 25, 2022
BigLake is now generally available (GA). You can now create BigQuery ML models using data in Cloud Storage by using BigLake and publish BigLake tables as Analytics Hub listings.
The new Migrate section in the BigQuery documentation helps you migrate to BigQuery. This includes high-level guidance with a migration overview, an introduction to free-to-use tools that help you with each phase of migration, and platform-specific migration guides.
July 20, 2022
Analytics Hub is now available in additional regions across the Americas, Asia Pacific, and Europe. For more information, see Analytics Hub supported regions.
July 14, 2022
Previously, the Storage Write API had a maximum concurrent connection limit of 100 connections for non-multi-regions such as Montreal (northamerica-northeast1). This limit has now been increased to 1,000 connections across all non-multi-regions. For more information, see Storage Write API quotas and limits.
July 12, 2022
You can now select a job type when assigning a folder, organization, or project to a reservation in the Cloud console. This feature is now generally available (GA).
July 11, 2022
The google.cloud.bigquery.reservation.v1beta1.api package is deprecated and will be removed on September 27, 2022. After that date, requests to that package will fail. Data created by using google.cloud.bigquery.reservation.v1beta1.api are accessible by using the google.cloud.bigquery.reservation.v1.api package.
Next steps:
- If you use the API directly, you should switch to google.cloud.bigquery.reservation.v1.api, the GA version of the API, to prevent any impact on your workflow.
- If you only use the Cloud console to manage BigQuery reservations, no action is needed.
- If you use the bq command-line tool to manage BigQuery reservations, upgrade the tool to the latest version.
July 07, 2022
Azure workload identity federation is now available in preview for BigQuery Omni connections. This feature helps you secure data by allowing you to grant Google access to an application you manage in your Azure tenant so that neither you nor Google must manage application client secrets.
July 06, 2022
The APPENDS
change history TVF is now in preview. This table-valued function provides a history of table appends over a window of time.
July 01, 2022
An updated version of JDBC driver for BigQuery is now available. This version includes a fix for an issue with connector returning stack overflow in some cases when executing complex long queries.
June 29, 2022
You can now set the view
field in the tables.get()
API method to indicate which table information is returned. Setting the value to BASIC
reduces latency by omitting some storage statistics.
Previously, all BigQuery BI Engine projects had a maximum reservation size per project per location limit of 100 GB. This limit is now 250 GB. For more information, see BI Engine quotas and limits.
June 23, 2022
The BI Engine preferred tables feature lets you limit BI Engine acceleration to a specified set of tables. This feature is now in preview.
June 21, 2022
Query queues are now available in preview for on-demand and flat-rate customers. When query queues are enabled, BigQuery automatically determines the query concurrency rather than setting a fixed limit. Flat-rate customers can override this setting with a custom concurrency target. Additional queries beyond the concurrency target are queued until processing resources become available.
June 15, 2022
Deterministic encryption SQL functions are now generally available (GA). New AEAD encryption functions include DETERMINISTIC_ENCRYPT
, DETERMINISTIC_DECRYPT_BYTES
, and
DETERMINISTIC_DECRYPT_STRING
. These functions allow column-level encryption and decryption of data while supporting aggregation and table joins.
June 14, 2022
You can now use the Cloud console to set up VPC service control perimeters to restrict access from BigQuery Omni to external clouds. You can also specify whether you want to grant read or write permission on your external resource. This feature is now generally available (GA).
You can now explore data in Data Studio by using links from your BigQuery query results in the Cloud console. This feature is now generally available (GA).
June 13, 2022
A new system variable, @@dataset_project_id
, is now generally available. @@dataset_project_id
allows you to set a default project where one is not specified for a dataset in your query. This variable is also available as a Connection Property.
June 10, 2022
Quotas for multi-statement queries have changed. The cumulative time limit for a multi-statement query has increased from 6 hours to 24 hours.
June 08, 2022
Batch and interactive translation services are now generally available (GA), and include support for most major SQL dialects. This release also includes preview availability of SQL object name mapping and metadata extraction tools that you can use to increase the accuracy of your batch translation jobs.
June 06, 2022
You can now attach Resource Manager tags to datasets. This feature is supported in Preview. Tags let you conditionally apply Identity and Access Management (IAM) policies to resources.
The following Storage Read API quotas and limits have changed:
- There is now a limit of 2,000 concurrent
ReadRows
calls per project in theUS
andEU
multi-regions and 400 concurrentReadRows
calls in other regions. - The number of data plane requests per user per project per minute has increased from 5,000 to 25,000.
For more information, see Storage Read API quotas and limits.
June 03, 2022
BigQuery Omni now supports Reservation and Access Control DCL. This feature is in Preview.
May 31, 2022
Column-level data masking is now available in preview. You can use data masking to selectively obscure column data for groups of users, while still allowing access to the column. When you use data masking in combination with column-level access control, you can configure a range of access to column data, from full access to no access, based on the requirements of different groups of users.
May 24, 2022
You can now load data into BigQuery using Informatica Data Loader. This feature is generally available. Informatica provides connectors that can ingest data into BigQuery.
May 23, 2022
Metrics for query/statement_scanned_bytes and query/statement_scanned_bytes_billed are no longer delayed for 6 hours in order to smooth reporting over the duration of the job. Values are now reported every 180 seconds without smoothing. For more information about metrics, see Google Cloud metrics.
May 18, 2022
Updated versions of ODBC and JDBC drivers for BigQuery are now available that include enhancements.
May 05, 2022
The new format element %J
is generally available (GA) for DATE
, TIME
, DATETIME
, and TIMESTAMP
functions. This format element lets you use the ISO 8601 1-based day of the year.
PARSE_DATE
, PARSE_TIME
, PARSE_DATETIME
, and PARSE_TIMESTAMP
now support the following date and time format elements: %a
, %A
, %g
, %G
, %j
, %u
, %U
, %V
, %w
, and %W
.
May 02, 2022
Case-insensitive collation support for BigQuery is now available for Preview. Collation determines how strings are sorted and compared in collation-supported operations. If case-insensitive collation is used, case is ignored in comparison and sorting operations.
These operations support collation:
The COLLATE
function is now available for Preview in Google Standard SQL for BigQuery. With the COLLATE
function, you can pass in a STRING
and return a STRING
with a collation specification.
The DEFAULT COLLATE
clause is now available for Preview. With this clause, the default collation specification is applied to all column data types supporting collation. You can use the DEFAULT COLLATE
clause in the following DDL statements:
The COLLATE
clause is now available for Preview. With this clause, a collation specification is applied to a specific column in a table. You can use the COLLATE
clause in the following DDL statements:
April 25, 2022
The ability to configure the time travel window is now in Preview. You can specify the duration of the time travel window, from a minimum of two days to a maximum of seven days.
Three new INFORMATION_SCHEMA
views that show table storage metadata are now in
Preview.
- Use the
TABLE_STORAGE
view to get a snapshot of current storage usage for tables and materialized views. - Use the
TABLE_STORAGE_TIMELINE_BY_PROJECT
andTABLE_STORAGE_TIMELINE_BY_ORGANIZATION
views to understand table storage changes over time at either the project or the organization level.
BigQuery Admin Resource Charts are now generally available (GA) for on-demand users, enabling administrators to monitor key metrics and troubleshoot issues across the entire organization. Previously, it was only available for reservation users. A new permission, bigquery.jobs.listExecutionMetadata, has been added to make it easier to gain access to the full UI.
April 11, 2022
Starting in July 2022, the projects.list
API method will return results in unsorted order. Currently, the API returns the results in sorted order, although this is not a documented behavior of the API.
April 07, 2022
BigLake is now available in Preview. BigLake is a storage engine that allows you to query and unify cross-cloud data lakes and warehouses. Additionally, it provides fine-grained access controls to your tables, allowing you to set access policies on a column or row basis.
BigQuery now supports the creation of search indexes and a SEARCH
function. This feature is in Preview. This enables you to use Google Standard SQL to efficiently find data elements in unstructured text and semi-structured data.
April 06, 2022
Analytics Hub is now available in Preview. Analytics Hub is a new service in BigQuery that lets you create secure data exchanges and share analytics assets within and across organizations. This platform allows data providers to publish listings that reference shared datasets. Analytics Hub subscribers can then view and subscribe to these listings.
April 05, 2022
BigQuery Omni now supports cross-cloud transfer. This feature is in Preview. For more information, see Cross-cloud transfer (AWS) and Cross-cloud transfer (Azure).
March 31, 2022
The international public dataset for Data Signals for Google Search Trends is now available in Preview and available in the Google Cloud Marketplace and Analytics Hub.
March 17, 2022
The BigQuery slot recommender is now available in Preview. The slot recommender creates recommendations for customers using on-demand billing. These recommendations help you to understand the cost and performance tradeoffs of purchasing different amounts of slot capacity.
March 16, 2022
You can now explicitly specify a schema for BigQuery external tables created over Parquet, ORC, and Avro file formats. Previously, the schema was always auto-detected using the last lexicographic file.
March 04, 2022
Session support for BigQuery is now generally available (GA). In addition to the features available in the preview, you can:
- Terminate a session automatically or manually.
- Set a label for all queries in a session.
- Get sessions metadata in
INFORMATION_SCHEMA.SESSIONS_BY_PROJECT
andINFORMATION_SCHEMA.SESSIONS_BY_USER
views.
February 16, 2022
Remote functions are now available for preview. Remote functions allow you to implement your function in other languages than SQL and Javascript, or with libraries or services which are not allowed in BigQuery user-defined functions.
February 15, 2022
The table clones feature in BigQuery is now in Preview. A table clone is a lightweight, writable copy of a table. You are only charged for storing the data in a table clone that differs from its base table.
February 14, 2022
The INFORMATION_SCHEMA.STREAMING_TIMELINE_*
views are now generally available (GA).
The QUALIFY
clause, which lets you filter the results of analytic functions in Google Standard SQL, is now generally available (GA).
BigQuery reliability guide is now available. This guide describes how to build solutions with BigQuery that meet your application's needs for availability, durability, consistency, and data recovery. Topics include the following:
- Import reliability - Managed storage, methods, load jobs, and the Storage Write API
- Query reliability - Slots, reservations, and job optimization.
- Read reliability - Read methods, consistency concerns including quotas and limits, and the Storage Read API.
- Disaster planning - Disaster considerations and their mitigation.
February 10, 2022
BigQuery Omni now supports INFORMATION_SCHEMA.JOBS_*
and INFORMATION_SCHEMA.RESERVATION*
views. This feature is in Preview. For more information, see View resource metadata (AWS) and View resource metadata (Azure).
February 03, 2022
The BigQuery migration assessment is now available in Preview. Use this feature to assess the complexity of migrating from your current data warehouse to BigQuery.
February 02, 2022
The WITH RECURSIVE
feature has been added to Google Standard SQL for BigQuery and is now in Preview. This feature allows a query in a WITH
clause to refer to either itself or to queries defined later in the WITH
clause.
January 31, 2022
BigQuery now supports materialized views without aggregation and materialized views with inner join. This feature is generally available (GA).
January 06, 2022
BigQuery standard SQL now supports the JSON
data type for storing JSON data. The JSON
data type is in Preview. For more information, see Working with JSON data in Standard SQL.
December 23, 2021
Documentation now includes a series of introductory topics to orient you to BigQuery including:
- What is BigQuery? - Product overview, available tools, and learning resources
- Storage - Infrastructure, ingestion, and optimization
- Analytics - Strategies, SQL queries, and BI tools
- Administration - Resources, workload management, security, and monitoring
In addition, the table of contents is updated to guide you through your staged BigQuery deployment with stages including: Discovery, Get started, Design, Ingest, Analyze, Administer, Secure, and Develop.
December 16, 2021
The row-level security feature now supports administrator access to historical data for tables with row-level access policies.
December 14, 2021
BigQuery BI Engine SQL interface is now generally available.
December 01, 2021
BigQuery Data Transfer Service now supports Audit Logging, Cloud Logging, and Cloud Monitoring. These features are in preview status.
November 19, 2021
Updated versions of ODBC and JDBC drivers for BigQuery are now available that include enhancements.
November 16, 2021
BigQuery is now available in the Santiago (southamerica-west1) region.
BigQuery BI Engine is now available in the Santiago (southamerica-west1) region.
BigQuery Data Transfer Service is now available in the Santiago (southamerica-west1) region.
November 12, 2021
BigQuery now supports authorized datasets (General Availability).
November 09, 2021
The following scripting statements have been added to Google Standard SQL for BigQuery.
- CASE: Executes the first list of SQL statements where a boolean expression is
TRUE
. - CASE search_expression: Executes the first list of SQL statements where the search expression matches a
WHEN
expression. - LABELS: Provides an unconditional jump to the end of the block or loop associated with a label.
- REPEAT: Repeatedly executes a list of SQL statements until the boolean condition at the end of the list is
TRUE
. - FOR...IN: Loops over every row in a table expression.
These features are generally available (GA).
November 08, 2021
The following INFORMATION_SCHEMA views now support a DDL
column. The value of the column is the DDL statement that can be used to create the resource.
This feature is generally available (GA).
November 02, 2021
BigQuery now supports parameterized types. The following parameterized types are supported:
This feature is generally available GA.
October 28, 2021
The table snapshots feature is now generally available (GA). It includes the Cloud console interface and support for creating a table snapshot in a different project from its base table.
October 27, 2021
SQL column-level encryption using Cloud Key Management Service (KMS) is now generally available (GA), letting you encrypt keysets within AEAD encryption functions.
October 25, 2021
BigQuery Omni, a multi-cloud analytics solution, is now generally available.
October 12, 2021
The BigQuery Storage Write API is now generally available (GA). The Storage Write API combines the functionality of high-throughput streaming ingestion and batch loading into a single API.
October 04, 2021
BigQuery Migration Service is now in Preview. It includes the following features:
October 01, 2021
BigQuery pricing has changed as follows:
BigQuery Storage Read API has moved from a single regional SKU to a set of regional SKUs for bytes scanned. All BigQuery Storage Read API users can now read up to 300 TB of data per month at no charge. For more information, see BigQuery data extraction pricing.
BigQuery now charges BigQuery Storage Read API users for network egress. For more information, see BigQuery Storage Read API Network Egress Within Google Cloud.
BigQuery now supports the following geospatial data functions:
ST_BUFFER: Returns a
GEOGRAPHY
that represents the buffer around the inputGEOGRAPHY
. You specify the number of segments to determine how much the resulting geography can deviate from the ideal buffer radius.ST_BUFFERWITHTOLERANCE: Returns a
GEOGRAPHY
that represents the buffer around the inputGEOGRAPHY
. You specify the tolerance to determine how much the resulting geography can deviate from the ideal buffer radius.
These functions are available as a preview.
September 28, 2021
Table functions are now generally available (GA). With the GA release, authorized table functions are now supported.
September 27, 2021
BigQuery now supports the following geospatial data functions:
ST_BOUNDINGBOX: Returns a
STRUCT
that represents the bounding box for a geography.ST_EXTENT: Returns a
STRUCT
that represents the bounding box for a set of geographies.S2_COVERINGCELLIDS: Returns an array of S2 cell IDs that cover a geography.
S2_CELLIDFROMPOINT: Returns the S2 cell ID covering a point geography.
These functions are generally available (GA).
September 21, 2021
When saving query results from the Cloud console to a CSV file, the available download size is now 10 MB. Previously the limit was 16,000 rows. Also, you can now download tables with nested and repeated data to CSV files.
September 17, 2021
BigQuery now supports the following geospatial data functions:
ST_EXTERIORRING: Returns a linestring geography that corresponds to the outermost ring of a polygon geography.
ST_INTERIORRINGS: Returns an array of linestring geographies that corresponds to the interior rings of a polygon geography.
ST_ANGLE: Returns the angle between two intersecting lines.
ST_AZIMUTH: Returns the azimuth of a line segment formed by two points.
ST_NUMGEOMETRIES: Returns the number of geometries in a geography.
ST_GEOMETRYTYPE: Returns the Open Geospatial Consortium (OGC) geometry type that describes a geography as a string.
These functions are generally available (GA).
September 08, 2021
Deleting the metadata for a specific job using the bq
command-line tool is now generally available (GA).
Session support for BigQuery is now in Preview. With sessions:
- You can associate your SQL activities in a session across scripts and multi-statement transactions in BigQuery with a unique session identifier.
- You can use session variables (for example, default timezone or dataset) and temporary tables throughout the life of the session and also across scripts and transactions
- When you enable sessions, all actions performed across multiple sessions can be viewed using the
SESSION_ID
column now available in jobsINFORMATION_SCHEMA
views.
August 31, 2021
An updated version of ODBC driver for BigQuery is now available that includes enhancements.
August 30, 2021
Exporting table data in Parquet format is now generally available (GA).
August 23, 2021
BigQuery Admin Resource Charts are now generally available (GA) for reservation users, enabling administrators to more easily monitor and troubleshoot their BigQuery environment. They provide visibility into key metrics such as slot consumption, job concurrency, job execution time, job errors, and bytes processed across the entire organization.
BigQuery Slot Estimator is now in Preview for reservation users. This tool analyzes slot utilization data to help administrators estimate the right number of slots to purchase, and provides insights on how job performance might be impacted by adding or reducing slot capacity for the entire organization or specific reservations.
August 19, 2021
Cloud Spanner federated queries are now generally available (GA).
August 03, 2021
BigQuery now supports the ALTER COLUMN SET DATA TYPE data definition language (DDL) statement. This feature is generally available (GA).
BigQuery now supports the following data definition language (DDL) statement:
This feature is generally available (GA).
BigQuery is now available in the Toronto (northamerica-northeast2) region.
BigQuery Data Transfer Service is now available in the Toronto (northamerica-northeast2) region.
BigQuery BI Engine is now available in the Toronto (northamerica-northeast2) region.
August 02, 2021
An updated version of JDBC driver for BigQuery is now available that includes Enhancements & New Features.
July 27, 2021
BigQuery now supports the INTERVAL
type, which represents a duration or an amount of time. This type is in Preview.
July 26, 2021
DML query jobs now return statistics about the number of rows that were inserted, deleted, or updated. For more information, see DmlStats
in the Job
resource type. In addition, DML statistics are now available in the INFORMATION_SCHEMA.JOBS_BY_*
views. This feature is generally available (GA).
July 19, 2021
BigQuery now supports workload management data control language (DCL) statements:
This feature is generally available GA.
BigQuery now supports the following SQL query operators:
This feature is generally available (GA).
BigQuery standard SQL now supports the CONTAINS_SUBSTR
function. This feature is generally available (GA).
July 07, 2021
BigQuery now supports materialized views without aggregation and materialized views with inner join. This feature is in Preview.
July 01, 2021
An updated version of ODBC driver for BigQuery is now available that includes bug fixes, parameterized data type support, and metadata retrieval performance improvements.
An updated version of JDBC driver for BigQuery is now available that includes bug fixes, parameterized data type support, and job retry improvements.
June 29, 2021
BigQuery is now available in the Delhi (asia-south2) region.
BigQuery Data Transfer Service is now available in the Delhi (asia-south2) region.
BigQuery BI Engine is now available in the Delhi (asia-south2) region.
BigQuery now supports multi-statement transactions. These allow you to perform mutating operations, such as inserting or deleting rows, on one or more tables, and either commit or roll back the changes atomically. This feature is in Preview.
June 28, 2021
BigQuery now supports access management data control language (DCL) statements and corresponding views:
GRANT and REVOKE statements are generally available (GA). OBJECT_PRIVILEGES table is available in Preview.
BigQuery now supports the following casting features:
- PARSE_BIGNUMERIC
- PARSE_NUMERIC
- Format clause for CAST available for the following data types:
- String type
- Date type
- Datetime type
- Time type
- Timestamp type
- Numeric types
- Bytes type
- Numeric type INT64 aliases (INT, SMALLINT, INTEGER, BIGINT, TINYINT, BYTEINT)
- ST_GEOGFROM
These features are generally available (GA).
BigQuery now supports the ALTER COLUMN SET OPTIONS data definition language (DDL) statement. This feature is generally available (GA).
Table functions are now available in Preview. These user-defined functions, commonly known as table-valued functions (TVFs), return a table value.
The Google Trends dataset is now available in Preview and available in the Google Cloud Marketplace.
Audit logging, Cloud Logging, and Cloud Monitoring for the BigQuery Data Transfer Service are now generally available (GA).
June 25, 2021
BigQuery table snapshots are now in Preview. A table snapshot is a low-cost, read-only copy of a table's data as it was at a particular time. For more information, see Introduction to table snapshots.
June 22, 2021
BigQuery Data Transfer Service now supports Google Merchant Center data transfers for local inventories and regional inventories.
June 21, 2021
Row-level security on table data is now generally available in BigQuery.
BigQuery is now available in the Melbourne (australia-southeast2) region.
BigQuery Data Transfer Service is now available in the Melbourne (australia-southeast2) region.
BigQuery BI Engine is now available in the Melbourne (australia-southeast2) region.
June 07, 2021
BigQuery now supports parameterized types. The following parameterized types are supported:
This feature is in Preview.
May 25, 2021
The free trial period for BigQuery BI Engine's SQL interface has been extended to July 15th, 2021. You must enroll to participate in the preview. With this feature, BI Engine now interacts with popular BI tools such as Looker, Tableau, and more, by means of an interactive SQL interface.
May 20, 2021
BigQuery GIS now supports loading geography data from newline-delimited GeoJSON files. This feature is generally available (GA). For more information, see Loading GeoJSON data.
BigQuery GIS now supports the following functions. These functions are generally available (GA).
These functions return a point of a linestring geography as a point geography.
May 19, 2021
BigQuery now supports the ability to rename tables using SQL. See ALTER TABLE RENAME TO. This feature is generally available (GA).
May 11, 2021
Updated version of ODBC driver for BigQuery includes bug fixes and install guide improvements.
Updated version of JDBC driver for BigQuery includes bug fixes, service account keyfile support, connection property enhancements, and BigQuery client library updates.
May 10, 2021
BigQuery now supports the following SQL query clauses and operators:
This feature is in Preview.
April 30, 2021
BigQuery now supports the following data definition language (DDL) statements:
This feature is in GA.
April 21, 2021
BigQuery supports changing an existing non-clustered table to a clustered table and vice versa. You can also update the set of clustered columns of a clustered table. This feature was first documented in October 2020 but was not included in a release note. For more information, see Modifying clustering specification.
April 12, 2021
The BigQuery Admin Resource Charts Preview is now available for Reservation users, enabling administrators to more easily monitor and troubleshoot their BigQuery environment. It provides visibility into key metrics such as slot consumption, job concurrency, and job execution time across the entire organization.
April 09, 2021
BigQuery now has better support for loading ENUM
and LIST
types in Parquet files.
ENUM
logical types can be converted toSTRING
orBYTES
.- Schema inference is supported for
LIST
logical types.
For more information, see Loading Parquet data from Cloud Storage.
April 07, 2021
Beginning in early Q3 2021, BigQuery Storage Read API will start charging for network egress. In addition, BigQuery Storage Read API will become available in all locations, with appropriate pricing. Another release note will be issued when these changes take effect.
April 06, 2021
The BigQuery Storage Write API is now in Preview. The Storage Write API is a stream-based API for ingesting data into BigQuery at low cost and high throughput. It provides exactly-once delivery semantics with real-time latency. For more information, see Using the BigQuery Storage Write API.
April 02, 2021
BigQuery standard SQL now supports the ALTER TABLE DROP COLUMN. This feature is in Preview.
The maximum length has been increased from 128 characters to 300 characters for the following BigQuery fields: table column names, column alias names, and user-defined function names.
March 31, 2021
BigQuery standard SQL now supports the following statements for creating, configuring, and deleting datasets:
These statements are generally available (GA).
BigQuery standard SQL now supports the TABLESAMPLE
operator, which lets you query random subsets of data from large BigQuery tables. For more information, see Table sampling. This feature is in Preview.
BigQuery standard SQL now supports the following JSON functions:
These statements are generally available (GA).
INFORMATION_SCHEMA
views for table partitions are now available. This feature is in Preview.
The INFORMATION_SCHEMA.TABLES
view now includes a DDL
column that can be used to recreate the table. This feature is in Preview.
Support for the BigNumeric type in BigQuery standard SQL is now generally available (GA).
March 24, 2021
BigQuery is now available in the Warsaw (europe-central2) region.
BigQuery Data Transfer Service is now available in the Warsaw (europe-central2) region.
BigQuery BI Engine is now available in the Warsaw (europe-central2) region.
March 02, 2021
Updated version of Magnitude Simba ODBC driver includes bug fixes, performance improvements, and enhancements such as support for dynamic SQL and additional DDL and DML keywords.
Updated version of Magnitude Simba JDBC driver includes bug fixes and performance improvements.
February 25, 2021
BigQuery BI Engine now interacts with popular BI tools such as Looker, Tableau, and more, by means of an SQL interface. You must enroll to participate in the preview.
BigQuery materialized views are now generally available (GA). BigQuery materialized views are now generally available (GA). Materialized views are precomputed views that periodically cache the results of a query, enhancing performance and efficiency, and reducing costs, particularly for aggregated queries. For more information, see Introduction to materialized views.
February 24, 2021
The BigQuery Data Transfer Service's 1-hour minimum file age requirement for transfers from Cloud Storage has been eliminated.
February 23, 2021
The BigQuery Data Transfer Service's minimum interval time between recurring transfers from Cloud Storage has been reduced from one hour to 15 minutes.
February 16, 2021
BigQuery now supports exporting table data in Parquet format. This feature is in Preview. For more information, see Parquet export details.
February 12, 2021
BigQuery standard SQL queries on Google Cloud Storage data are now supported by cached query results.
February 02, 2021
Updated version of Magnitude Simba JDBC driver includes bug and security fixes and enhancements for additional DDL keywords and dynamic SQL.
January 29, 2021
Clustered tables now support the DATETIME
type for clustering columns. For more information, see Creating and using clustered tables. This feature is generally available.
January 19, 2021
BigQuery is now available in the Iowa (us-central1) region.
The BigQuery Data Transfer Service is now available in the Iowa (us-central1) region.
BigQuery BI Engine is now available in the Iowa (us-central1) region.
January 14, 2021
BigQuery's Cloud console UI has been updated with many usability improvements for analyzing data, including multi-tab navigation, a new resource panel, and a new SQL editor. These updates are in Preview. For more information, see Using the Cloud console.
December 21, 2020
BigQuery standard SQL now supports the BigNumeric data type for high-precision computations. The BigNumeric data type is in Preview.
December 16, 2020
BigQuery Data Transfer Service is now fully integrated with VPC Service Controls, and can be protected using a service perimeter. Please refer to VPC-SC supported products page for more info.
December 10, 2020
Updated version of Magnitude Simba ODBC driver includes bug fixes and enhancements such as support for BigNumeric data and improved driver logic.
Updated version of Magnitude Simba JDBC driver includes bug and security fixes and enhancements such as support for Java 11, SSL trust store, BigNumeric data, and version-agnostic fully-qualified class names.
November 17, 2020
BigQuery column-level security is now generally available. Policy tags can be replicated across locations. For more information, see Introduction to BigQuery column-level security.
October 26, 2020
BigQuery standard SQL now supports the SUBSTRING function. This function is generally available (GA).
October 22, 2020
The ST_GEOGFROMGEOJSON
and ST_GEOGFROMTEXT
geographic functions support a new make_valid
parameter. If set to TRUE
, the function attempts to correct polygon issues when importing geography data.
The ST_GEOGFROMTEXT
function also supports a new planar
parameter. If set to TRUE
, the function treats imported WKT geometries as having planar edges.
These new function parameters are in Beta.
Updated version of Magnitude Simba ODBC driver includes improvements to performance, logging, OpenSSL support, and bug fixes.
October 19, 2020
BigQuery Audit Logs stopped using the following checks for redacting resource names for cross-project access and caller identities: The bigquery.jobs.create
permission check and the internal setting for a project domain. Please review the documentation at Caller identities and resource names.
October 14, 2020
Dynamic SQL is now generally available (GA). Dynamic SQL lets you generate and execute SQL statements dynamically at runtime. For more information, see EXECUTE IMMEDIATE.
BigQuery standard SQL now supports the following new functions. These functions are generally available (GA).
- ASCII
- CHR
- INITCAP
- INSTR
- LAST_DAY
- LEFT
- OCTET_LENGTH
- REGEXP_EXTRACT with 2 additional parameters (position and occurrence)
- REGEXP_INSTR
- REGEXP_SUBSTR
- RIGHT
- SOUNDEX
- TRANSLATE
- UNICODE
BigQuery now supports the following new statements. These statements are generally available (GA).
BigQuery standard SQL now supports DATE arithmetics operators.
The following INFORMATION_SCHEMA
views are now generally available (GA).
BigQuery now supports Unicode table names. For more information, see Table naming.
Queries can now have duplicate column names.
October 06, 2020
Authorized user-defined functions (UDFs) are now generally available (GA). Authorized UDFs let you share query results without giving access to the underlying tables. For more information, see Authorized UDFs.
The Cloud console now lets you opt in to search and autocomplete powered by Data Catalog. This feature is in beta.
October 01, 2020
BigQuery table-level access control is now generally available. For more information, see Introduction to table access controls .
September 21, 2020
The following time-unit partitioning features are now Generally Available (GA):
- Creating partitions using hourly, monthly, and yearly time-unit granularities.
- Using a
DATETIME
column as the partitioning column.
September 18, 2020
Beginning in late 2020, all BigQuery predefined roles with permission to create jobs, including queries, will also be able to get information about their own jobs and cancel their own jobs. This change does not include permission to see jobs created by other users.
September 17, 2020
The BigQuery Data Transfer Service is now available in the following regions: Los Angeles (us-west2), São Paulo (southamerica-east1), South Carolina (us-east1), Hong Kong (asia-east1) and Osaka (asia-northeast2).
The BigQuery Data Transfer Service is now available in the following regions: Los Angeles (us-west2), São Paulo (southamerica-east1), South Carolina (us-east1), Hong Kong (asia-east1) and Osaka (asia-northeast2).
September 11, 2020
You can now use the BQ.JOBS.CANCEL
system procedure to cancel a job. For more information, see Canceling jobs.
August 31, 2020
Updated version of Magnitude Simba JDBC driver includes bug fixes and enhancements such as automatically turning on the BigQuery Storage API for anonymous table reads (no charge for temporary table reads).
August 26, 2020
INFORMATION_SCHEMA
views for BigQuery reservations are now Generally Available (GA).
August 25, 2020
INFORMATION_SCHEMA
views for jobs metadata by timeslice are now available.
August 24, 2020
BigQuery support for using service account credentials with scheduled queries is Generally Available (GA).
August 19, 2020
When using consecutive ON
/USING
clauses, parentheses are now optional and can be omitted. For example, you can use either of the following statements:
FROM A JOIN (B JOIN C ON B.x = C.y) USING (z)
FROM A JOIN B JOIN C ON B.x = C.y USING (z)
August 18, 2020
You can now explicitly assign projects to use on-demand pricing with Reservations. See Assign a project to None.
August 13, 2020
The exports per day (Extract Bytes) default quota has been raised from 10 TB to 50 TB per day.
August 11, 2020
For flat-rate pricing, the minimum slot purchase is now 100 slots. Slots can be purchased in 100-slot increments.
August 06, 2020
BigQuery is now available in the following regions: Oregon (us-west1), Belgium (europe-west1), and Netherlands (europe-west4).
BigQuery Data Transfer Service is now available following regions: Oregon (us-west1), Belgium (europe-west1), and Netherlands (europe-west4).
BigQuery BI Engine is now available following regions: Oregon (us-west1), Belgium (europe-west1), and Netherlands (europe-west4).
July 31, 2020
Updated version of Magnitude Simba ODBC driver includes performance improvements and bug fixes.
July 27, 2020
INFORMATION_SCHEMA
views for streaming metadata are now in alpha. You can use these views to retrieve historical
and real-time information about streaming data into BigQuery.
July 24, 2020
BigQuery Data Transfer Service is now available in the following regions: Montréal (northamerica-northeast1), Frankfurt (europe-west3), Mumbai (asia-south1), and Seoul (asia-northeast3).
BigQuery Data Transfer Service is now available in the following regions: Montréal (northamerica-northeast1), Frankfurt (europe-west3), Mumbai (asia-south1), and Seoul (asia-northeast3).
July 16, 2020
BigQuery GIS now supports two new functions, ST_CONVEXHULL
and ST_DUMP
:
ST_CONVEXHULL
returns the smallest convexGEOGRAPHY
that covers the input.ST_DUMP
returns anARRAY
of simpleGEOGRAPHY
s where each element is a component of the inputGEOGRAPHY
.
For more information, see the ST_CONVEXHULL
and ST_DUMP
reference pages.
July 13, 2020
The Standard SQL statement ASSERT
is now supported. You can use ASSERT
to validate that data matches specified expectations.
July 06, 2020
Updated version of Magnitude Simba ODBC driver. This version includes some performance improvements and bug fixes, and it catches up with the JDBC driver by adding support for user defined functions and variable time zones using the connection string.
June 29, 2020
Flex slots are now generally available (GA).
The BigQuery SLA has been updated to >= 99.99% Monthly Uptime Percentage for all users.
June 26, 2020
Region qualified INFORMATION_SCHEMA
views are now in beta.
Starting in mid-August, unqualified INFORMATION_SCHEMA
queries for SCHEMATA
and SCHEMATA_OPTIONS
views will default to returning metadata from the US
multi-region. For information about how to specify a region, see region qualifier syntax.
June 16, 2020
The Top Brands report for Google Merchant Center Best Sellers exports is now in beta.
June 09, 2020
Clustering for non-partitioned tables is now supported. For more information about clustered tables, see Introduction to clustered tables.
June 08, 2020
BigQuery is now available in the Jakarta (asia-southeast2) region.
BigQuery BI Engine is now available in the Jakarta (asia-southeast2) region.
BigQuery Data Transfer Service is now available in the Jakarta (asia-southeast2) region.
June 04, 2020
BigQuery Table ACL is now available as a beta release. For more information, see Introduction to table access controls .
June 02, 2020
You can now purchase BigQuery slots using the bq
command-line tool. BigQuery Reservations allows you to purchase slots to take advantage of BigQuery flat-rate pricing and allocate slots for workload management.
A new GIS function, ST_Simplify
, is available. ST_Simplify
returns a simplified version of the input GEOGRAPHY
by replacing sections with straight lines.
Standard SQL view definition bodies can now contain references without project qualifiers, as long as the view is created by the tables.insert
REST API or is in the same project used to run the CREATE VIEW
DDL query.
Standard SQL user-defined function definition bodies can now contain references to tables and views.
May 21, 2020
The BigQuery Storage API now supports reading small anonymous (cached) tables without any limitations.
May 20, 2020
Happy 10th birthday, BigQuery!
Cloud SQL federated queries are now generally available (GA).
Hourly partitioned tables are now in beta.
Dynamic SQL is now available as a beta release in all BigQuery regions. Dynamic SQL lets you generate and execute SQL statements dynamically at runtime. For more information, see EXECUTE IMMEDIATE.
BigQuery Trial slots are now available in US and EU multi-regions. Trial slots are a limited promotion for qualified customers.
May 13, 2020
Updated versions of Magnitude Simba JDBC drivers have been released.
May 12, 2020
Updated versions of Magnitude Simba ODBC drivers have been released.
May 08, 2020
May 01, 2020
A new function, JSON_EXTRACT_ARRAY
, has been added to the list of JSON functions. This function allows you to extract the contents of a JSON document as a string array. For more information, see the JSON_EXTRACT_ARRAY
reference section.
The ORDER BY
clause now supports the NULLS FIRST
and NULLS LAST
clauses. These clauses allow you to specify the sort order of null and non-null values. For more information, see the ORDER BY
reference section.
April 30, 2020
BigQuery Data Transfer Service is now available in the Taiwan (asia-east1) region.
The BigQuery Data Transfer Service is now available in the Taiwan (asia-east1) region.
April 27, 2020
BigQuery is now available in the Las Vegas (us-west4) region.
BigQuery BI Engine is now available in the Las Vegas (us-west4) region.
BigQuery Data Transfer Service is now available in the Las Vegas (us-west4) region.
April 16, 2020
BigQuery Reservations is now Generally Available (GA). BigQuery Reservations allows you to purchase BigQuery slots to take advantage of BigQuery flat-rate pricing and allocate slots for workload management.
Around the end of April 2020, INFORMATION_SCHEMA
(Beta) views for dataset metadata will
return metadata about all datasets in a region. Currently,
these views return metadata about all datasets in the project across all regions. This
upcoming change will also provide support for querying a specific
region's metadata (for example, region-us.INFORMATION_SCHEMA.SCHEMATA
instead of INFORMATION_SCHEMA.SCHEMATA
).
You can replicate this future behavior now by filtering
on the SCHEMATA
view's LOCATION
column (for example, LOCATION = 'US'
).
April 14, 2020
BigQuery Data Transfer Service now supports Google Merchant Center data transfers for best sellers data.
April 09, 2020
Scheduling queries no longer requires the bigquery.transfers.update
permission. The bigquery.jobs.create
permission can now be used to schedule queries. See Scheduling queries for details.
April 08, 2020
BigQuery materialized views are now available as a beta release. For more information, see Introduction to materialized views .
April 02, 2020
BigQuery Reservations is now available in all BigQuery regions.
March 31, 2020
INFORMATION_SCHEMA
views for BigQuery reservations are now in public alpha.
March 30, 2020
Scripting and stored procedures are now Generally Available.
March 27, 2020
BigQuery Column-level security is now available as a beta release. For more information, see Introduction to BigQuery Column-level security.
March 24, 2020
BigQuery Data Transfer Service is now available in the Northern Virginia (us-east4) region and the Salt Lake City (us-west3) region.
March 23, 2020
BigQuery Data Transfer Service now supports Google Merchant Center data transfers for pricing competitiveness.
March 13, 2020
BigQuery Data Transfer Service now supports the Finland region.
March 11, 2020
BigQuery Data Transfer Service now supports the Zürich region.
March 02, 2020
The BigQuery Storage API is now generally available. For more information, see the Storage API overview.
Support for querying Cloud Storage data in ORC and Parquet format is now Generally Available.
Support for loading and querying externally partitioned data that is stored on Cloud Storage using a default hive partitioning layout is now Generally Available. The data must be in Avro, Parquet, ORC, CSV, or JSON format.
Personally identifiable information (PII) in audit log records for successful reads of resources with restricted access is no longer redacted. For more information, see Caller identities and resource names.
February 27, 2020
Cloud Storage transfers to BigQuery now have the option of appending data or refreshing (overwriting) data.
February 26, 2020
BigQuery Flex slots are now available in beta in all BigQuery Reservations regions. See Workload management and Commitment plans for more information.
BigQuery DML statements no longer have quota limits.
February 25, 2020
Support for data transfers from Amazon S3 to BigQuery is now generally available.
February 24, 2020
BigQuery is now available in the Salt Lake City (us-west3) region.
BI Engine is now available in the Salt Lake City (us-west3) region.
February 20, 2020
BigQuery BI Engine is now GA.
Added support for custom queries and views from Data Studio.
Increased reservation size to 100 GB.
Support for Amazon Redshift data warehouse migrations to BigQuery is now generally available.
Support for migrating Amazon Redshift private instances to BigQuery with VPC is also now generally available.
February 18, 2020
Support for tables partitioned on an integer column is now GA. For more information, see Creating and using integer range partitioned tables.
February 14, 2020
INFORMATION_SCHEMA
views for live and historical jobs metadata are now in beta.
February 13, 2020
The BigQuery sandbox is now GA.
February 11, 2020
Updated versions of Magnitude Simba ODBC drivers are now available.
January 24, 2020
BigQuery is now available in the Seoul (asia-northeast3) region.
January 22, 2020
The BigQuery Storage API is now available in all BigQuery regional and multi-regional locations. For more information, see Dataset locations.
January 16, 2020
Updated versions of Magnitude Simba JDBC and ODBC drivers have been released.
January 14, 2020
There is no longer a charge to use Google's ads management connectors and YouTube connectors for the BigQuery Data Transfer Service. Normal BigQuery analysis and storage pricing will continue to apply after any transfers.
BigQuery Reservations is now available in the London (europe-west2) region.
December 20, 2019
The ALTER TABLE
and ALTER VIEW
Data Definition Language statements are now Generally Available (GA). These statements let you set options on tables and views.
BigQuery Reservations is now available in the Tokyo (asia-northeast1
) region.
December 19, 2019
Third party transfers for loading data from external data sources are now generally available GA.
Support for transferring Google Play reports is now generally available GA.
December 13, 2019
BigQuery Reservations is now in beta in US and EU regions. BigQuery Reservations allows you to purchase BigQuery slots to take advantage of BigQuery flat-rate pricing and allocate slots for workload management.
December 11, 2019
Support for tables partitioned on an integer column is now in Beta. For more information, see Creating and using integer range partitioned tables.
December 10, 2019
For users billed on the on-demand pricing model, BigQuery can now burst beyond the default slot capacity to accelerate queries. There is no extra charge associated with query acceleration.
December 09, 2019
Support for Cloud Storage Transfers is now generally available GA.
November 26, 2019
INFORMATION_SCHEMA
now has updated permissions checks. INFORMATION_SCHEMA
is currently in beta.
Previously, queries against INFORMATION_SCHEMA
views filtered the results based on the IAM permissions needed to get metadata about the resource. Now, queries return accessDenied (403) errors if users have insufficient IAM permissions to get metadata.
November 22, 2019
Support for persistent standard SQL UDFs is now Generally Available (GA).
You can now use INFORMATION_SCHEMA
views to get routine metadata in BigQuery. INFORMATION_SCHEMA
is currently in beta.
November 21, 2019
BigQuery now supports using service account credentials with scheduled queries. This feature is in Beta.
November 20, 2019
BigQuery is now available in the South Carolina (us-east1) region.
November 12, 2019
Support for omitting column names in INSERT and MERGE statements is now Generally Available (GA).
October 31, 2019
Support for querying Cloud Storage data in ORC and Parquet format is now in beta.
Support for loading and querying externally partitioned data that is stored on Cloud Storage using a default hive partitioning layout is now in beta. The data must be in Avro, Parquet, ORC, CSV, or JSON format.
October 23, 2019
The BigQuery service is being renamed from:
bigquery-json.googleapis.com
to
bigquery.googleapis.com
.
bigquery-json.googleapis.com
will become an alias of bigquery.googleapis.com going
forward.
This change is largely transparent to users and does not affect runtime traffic or management operations. Risks and mitigation are discussed below.
Calls to the BigQuery API will not be affected.
Timing
The change is a two phase process that we plan to release during November and December 2019.
Impact when using Service Management and Service Usage APIs
Responses from the Service Management API and the Service Usage API for information about the BigQuery service will contain information about bigquery.googleapis.com
, even when you request data for bigquery-json.googleapis.com
. Other than the name, the responses from the Service Management API and the Service Usage API will be the same. For example, listing enabled services will now return bigquery.googleapis.com
, instead of bigquery-json.googleapis.com
.
Risks and Mitigation
During the week of the second phase, alerts on Stackdriver Transparent Service Level Indicators (SLIs) for the
bigquery-json.googleapis.com
will break. Users should add a duplicate alert forbigquery.googleapis.com
. The following are the serviceruntime SLIs where you now need to usebigquery.googleapis.com
:serviceruntime.googleapis.com/api/request_count
serviceruntime.googleapis.com/api/request_latencies
serviceruntime.googleapis.com/api/request_latencies_backend
serviceruntime.googleapis.com/api/request_latencies_overhead
serviceruntime.googleapis.com/api/request_sizes
serviceruntime.googleapis.com/api/response_sizes
The Google provider for Terraform has been updated to use the new name,
bigquery.googleapis.com
. Terraform users should upgrade to a Google provider with version >= 2.17.0 before November 4, 2019.
October 09, 2019
The RANGE_BUCKET
mathematical function has been added. This function scans through a sorted array and returns the 0-based position of a given point's upper bound. This function can be useful if you need to group your data to build partitions, histograms, business-defined rules, and more.
October 04, 2019
Updated versions of Magnitude Simba JDBC and ODBC drivers have been released.
October 03, 2019
BigQuery Data Transfer Service now supports Search Ads 360 data transfers. Search Ad 360 transfers are in beta.
The ability to use scripting, stored procedures, and temporary tables is now in Beta.
September 27, 2019
- The maximum size of a BI Engine reservation is now 50 GB per project per location. For more information, see Capacity limits.
- We added more options for BI Engine capacity that is bundled with flat-rate pricing. For more information, see Flat-rate pricing.
September 26, 2019
BigQuery Data Transfer Service support for data & schema migration from a Teradata data warehouse to BigQuery has been upgraded. On-demand, one-time transfers are now generally available (GA). Incremental, recurring transfers are in Beta.
September 17, 2019
BigQuery is now available in the Frankfurt (europe-west3) region.
BigQuery BI Engine is now available in the Frankfurt (europe-west3) region.
September 11, 2019
Billing is now enabled for BigQuery BI Engine. You are charged according to the information on the Pricing page.
August 30, 2019
Support for copying datasets, including across regions, is now in Beta.
August 27, 2019
When you create an external table that is linked to a Google Sheets file stored in Google Drive, you can now specify an individual sheet or a cell range. You are no longer limited to the first sheet only.
August 22, 2019
Updated version of Magnitude Simba ODBC driver.
You no longer have to manually re-cluster data in clustered tables. BigQuery now performs automatic re-clustering on all clustered tables.
August 19, 2019
If you stream data into BigQuery without populating the insertId
field, you get the following higher quotas in the US
multi-region location. These higher quotas are currently in beta.
- Maximum rows per second: 1,000,000
- Maximum bytes per second: 1 GB
For more information, see the Quotas and limits page.
August 07, 2019
You can now query data in BigQuery and Cloud SQL with a federated query. Cloud SQL federated queries are now in beta.
August 05, 2019
The BigQuery Storage API now supports reading data from BigQuery Storage in ARROW format. The BigQuery Storage API is currently in beta.
July 25, 2019
Updated version of Magnitude Simba ODBC driver. This version adds support for the BigQuery Storage API. The structure of the Linux ODBC tar package has also been changed to contain both 32-bit and 64-bit driver versions.
July 19, 2019
The scheduled query feature is now Generally Available.
July 01, 2019
Support for persistent user-defined functions (UDFs) is now in Beta.
BigQuery Data Transfer Service now supports Google Merchant Center data transfers. Merchant Center transfers are in the beta stage.
June 17, 2019
Support for Amazon Redshift data warehouse migrations to BigQuery is now in Beta.
June 13, 2019
Updated version of Magnitude Simba JDBC driver.
June 10, 2019
BigQuery is moving to the Cloud console. The classic web UI remains available through June 2020. For information on migrating to the Cloud console, see Moving to the Cloud console.
June 03, 2019
Support for omitting column names in INSERT
and MERGE
statements is now in Beta.
May 28, 2019
The maximum response size for query results has increased from 128 MB to 10 GB.
May 15, 2019
Support for clustering a table by using a GEOGRAPHY
column is now Generally Available (GA).
May 13, 2019
The concurrent rate limit for on-demand, interactive queries has increased from 50 to 100 concurrent queries.
April 29, 2019
BigQuery BI Engine is now available in several additional regions:
- Los Angeles
- Montréal
- Northern Virginia
- São Paulo
- Zürich
- Hong Kong
- Mumbai
- Taiwan
- Singapore
- Sydney
For a complete list of BigQuery BI Engine locations, see Supported regions
April 22, 2019
The following quotas have changed:
- The number of copy jobs per project per day has increased from 10,000 to 100,000.
- The number of load jobs per project per day has increased from 50,000 to 100,000.
- The number of export jobs per project per day has increased from 50,000 to 100,000.
April 18, 2019
BigQuery is now available in the Osaka (asia-northeast2) region. For upcoming regional launches, see the Google Cloud blog.
April 10, 2019
BigQuery BI Engine is now in beta.
Support for clustered tables is now generally available (GA).
BigQuery standard SQL now supports AEAD encryption functions.
The BigQuery Data Transfer Service now supports transferring data into BigQuery from Amazon S3. Amazon S3 transfers are now in beta.
Third party transfers for loading data from external data sources are now in beta.
April 05, 2019
BigQuery now allows you to purchase as few as 500 slots when you enroll in a flat-rate pricing plan. You are also now able to choose either a monthly or a yearly flat-rate pricing plan. The new flat-rate pricing plans are currently in alpha.
The BigQuery Data Transfer Service is now available in beta in the Cloud console.
April 04, 2019
The BigQuery Data Transfer Service for Google Ads has been updated to support v201809 of Google Ads reports.
March 27, 2019
Combined with a special migration agent, the BigQuery Data Transfer Service now provides a database migration service from an on-premises data warehouse system to BigQuery. Support for Teradata migrations is now in beta.
March 12, 2019
Updated version of Simba ODBC and JDBC driver. The Simba JDBC driver now includes support for the BigQuery Storage API.
March 11, 2019
BigQuery is now available in the Zürich (europe-west6) region. For upcoming regional launches, see the Google Cloud blog.
March 01, 2019
The BigQuery web UI in the Cloud console now supports assigning access controls to BigQuery datasets.
When you create a scheduled query, the minimum time interval for custom schedules has changed from three hours to 15 minutes.
February 25, 2019
Support for DML MERGE statements is now generally available (GA).
You can now save query results to Google Drive by using the BigQuery web UI in the Cloud console. Saving query results to Google Drive is currently in beta.
February 22, 2019
The BigQuery Storage API is now in beta.
February 15, 2019
BigQuery is now available in the São Paulo (southamerica-east1) region. For upcoming regional launches, see the Google Cloud blog.
February 13, 2019
The following load job quota has changed:
- The 16 MB maximum block size limit for Avro files has been removed.
February 11, 2019
The following DML quotas have changed:
- The maximum number of combined UPDATE, DELETE, and MERGE statements per day per table has increased from 200 to 1,000.
- The limit of 10,000 combined UPDATE, DELETE, and MERGE statements per day per project has been removed. There is no longer a project-level limit on DML statements.
February 04, 2019
You can now use INFORMATION_SCHEMA
views to retrieve resource metadata in BigQuery. INFORMATION_SCHEMA
is currently in beta.
The Data Definition Language now supports ALTER TABLE
statements to set options on tables and views. ALTER TABLE
statements are currently in beta.
January 31, 2019
You can now try BigQuery without providing a credit card or attaching a billing account to your project by using the BigQuery sandbox. The sandbox is now in beta and is available to all Google Cloud customers.
The daily export quota has been changed to measure the actual export file size in bytes, when counted toward your daily export quota. Usage is no longer measured by rounding up to the nearest GB.
The AudienceExplorer table has been deprecated by Google Ad Manager, and has been removed from Google Ad Manager report transformations.
January 28, 2019
BigQuery is now available in the Los Angeles (us-west2) and Hong Kong (asia-east2) regions. For upcoming regional launches, see the Google Cloud blog.
January 22, 2019
BigQuery audit logging now writes enhanced messages using the BigQueryAuditMetadata
format.
December 18, 2018
BigQuery is now available in the Mumbai (asia-south1), and Montréal (northamerica-northeast1) regions. For upcoming regional launches, see the Google Cloud blog.
December 12, 2018
Raised the maximum length of standard SQL queries to 1 MB.
December 07, 2018
BigQuery is now available in the Taiwan (asia-east1), Finland (europe-north1), and Northern Virginia (us-east4) regions. For upcoming regional launches, see the Google Cloud blog.
December 05, 2018
BigQuery now supports converting Avro logical types when you load Avro data.
You can now set or update a default partition expiration time on a dataset.
December 04, 2018
Updated versions of the Simba ODBC and JDBC drivers.
November 26, 2018
Updated version of the Simba ODBC drivers.
November 13, 2018
BigQuery is now available in the Singapore (asia-southeast1) region. For upcoming regional launches, see the Google Cloud blog.
October 29, 2018
BigQuery now supports the bigquery.metadataViewer
predefined IAM role.
October 22, 2018
BigQuery is now available in the Sydney (australia-southeast1) region. For upcoming regional launches, see the Google Cloud blog.
October 17, 2018
You can now try BigQuery without providing a credit card or attaching a billing account to your project by using the BigQuery sandbox for Firebase. The sandbox is now in beta.
October 10, 2018
BigQuery is now available in the London (europe-west2) region. For upcoming regional launches, see the Google Cloud blog.
Support for tables partitioned on a DATE
or TIMESTAMP
column is now generally available (GA).
Support for using DML statements against partitioned tables is also generally available (GA).
October 09, 2018
The location is no longer required when executing jobs that reference resources outside the US
and EU
regions.
October 03, 2018
Support for Cloud Storage Transfers is now in Beta.
September 25, 2018
BigQuery standard SQL now supports the GENERATE_UUID
function.
September 20, 2018
Updated versions of Simba ODBC and JDBC drivers to support regional resources.
September 11, 2018
Scheduling queries is now in Beta.
September 10, 2018
August 27, 2018
Updated version of the Simba JDBC driver.
August 20, 2018
The NUMERIC
data type is now generally available.
BigQuery now supports loading Cloud Firestore export data and querying Cloud Firestore data stored in Cloud Storage.
July 31, 2018
The BigQuery Data Transfer Service no longer requires an enrollment step for each data source when you enable the service.
July 30, 2018
BigQuery support for clustered tables is now Beta.
July 25, 2018
BigQuery has achieved approximately 5x improvement in TPC-DS 10T power run geomean over the past 18 months.
Data Definition Language (DDL) support in BigQuery is now Generally Available (GA).
July 19, 2018
The BigQuery web UI in the Cloud console is now Beta.
BigQuery support for clustered tables is now Alpha.
July 18, 2018
Support for Google Play reports is now in Beta.
July 11, 2018
BigQuery Data Transfer Service for Google Ads is now GA (generally available).
Google Ads transfers now support API version v201806.
June 28, 2018
BigQuery standard SQL now supports the functions FROM_BASE32
and TO_BASE32
.
BigQuery standard SQL now supports templated SQL UDFs.
BigQuery standard SQL now supports the function GENERATE_TIMESTAMP_ARRAY
.
June 27, 2018
BigQuery Data Transfer Service support for YouTube Channel reports and YouTube Content Owner reports is now GA (generally available).
June 18, 2018
Loading data in Parquet format is now GA (generally available).
Loading data in ORC format is now Beta.
June 06, 2018
Updated version of the Simba ODBC drivers.
May 25, 2018
The following quotas have changed for load jobs:
- The row and cell size limit for CSV files has changed from 10 MB to 100 MB.
- The row size limit for JSON files has changed from 10 MB to 100 MB.
May 15, 2018
Support for the NUMERIC
data type is now in Beta.
May 04, 2018
The following quotas have changed:
- The Maximum number of partitions per partitioned table has changed from 2,500 to 4,000.
- The Maximum number of combined UPDATE, DELETE, and MERGE DML statements per day per table has changed from 96 to 200.
May 01, 2018
Updated version of the Simba JDBC driver.
When you create a Google Ads transfer, the maximum value for the refresh window is now 7 days. Existing transfers are unaffected by this change.
April 17, 2018
BigQuery is now available in the Tokyo (asia-northeast1) region.
BigQuery now includes bigquery.jobs.listAll
permissions. bigquery.jobs.list
still allows users to list all jobs in a project, but details and metadata are redacted for jobs submitted by other users.
April 13, 2018
By default, the number of export jobs per project per day has increased from 1,000 to 50,000.
April 10, 2018
Support for DML MERGE
statements is now in Beta.
April 06, 2018
Google Ad Manager transfers now support API version v201802.
March 30, 2018
BigQuery standard SQL now supports: the ERROR function to return errors and the SAFE prefix to suppress errors.
BigQuery now allows you to monitor slot usage by job type in Stackdriver.
March 05, 2018
BigQuery data definition language now supports creating a table from the result of a query.
BigQuery support for using DML statements with partitioned tables is now in Beta.
March 01, 2018
Support for loading Parquet files is now in Beta.
Updated version of Simba JDBC driver.
February 16, 2018
Google Ads transfers now support manager accounts (MCCs).
Google Ads transfers now support API version 201710.
February 15, 2018
BigQuery Data Transfer Service run notifications are now in Alpha.
February 08, 2018
BigQuery support for tables partitioned using a TIMESTAMP
or DATE
column is now in Beta.
Several API methods will begin including a location field in their responses: jobs.insert
, jobs.get
, jobs.cancel
, jobs.query
, jobs.getQueryResults
, datasets.list
, and jobs.list
. The location field is currently not required in requests.
For example, the id
property will include a location in the following format: "id": "[PROJECT_ID]:[LOCATION].[JOB_ID]"
. The "id"
property should continue to be treated as opaque. Also, the "jobReference"
property will begin including a location field in the following format:
"jobReference": {
"jobId": "[JOB_ID]",
"location": "[LOCATION]",
...
Added documentation on manually changing table schemas.
January 17, 2018
BigQuery data definition language support is now in Beta.
January 08, 2018
Google Ads transfers now support the following reports:
- Age Range Performance Report
- Criteria Performance Report
- Gender Performance Report
- Bid Goal Performance Report
- Budget Performance Report
- Parental Status Performance Report
For more information, see Google Ads Report Transformation.
January 05, 2018
ISOWEEK
and ISOYEAR
are now supported by the following functions: DATE_TRUNC
, DATETIME_TRUNC
, TIMESTAMP_TRUNC
, DATE_DIFF
, and DATETIME_DIFF
.
December 11, 2017
You can now exclude removed or deleted items when setting up a Google Ads transfer.
You can now check the Configure Jobs option to allow the BigQuery Data Transfer Service to manage YouTube Channel and Content Owner reporting jobs for you.
Backfill requests for Google Ads do not overwrite Match Tables.
December 08, 2017
Updated versions of Simba ODBC and JDBC releases.
December 04, 2017
The BigQuery web UI supports additional dimensions for filtering your query history.
November 30, 2017
INTERSECT DISTINCT
and EXCEPT DISTINCT
are now supported set operators in standard SQL.
FARM_FINGERPRINT
is now a supported function in legacy SQL.
WEEK(<WEEKDAY>)
is now a valid date part for the EXTRACT function.
There is now a quota of 300 concurrent API requests per user.
IAM custom roles are now available for BigQuery.
The _PARTITIONTIME
pseudo column is now supported when querying data using a wildcard table over partitioned tables.
November 16, 2017
BigQuery Data Transfer Service is now generally available.
November 14, 2017
High compute queries are being phased out so you are no longer required to opt-in. All high compute queries under tier 100 are billed as tier 1. Running a high compute query does not require setting a threshold flag.
All queries above tier 100 will fail with a RESOURCES_EXCEEDED_PER_BYTE
error unless the query is running in a reserved instance. Existing users who have run a high compute query higher than tier 1