BigQuery release notes

This page documents production updates to BigQuery. We recommend that BigQuery developers periodically check this list for any new announcements. BigQuery automatically updates to the latest release and cannot be downgraded to a previous version.

You can see the latest product updates for all of Google Cloud on the Google Cloud page, browse and filter all release notes in the Google Cloud console, or programmatically access release notes in BigQuery.

To get the latest product updates delivered to you, add the URL of this page to your feed reader, or add the feed URL directly: https://cloud.google.com/feeds/bigquery-release-notes.xml

March 04, 2024

A weekly digest of client library updates from across the Cloud SDK.

Node.js

Changes for @google-cloud/bigquery

7.5.0 (2024-02-21)

Features

You can now selectively refresh the metadata cache for BigLake tables by using the BQ.REFRESH_EXTERNAL_METADATA_CACHE system procedure. This feature is generally available (GA).

February 29, 2024

The following BigQuery cross-cloud features are now generally available (GA):

The consolidated SQL translator API combines the interactive and batch translator into a single workflow, improving the efficiency and stability of your translation jobs created using the API. This feature is available in preview.

February 28, 2024

Materialized views can now reference logical views. This feature is in preview.

The ability to perform anomaly detection with BigQuery ML multivariate time series (ARIMA_PLUS_XREG) models is now in preview. This feature enables you to detect anomalies in historical time series data or in new data with multiple feature columns. Try this new feature by using the Perform anomaly detection with a multivariate time-series forecasting model tutorial.

The following statements are now generally available (GA) with billing enabled:

  • CREATE TABLE AS SELECT
  • CREATE TABLE IF NOT EXISTS AS SELECT
  • CREATE OR REPLACE TABLE AS SELECT
  • INSERT INTO SELECT

These statements let you filter data from files in Amazon S3 and Azure Blob Storage before transferring results into BigQuery tables.

February 27, 2024

You can now use time series and range functions to support time series analysis. This feature is in preview.

You can now use data manipulation language (DML) statements to efficiently delete entire partitions. If a DELETE statement targets all rows in a partition, then the entire partition is deleted without scanning bytes or consuming slots. This feature is now generally available (GA).

February 26, 2024

A weekly digest of client library updates from across the Cloud SDK.

Java

Changes for google-cloud-bigquery

2.38.0 (2024-02-22)

Features
  • Add MetadataCacheStatistics to Job QueryStatistics (#3133) (f3f387b)
Dependencies
  • Update dependency com.google.apis:google-api-services-bigquery to v2-rev20240211-2.0.0 (#3152) (e5d6888)
  • Update github/codeql-action action to v2.24.3 (#3148) (a0a7b01)
  • Update github/codeql-action action to v2.24.3 (#3150) (042fcf0)
  • Update github/codeql-action action to v2.24.4 (#3161) (531b1a0)

The BigQuery Data Transfer Service can now transfer data from the following data sources:

Transfers from these data sources are supported in preview.

The following SQL features are now generally available (GA):

The GROUP BY ALL clause, which groups rows by inferring grouping keys from the SELECT items, is now in preview.

February 22, 2024

The following BigQuery text embedding features are now generally available (GA):

February 19, 2024

A weekly digest of client library updates from across the Cloud SDK.

Go

Changes for bigquery/storage/apiv1beta1

1.59.1 (2024-02-12)

Bug Fixes
  • bigquery: Align return time.Time values to UTC (#9411) (4ac005d)

Java

Changes for google-cloud-bigquery

2.37.2 (2024-02-14)

Dependencies
  • Update dependency com.google.api.grpc:proto-google-cloud-bigqueryconnection-v1 to v2.37.0 (#3132) (3a1efc2)
  • Update dependency com.google.apis:google-api-services-bigquery to v2-rev20240203-2.0.0 (#3126) (5e28419)
  • Update dependency com.google.cloud:google-cloud-datacatalog-bom to v1.41.0 (#3135) (9ab79ec)
  • Update dependency com.google.cloud:google-cloud-shared-dependencies to v3.25.0 (#3140) (e61a7bc)
  • Update github/codeql-action action to v2.24.1 (#3139) (4b3a429)

February 15, 2024

The following Generative AI features are now generally available (GA):

After you run a query in the query editor, in the Chart tab, you can now see a visualization of your query results. This feature is generally available (GA).

February 12, 2024

A weekly digest of client library updates from across the Cloud SDK.

Node.js

Changes for @google-cloud/bigquery

7.4.0 (2024-02-06)

Features
Bug Fixes
  • Prefer usage of projectId from the Dataset (#1326) (9e85219)

Go

Changes for bigquery/storage/apiv1beta1

1.59.0 (2024-02-06)

Features
  • bigquery: Add ExportDataStatstics to QueryStatistics (#9371) (261c8d9)
  • bigquery: Switch all timestamp representations to int64 usec (#9368) (8c1fb7d)
Bug Fixes
  • bigquery/storage/managedwriter: Resolve data races (#9360) (fa31ec0)
  • bigquery: Enable universe domain resolution options (fd1d569)
  • bigquery: Support more timestamp formats for query param (#9236) (cc98509), refs #9221

Java

Changes for google-cloud-bigquery

2.37.1 (2024-02-06)

Features
Dependencies
  • Update actions/upload-artifact action to v4.3.1 (#3121) (3abdc70)
  • Update dependency com.google.apis:google-api-services-bigquery to v2-rev20240124-2.0.0 (#3104) (6eff68e)
  • Update dependency com.google.cloud:google-cloud-shared-dependencies to v3.24.0 (#3109) (5ad778c)
  • Update dependency org.graalvm.buildtools:junit-platform-native to v0.10.0 (#3110) (3f8e8d1)
  • Update dependency org.graalvm.buildtools:native-maven-plugin to v0.10.0 (#3111) (2858e96)
  • Update dependency org.junit.vintage:junit-vintage-engine to v5.10.2 (#3119) (4b4fdd8)
  • Update github/codeql-action action to v2.23.2 (#3102) (2cc545e)
  • Update github/codeql-action action to v2.24.0 (#3114) (01f0405)

Python

Changes for google-cloud-bigquery

3.17.2 (2024-01-30)

Bug Fixes
  • Change load_table_from_json autodetect logic (#1804) (6249032)
Documentation
  • Update to use API (#1781) (81563b0)
  • Update client_query_destination_table.py sample to use query_and_wait (#1783) (68ebbe1)
  • Update query_external_sheets_permanent_table.py to use query_and_wait API (#1778) (a7be88a)
  • Update sample for query_to_arrow to use query_and_wait API (#1776) (dbf10de)
  • Update the query destination table legacy file to use query_and_wait API (#1775) (ef89f9e)
  • Update to use query_and_wait in client_query_w_positional_params.py (#1786) (410f71e)
  • Update to use query_and_wait in samples/client_query_w_timestamp_params.py (#1785) (ba36948)
  • Update to_geodataframe to use query_and_wait functionality (#1800) (1298594)

February 08, 2024

Custom data masking is now generally available (GA). You can define custom masking routines for custom masking capabilities such as salt based hash. The feature is available on the Enterprise Plus edition.

BigQuery now offers entity resolution. This feature lets users match records across datasets even when a common identifier is missing. It utilizes an identity provider for this process; BigQuery supports LiveRamp and provides a framework for other identity providers to offer similar services. This feature is generally available (GA).

February 07, 2024

You can now view query plans to see details of SQL pushdowns in federated queries. This feature is now generally available.

February 06, 2024

Billing for Spark stored procedures begins on March 12, 2024. Until that date, Spark stored procedures are offered at no extra cost.

January 31, 2024

The following information schema views display the history of configuration changes to the options of your organization and projects:

This feature is now in preview.

BigQuery now supports vector search and vector indexes. These features are in preview.

You can use the VECTOR_SEARCH function to search embeddings in order to identify semantically similar entities.

You can use vector indexes to make VECTOR_SEARCH more efficient, with the trade-off of returning more approximate results.

Try the new vector search and vector index capabilities with the Search embeddings with vector search tutorial.

January 29, 2024

A weekly digest of client library updates from across the Cloud SDK.

Java

Changes for google-cloud-bigquery

2.37.0 (2024-01-25)

Features
Dependencies
  • Update actions/upload-artifact action to v4.1.0 (#3071) (3fbb2bb)
  • Update actions/upload-artifact action to v4.2.0 (#3081) (af81354)
  • Update actions/upload-artifact action to v4.3.0 (#3091) (f4411b0)
  • Update arrow.version to v15 (#3084) (4d4cbae)
  • Update dependency com.google.api.grpc:proto-google-cloud-bigqueryconnection-v1 to v2.35.0 (#3066) (48cdaa8)
  • Update dependency com.google.api.grpc:proto-google-cloud-bigqueryconnection-v1 to v2.36.0 (#3093) (24456a3)
  • Update dependency com.google.apis:google-api-services-bigquery to v2-rev20240105-2.0.0 (#3073) (f371d67)
  • Update dependency com.google.cloud:google-cloud-datacatalog-bom to v1.39.0 (#3067) (6ff4f04)
  • Update dependency com.google.cloud:google-cloud-datacatalog-bom to v1.40.0 (#3094) (110bcc5)
  • Update dependency com.google.cloud:google-cloud-shared-dependencies to v3.22.0 (#3080) (a5b119c)
  • Update dependency com.google.cloud:google-cloud-shared-dependencies to v3.23.0 (#3096) (0933b34)
  • Update dependency com.google.oauth-client:google-oauth-client-java6 to v1.35.0 (#3078) (2614df2)
  • Update dependency com.google.oauth-client:google-oauth-client-jetty to v1.35.0 (#3079) (f03c4fc)
  • Update github/codeql-action action to v2.23.0 (#3061) (0fbdfba)
  • Update github/codeql-action action to v2.23.1 (#3077) (e3f417c)

Python

Changes for google-cloud-bigquery

3.17.1 (2024-01-24)

Bug Fixes
  • Add pyarrow.large_strign to the _ARROW_SCALAR_IDS_TO_BQ map (#1796) (b402a6d)
  • Retry 'job exceeded rate limits' for DDL queries (#1794) (39f33b2)

3.17.0 (2024-01-24)

Features
Bug Fixes
  • query_and_wait now retains unknown query configuration _properties (#1793) (4ba4342)
  • Raise ValueError in query_and_wait with wrong job_config type (4ba4342)
Documentation
  • Remove unused query code sample (#1769) (1f96439)
  • Update snippets.py to use query_and_wait (#1773) (d90602d)
  • Update multiple samples to change query to query_and_wait (#1784) (d1161dd)
  • Update the query with no cache sample to use query_and_wait API (#1770) (955a4cd)
  • Updates query to query and wait in samples/desktopapp/user_credentials.py (#1787) (89f1299)

You can now use tags on BigQuery tables to conditionally grant or deny access with Identity and Access Management (IAM) policies. This feature is in preview.

Cloud console updates: You can now sort query results by column. Click Open sort menu next to the column name and select a sort order. This feature is generally available (GA).

January 24, 2024

BigQuery now natively supports the Delta Lake format for Amazon S3 and Azure tables. This feature is now in preview.

BigQuery ML has added a new residual column to the output of the ML.EXPLAIN_FORECAST function for ARIMA_PLUS and ARIMA_PLUS_XREG models. The residual column contains the difference between the actual time series and the fitted time series for the historical data. This lets you compare the modeled historical data that is returned in the other output columns of ML.EXPLAIN_FORECAST with the actual historical data.

To improve BigQuery ML training performance, the APPROX_GLOBAL_FEATURE_CONTRIB argument now defaults to TRUE when you set the ENABLE_GLOBAL_EXPLAIN argument to TRUE, and you set the NUM_PARALLEL_TREE argument to greater than 10 for boosted tree models or greater than 50 for random forest models.

BigQuery now supports the ST_LINEINTERPOLATEPOINT geography function, which gets a point at a specific fraction in a linestring. This feature is generally available (GA).

January 22, 2024

A weekly digest of client library updates from across the Cloud SDK.

Go

Changes for bigquery/storage/apiv1beta1

1.58.0 (2024-01-18)

Features
  • bigquery/storage/managedwriter: Improve schema comparison stability (#9241) (faccb68)
  • bigquery/storage: Add ability to request compressed ReadRowsResponse rows (c3f1174)
  • bigquery: Add DataGovernanceType to routines (#8990) (57491ae)
  • bigquery: Add schema support for RANGE type (#9050) (477ccee)
  • bigquery: Add support for AllowNonIncrementalDefinition and Staleness on MaterializedView (#8673) (6ec2bb2)
  • bigquery: Add table resource tags support (#9084) (3569cc2)
  • bigquery: Expose query id on row iterator if available (#9224) (bbff8ac)
Documentation
  • bigquery/storage/managedwriter: Add documentation for ingesting JSON to a ManagedStream (#9137) (cae54ed)
  • bigquery: Update package docs to show query parameterization (#8965) (cc76ab3)

BigQuery is now available in the Berlin (europe-west10) region.

January 16, 2024

You can now use cross-cloud joins to run queries that span both Google Cloud and BigQuery Omni regions. This feature is in preview.

January 15, 2024

A weekly digest of client library updates from across the Cloud SDK.

Java

Changes for google-cloud-bigquery

2.36.0 (2024-01-10)

Features
Bug Fixes
  • bigquery.create NullPointerException when job already exists (#3035) (38191b1)
Dependencies
  • Update actions/upload-artifact action to v4 (#3055) (7d76100)
  • Update arrow.version to v14.0.2 (#3050) (b0dc33a)
  • Update dependency com.google.api.grpc:proto-google-cloud-bigqueryconnection-v1 to v2.34.0 (#3033) (a710632)
  • Update dependency com.google.cloud:google-cloud-shared-dependencies to v3.21.0 (#3060) (78995c4)
  • Update github/codeql-action action to v2.22.11 (#3002) (52d5e97)

Python

Changes for google-cloud-bigquery

3.16.0 (2024-01-12)

Features
Bug Fixes

3.15.0 (2024-01-09)

Features
  • Support JSON type in insert_rows and as a scalar query parameter (#1757) (02a7d12)
  • Support RANGE in schema (#1746) (8585747)
Bug Fixes
  • Deserializing JSON subfields within structs fails (#1742) (0d93073)
  • Due to upstream change in dataset, updates expected results (#1761) (132c14b)
  • Load_table_from_dataframe for higher scale decimal (#1703) (b9c8be0)
  • Updates types-protobuf version for mypy-samples nox session (#1764) (c0de695)
Performance Improvements
  • DB-API uses more efficient query_and_wait when no job ID is provided (#1747) (d225a94)

January 02, 2024

Analytics Hub listings can now include data encrypted with customer-managed encryption keys (CMEK). This feature is generally available (GA).

December 18, 2023

A weekly digest of client library updates from across the Cloud SDK.

Python

Changes for google-cloud-bigquery

3.14.1 (2023-12-13)

Bug Fixes
  • Add missing handler for deserializing json value (#1587) (09017a9)

3.14.0 (2023-12-08)

Features
  • Add Client.query_and_wait which directly returns a RowIterator of results (#1722) (89a647e)
  • Add job_id, location, project, and query_id properties on RowIterator (#1733) (494f275)
  • Add job_timeout_ms to job configuration classes (#1675) (84d64cd)
  • Add support dataset.max_time_travel_hours (#1683) (f22eff2)
  • Add support for Dataset.isCaseInsensitive (#1671) (386fa86)
  • Add support for Python 3.12 (#1736) (3c0976a)
  • Removed pkg_resources from all test files and moved importlib into pandas extra (#1726) (1f4ebb1)
  • Support data_governance_type (#1708) (eff365d)
Bug Fixes
  • load_table_from_dataframe now assumes there may be local null values (#1735) (f05dc69)
  • Ensure query job retry has longer deadline than API request deadline (#1734) (5573579)
  • Keep RowIterator.total_rows populated after iteration (#1748) (8482f47)
  • Move grpc, proto-plus and protobuf packages to extras (#1721) (5ce4d13)
Performance Improvements
  • Use the first page a results when query(api_method="QUERY") (#1723) (6290517)

Operational Health administrative resource charts are now in preview. You can use charts to view slot and shuffle usage, job concurrency, errors, and other metrics.

December 14, 2023

The BigQuery Data Transfer Service now supports federated workforce identities when creating a data transfer from most data sources. This feature is generally available (GA).

The Apache Hive connector is now generally available (GA) for data analytics pipeline migration.

December 12, 2023

The following BigQuery cross-cloud features are now in preview:

December 11, 2023

A weekly digest of client library updates from across the Cloud SDK.

Java

Changes for google-cloud-bigquery

2.35.0 (2023-12-01)

Features
Bug Fixes
Dependencies
  • Update actions/github-script action to v7 (#3001) (d1bdeab)
  • Update actions/setup-java action to v4 (#3018) (14ed571)
  • Update arrow.version to v14 (#3023) (759fd64)
  • Update dependency com.google.apis:google-api-services-bigquery to v2-rev20231111-2.0.0 (#3020) (ef48002)
  • Update dependency com.google.cloud:google-cloud-shared-dependencies to v3.20.0 (#3019) (0293edb)

December 07, 2023

The following BigQuery ML data preprocessing features are now in preview:

  • The ML.TRANSFORM function, which you can use to preprocess feature data. This function processes input data by applying the data transformations captured in the TRANSFORM clause of an existing model.
  • Transform-only models which you can use to apply preprocessing functions to input data and return the preprocessed data. Transform-only models decouple data preprocessing from model training, making it easier for you to capture and reuse a set of data preprocessing rules.

December 04, 2023

A weekly digest of client library updates from across the Cloud SDK.

Python

Changes for google-cloud-bigquery

3.14.0rc0 (2023-11-30)

Features
  • Add job_id, location, project, and query_id properties on RowIterator (#1733) (494f275)
  • Add job_timeout_ms to job configuration classes (#1675) (84d64cd)
  • Add support dataset.max_time_travel_hours (#1683) (f22eff2)
  • Add support for Dataset.isCaseInsensitive (#1671) (386fa86)
  • Removed pkg_resources from all test files and moved importlib into pandas extra (#1726) (1f4ebb1)
  • Support data_governance_type (#1708) (eff365d)
Bug Fixes
  • load_table_from_dataframe now assumes there may be local null values (#1735) (f05dc69)
  • Ensure query job retry has longer deadline than API request deadline (#1734) (5573579)
  • Move grpc, proto-plus and protobuf packages to extras (#1721) (5ce4d13)
Performance Improvements
  • Use the first page a results when query(api_method="QUERY") (#1723) (6290517)

November 30, 2023

You can use configuration YAML files to transform SQL code when you translate SQL queries from your source database. Configuration YAML files can be used with the batch SQL translator, the interactive SQL translator, and the batch translation Python client. This feature is now generally available (GA).

The slot estimator now supports project level cost-optimal commitment and autoscale recommendations for on-demand workloads. This feature is now in preview.

November 16, 2023

The following BigQuery ML features for Vertex AI large language models (LLMs) are now generally available (GA):

  • The SQL syntax for remote models has been updated to provide access to all text generation and text embedding LLMs (for example, text-bison-32k and textembedding-gecko-multilingual) and also to provide support for different LLM versions.

  • Region support for text-bison* LLM models has been expanded to include the following locations in addition to us and us-central1:

    • asia-northeast3
    • asia-southeast1
    • eu
    • europe-west1
    • europe-west2
    • europe-west3
    • europe-west4
    • europe-west9
    • us-west4

November 14, 2023

You can now see query performance insights about partition skew. This feature is in preview.

November 13, 2023

A weekly digest of client library updates from across the Cloud SDK.

Java

Changes for google-cloud-bigquery

2.34.2 (2023-11-07)

Dependencies
  • Update dependency com.google.api.grpc:proto-google-cloud-bigqueryconnection-v1 to v2.32.0 (#2989) (47a61a7)
  • Update dependency com.google.cloud:google-cloud-datacatalog-bom to v1.36.0 (#2990) (81c0727)

2.34.1 (2023-11-06)

Dependencies
  • Update dependency com.google.cloud:google-cloud-shared-dependencies to v3.19.0 (#2986) (0d400da)
  • Update dependency org.checkerframework:checker-compat-qual to v2.5.6 (#2982) (c137f1f)
  • Update dependency org.junit.vintage:junit-vintage-engine to v5.10.1 (#2984) (a64b91c)
  • Update github/codeql-action action to v2.22.5 (#2975) (0b88846)

The following BigQuery ML point-in-time lookup functions are now generally available (GA). These functions let you specify a point-in-time cutoff when retrieving features for training a model or running inference, in order to avoid data leakage.

The following AI features in BigQuery are now in preview:

November 07, 2023

The batch SQL translator has added enhancements when viewing SQL translation reports. You can now see a log summary of all issues during a translation job, as well as a code tab that displays a side-by-side comparison of your input and output files from a translation. This feature is in preview.

November 06, 2023

A weekly digest of client library updates from across the Cloud SDK.

Go

Changes for bigquery/storage/apiv1beta1

1.57.1 (2023-11-01)

Bug Fixes

1.57.0 (2023-10-30)

Features
  • bigquery/biglake: Promote to GA (e864fbc)
  • bigquery/storage/managedwriter: Support default value controls (#8686) (dfa8e22)
  • bigquery: Expose Apache Arrow data through ArrowIterator (#8506) (c8e7692), refs #8100
  • bigquery: Introduce query preview features (#8653) (f29683b)
Bug Fixes
  • bigquery: Handle storage read api Recv call errors (#8666) (c73963f)
  • bigquery: Update golang.org/x/net to v0.17.0 (174da47)
  • bigquery: Update grpc-go to v1.56.3 (343cea8)
  • bigquery: Update grpc-go to v1.59.0 (81a97b0)

Python

Changes for google-cloud-bigquery

3.13.0 (2023-10-30)

Features
  • Add Model.transform_columns property (#1661) (5ceed05)
  • Add support for dataset.default_rounding_mode (#1688) (83bc768)
Bug Fixes
Documentation
  • Remove redundant bigquery_update_table_expiration code sample (#1673) (2dded33)
  • Revised create_partitioned_table sample (#1447) (40ba859)
  • Revised relax column mode sample (#1467) (b8c9276)

The BigQuery Data Transfer Service can now transfer campaign reporting and configuration data from Display & Video 360 into BigQuery. This feature is in preview.

The following BigQuery ML features for time series forecasting are now generally available (GA):

November 02, 2023

BigQuery now supports text analysis configuration options for the following:

BigQuery now also provides support for the following advanced processing functions:

These features are now in preview.

November 01, 2023

The following INFORMATION_SCHEMA views that show metadata for table storage usage are now in preview:

You can now use cached results from the same query issued by other users in the same project when you use Enterprise or Enterprise Plus edition. This feature is generally available (GA).

October 31, 2023

BigQuery support for change data capture (CDC) by processing and applying streamed changes in real-time to existing data using the BigQuery Storage Write API is now generally available (GA).

You can now use data manipulation language (DML) to modify rows that have been recently written by the Storage Write API. This feature is in preview.

October 30, 2023

The BigQuery Data Transfer Service can now transfer data from Azure Blob Storage into BigQuery. This feature is now generally available (GA).

A weekly digest of client library updates from across the Cloud SDK.

Java

Changes for google-cloud-bigquery

2.34.0 (2023-10-26)

Features
  • Add BigLakeConfiguration Property in StandardTableDefinition.java (#2916) (1d660fa)
  • Add support for Dataset property storageBillingModel (#2913) (f452cf4)
  • Add support for preview features (#2923) (113b8f2)
Dependencies
  • Update actions/checkout action to v4.1.1 (#2950) (c556c18)
  • Update dependency com.google.api.grpc:proto-google-cloud-bigqueryconnection-v1 to v2.30.0 (#2942) (e760fca)
  • Update dependency com.google.api.grpc:proto-google-cloud-bigqueryconnection-v1 to v2.31.0 (#2967) (7ed55b5)
  • Update dependency com.google.apis:google-api-services-bigquery to v2-rev20231008-2.0.0 (#2946) (3d0da5b)
  • Update dependency com.google.cloud:google-cloud-datacatalog-bom to v1.34.0 (#2943) (18162c3)
  • Update dependency com.google.cloud:google-cloud-datacatalog-bom to v1.35.0 (#2968) (219db2c)
  • Update dependency com.google.cloud:google-cloud-shared-dependencies to v3.18.0 (#2955) (1ee18eb)
  • Update dependency org.graalvm.buildtools:junit-platform-native to v0.9.28 (#2956) (b03effd)
  • Update dependency org.graalvm.buildtools:native-maven-plugin to v0.9.28 (#2957) (6465e41)
  • Update github/codeql-action action to v2.22.2 (#2944) (f584e59)
  • Update github/codeql-action action to v2.22.3 (#2954) (1b2bc18)
  • Update github/codeql-action action to v2.22.4 (#2958) (de9bcee)
  • Update ossf/scorecard-action action to v2.3.1 (#2960) (855e698)

The administrative resource charts now supports the following features in preview:

October 23, 2023

Custom data masking now supports an expanded list of functions, including SHA hash functions with salt. This feature is in preview.

October 19, 2023

Stored procedures for Apache Spark are now available without enrollment. This feature is in preview.

October 18, 2023

The BigQuery migration assessment is now available for Apache Hive in preview. You can use this feature to assess the complexity of migrating data from your Apache Hive data warehouse to BigQuery.

October 16, 2023

A weekly digest of client library updates from across the Cloud SDK.

Java

Changes for google-cloud-bigquery

2.33.2 (2023-10-11)

Bug Fixes
Dependencies
  • Update dependency com.google.api.grpc:proto-google-cloud-bigqueryconnection-v1 to v2.29.0 (#2911) (052f5c2)
  • Update dependency com.google.apis:google-api-services-bigquery to v2-rev20230925-2.0.0 (#2921) (f0fb64f)
  • Update dependency com.google.cloud:google-cloud-datacatalog-bom to v1.33.0 (#2912) (e053494)
  • Update dependency com.google.cloud:google-cloud-shared-dependencies to v3.17.0 (#2931) (25a94f1)
  • Update github/codeql-action action to v2.22.0 (#2926) (33ce4ae)
  • Update github/codeql-action action to v2.22.1 (#2934) (7ae7b99)
  • Update ossf/scorecard-action action to v2.3.0 (#2927) (93bfd8e)

You can now use DLP functions to support encryption and decryption between BigQuery and DLP, using AES-SIV. This feature is in preview.

October 12, 2023

The following geography functions are now generally available (GA):

  • ST_LINESUBSTRING: Gets a segment of a single linestring at a specific starting and ending fraction.
  • ST_HAUSDORFFDISTANCE: Gets the discrete Hausdorff distance between two geometries.

October 09, 2023

A weekly digest of client library updates from across the Cloud SDK.

Go

Changes for bigquery/storage/apiv1beta1

1.56.0 (2023-10-05)

Features
  • bigquery/analyticshub: Add Subscription resource and RPCs (#8612) (9992249)
  • bigquery: Add external dataset reference (#8545) (1001acf)
  • bigquery: Add media options to LoadConfig (#8640) (62baf56)
Bug Fixes
  • bigquery/storage/managedwriter: Automatic retry for multiplex test (#8601) (6ef1945)
  • bigquery: Dependency detection on proto conversion (#8566) (763ab5d)
Documentation
  • bigquery/datatransfer: Update transferConfig.name description to indicate that it supports both formats (0449518)

Python

Changes for google-cloud-bigquery

3.12.0 (2023-10-02)

Features
  • Add Dataset.storage_billing_model setter, use client.update_dataset(ds, fields=["storage_billing_model"]) to update (#1643) (5deba50)
  • Search statistics (#1616) (b930e46)
  • Widen retry predicate to include ServiceUnavailable (#1641) (3e021a4)
Bug Fixes
  • Allow storage_billing_model to be explicitly set to None to use project default value (#1665) (514d3e1)
  • Relax timeout expectations (#1645) (1760e94)
  • Use isinstance() per E721, unpin flake8 (#1659) (54a7769)
Documentation

Queries now support additional ways to work with grouping sets, which include:

This feature is in preview.

Adding descriptions to the columns of a view is now generally available (GA). Use the CREATE VIEW or ALTER COLUMN DDL statements to add descriptions.

BigQuery is now available in the Dammam (me-central2) region.

BigQuery ML is now available in the Dammam (me-central2) region.

BigQuery Data Transfer Service is now available in the Dammam (me-central2) region.

October 05, 2023

The BigQuery migration assessment is now available for Snowflake in preview. You can use this feature to assess the complexity of migrating data from your Snowflake data warehouse to BigQuery.

October 04, 2023

You can now copy tables across regions. This feature is now in preview.

October 03, 2023

October 02, 2023

A weekly digest of client library updates from across the Cloud SDK.

Node.js

Changes for @google-cloud/bigquery

7.3.0 (2023-09-28)

Features
Bug Fixes
  • Avoid TypeError if resp is undefined (#1273) (ff51c1d)
  • Updated types from API discovery doc (#1284) (1d8a2b7)
  • Updating type for test blocking dependency updates (#1282) (1dbe0fe)

Java

Changes for google-cloud-bigquery

2.33.1 (2023-09-28)

Bug Fixes
  • Dry run NPE when there is no query parameters (#2899) (8f85a4d)

2.33.0 (2023-09-27)

Features
Bug Fixes
  • Update samples snippet to write to BYTES instead of ARRAY<BYTES> (#2876) (7e040e9)
Dependencies
  • Update actions/checkout action (#2893) (e3655af)
  • Update dependency com.google.cloud:google-cloud-shared-dependencies to v3.16.1 (#2892) (e1d9871)
  • Update dependency org.graalvm.buildtools:junit-platform-native to v0.9.27 (#2885) (2237ca2)
  • Update dependency org.graalvm.buildtools:native-maven-plugin to v0.9.27 (#2886) (539b4e6)
  • Update github/codeql-action action to v2.21.4 (#2829) (599e3b3)
  • Update github/codeql-action action to v2.21.8 - abandoned (#2897) (ab4e1d0)
  • Update github/codeql-action action to v2.21.8 (#2889) (b568026)
  • Update github/codeql-action action to v2.21.9 (#2901) (33a729f)

BigQuery native integration in Looker Studio enables monitoring features for Looker Studio queries, improves query performance, and supports many BigQuery features. This feature is in preview.

September 29, 2023

As a BigQuery administrator, to monitor your organization's slots utilization and BigQuery jobs' performance over time, use can now use administrative query inspector. This feature is now generally available.

September 28, 2023

The following BigQuery ML point-in-time lookup functions are now in preview. These functions let you specify a point-in-time cutoff when retrieving features for training a model or running inference, in order to avoid data leakage.

You can now use IAM conditions to control access to BigQuery resources. This feature is in preview.

September 25, 2023

Materialized views over BigLake metadata cache-enabled tables can reference structured data stored in Cloud Storage. These materialized views function like materialized views over BigQuery-managed storage tables, including the benefits of automatic refresh and smart tuning. This feature is now generally available (GA).

Authorized stored procedures are now generally available (GA). This feature lets you share stored procedures with users or groups without giving them direct access to the underlying tables.

Support for Google AdWords is now deprecated by the BigQuery Data Transfer Service. For information about transfers from Google Ads, see Google Ads transfers.

September 20, 2023

The maximum number of rows for results returned in Connected Sheets has increased as follows:

  • Pivot tables increased from 30,000 to 50,000 rows
  • Data extracts increased from 25,000 to 50,000 rows

September 18, 2023

A weekly digest of client library updates from across the Cloud SDK.

Go

Changes for bigquery/storage/apiv1beta1

1.55.0 (2023-09-11)

Features
  • bigquery/datapolicies: Support using custom UDF in the data policy (20725c8)
  • bigquery: Set PreferredMinStreamCount when creating read session (#8476) (22e095a), refs #8432
Bug Fixes
  • bigquery/storage/managedwriter: Address possible deadlocks (#8507) (48b08bf)
  • bigquery: Field descriptor proto name should not be lowercase (#8495) (4287e4b)
  • bigquery: Value for datasetID on foreign keys (#8447) (fa6e827), refs #8442

Java

Changes for google-cloud-bigquery

2.32.0 (2023-09-14)

Features
  • Add support for converting interval fields to threeten PeriodDuration (#2838) (2294c2f)
  • Add support for ExternalDatasetReference (#2871) (bbb86fd)
Dependencies
  • Update actions/checkout action to v4 (#2862) (902e9b9)
  • Update actions/upload-artifact action to v3.1.3 (#2867) (cbbf0fb)
  • Update dependency com.google.api.grpc:proto-google-cloud-bigqueryconnection-v1 to v2.26.0 (#2873) (6196625)
  • Update dependency com.google.cloud:google-cloud-datacatalog-bom to v1.30.0 (#2874) (6cafedf)
  • Update dependency com.google.cloud:google-cloud-shared-dependencies to v3.15.0 (#2870) (f24439b)
  • Update dependency org.graalvm.buildtools:junit-platform-native to v0.9.26 (#2868) (d01031c)
  • Update dependency org.graalvm.buildtools:native-maven-plugin to v0.9.26 (#2869) (edd7141)

The BigQuery Data Transfer Service now supports transfers from Search Ads 360 using the new Search Ads 360 reporting API. This feature is in preview. Customers with existing Search Ads 360 transfers should migrate their workflows to be compatible with the new Search Ads 360. The BigQuery Data Transfer Service will stop its support for the old Search Ads 360 reporting API on May 31st, 2024.

September 13, 2023

You can now create a federated dataset in BigQuery that federates to an existing database in AWS Glue. This feature is in preview.

September 12, 2023

The array subscript operator now returns a value in an array directly by index. Previously, only offset and ordinal were available. This feature is generally available (GA).

The struct subscript operator has been added. With this operator, you can access a STRUCT field by index, offset, or ordinal. This feature is generally available (GA).

September 11, 2023

A weekly digest of client library updates from across the Cloud SDK.

Java

Changes for google-cloud-bigquery

2.31.2 (2023-09-05)

Bug Fixes

August 30, 2023

You can now use EXPORT DATA statements to directly export BigQuery data to Bigtable. This feature is in preview.

August 29, 2023

Data clean rooms is now in preview. Data clean rooms provide a secure environment in which multiple parties can share, join, and analyze their data assets without moving or revealing the underlying data. To learn more, see the following topics:

Duet AI in BigQuery, an AI-powered collaborator in Google Cloud, can help you complete, generate, and explain SQL queries. This feature is in preview.

BigQuery Studio is now in preview. BigQuery Studio offers features to make it easier for you to discover, explore, analyze, and run inference on data in BigQuery, including:

  • Python notebooks, powered by Colab Enterprise. Notebooks provide one-click Python development runtimes, and built-in support for BigQuery DataFrames.
  • Asset management and version history for notebooks and saved queries, powered by Dataform.

BigQuery DataFrames is now in preview. BigQuery DataFrames is a Python API that you can use to analyze data and perform machine learning tasks in BigQuery. BigQuery DataFrames consists of the following parts:

  • bigframes.pandas implements a DataFrame API (with partial Pandas compatibility) on top of BigQuery.
  • bigframes.ml implements a Python API for BigQuery ML (with partial scikit-learn compatibility).

Get started with BigQuery DataFrames by using the BigQuery DataFrames quickstart.

The following Generative AI features are now generally available (GA) in BigQuery ML:

Try these features with the Generate text by using a remote model and the ML.GENERATE_TEXT function tutorial.

August 24, 2023

The following text embedding features are now available in preview:

For more information, see the tutorial for performing basic semantic search with text embeddings.

The following BigQuery ML inference features are now generally available (GA):

August 22, 2023

BigQuery now allows you to create your own masking routines for your data. You can use the REGEX_REPLACE scalar function to create custom masking rules to obfuscate your sensitive data. This feature is currently in preview.

August 21, 2023

You can now scan tables to create data profiles and monitor data quality. These features help you better understand your data and ensure it is accurate and reliable. These features are generally available (GA).

A weekly digest of client library updates from across the Cloud SDK.

Node.js

Changes for @google-cloud/bigquery

7.2.0 (2023-08-17)

Features
Bug Fixes
  • deps: Update dependency @google-cloud/common to v5 (#1253) (0cbfdcd)
  • deps: Update dependency @google-cloud/paginator to v5 (#1254) (658ff1a)
  • deps: Update dependency @google-cloud/precise-date to v4 (#1261) (7f12f7d)
  • deps: Update dependency @google-cloud/promisify to v4 (#1252) (6f04f39)
  • Manual update for renovate-bot google cloud storage v7 (#1262) (b58fc29)

Go

Changes for bigquery/storage/apiv1beta1

1.54.0 (2023-08-17)

Features
  • bigquery/biglake: Start generating apiv1 and apiv1alpha1 (#8433) (916f779)
  • bigquery/datatransfer: Add EncryptionConfiguration to TransferConfig (e3f8c89)
  • bigquery/storage/managedwriter: Refine connection metrics (#8324) (54fcf36)
  • bigquery/storage: Add beta2 deprecation message (b3dbdde)
  • bigquery/storage: Add default_missing_value_interpretation field; indicate KMS_SERVICE_ERROR is retryable (b3dbdde)
  • bigquery: Add ProjectID to JobIDConfig (#8405) (7fafd80)
Bug Fixes
  • bigquery/datapolicies: Sync the new PredefinedEpxression types to the client library (fcb41cc)
  • bigquery: Use JobFromProject for storage iterator (#8372) (bfbf18c)
Documentation
  • bigquery/storage/managedwriter: Augment context usage in package docs (#8334) (b30a7f4)
  • bigquery: Improve RowIterator docs and out of process pagination with Storage API (#8419) (40f8f80)

Analytics Hub now lets you manage subscriptions. The subscription resource stores relevant information about the subscriber and represents the connection between the shared resource and linked dataset. This feature is now in preview.

August 17, 2023

You can now replicate a dataset from the source region to one or more other regions with cross-region dataset replication. This feature is now in preview.

August 14, 2023

A weekly digest of client library updates from across the Cloud SDK.

Java

Changes for google-cloud-bigquery

2.31.1 (2023-08-09)

Dependencies
  • Update dependency com.google.api.grpc:proto-google-cloud-bigqueryconnection-v1 to v2.25.0 (#2845) (d940f8d)
  • Update dependency com.google.cloud:google-cloud-datacatalog-bom to v1.29.0 (#2846) (87a0a10)
  • Update dependency com.google.cloud:google-cloud-shared-dependencies to v3.14.0 (#2834) (79fe14c)
  • Update dependency org.graalvm.buildtools:junit-platform-native to v0.9.24 (#2839) (ae752bc)
  • Update dependency org.graalvm.buildtools:native-maven-plugin to v0.9.24 (#2840) (1ae6cb9)
  • Update github/codeql-action action to v2.21.1 (#2824) (9978971)
  • Update jmh.version to v1.37 (#2836) (4b3a3c2)

Starting September 15, 2023, prices will apply for network egress from a BigQuery Google Cloud region to another Google Cloud region on the same continent and between different continents. For more information, see BigQuery Network Egress Traffic Pricing Charges Announcement.

August 10, 2023

You can now see query performance insights about high cardinality joins. This feature is generally available (GA).

The September 14, 2022 release notes announced that you could configure the connector to authenticate the connection using an external account with workload identity federation for ODBC driver update release 2.5.0 1001, but workload identity federation is not supported. Workforce identity federation is still supported.

You can now use user-defined functions to export BigQuery data as Protocol Buffer (Protobuf) columns. This feature is generally-available.

August 08, 2023

The following features are now generally available (GA) in queries and materialized views:

  • HAVING MAX and HAVING MIN clauses for the ANY_VALUE function.

  • MAX_BY function, which is a synonym for ANY_VALUE(x HAVING MAX y).

  • MIN_BY function, which is a synonym for ANY_VALUE(x HAVING MIN y).

August 07, 2023

The quantitive LIKE operator is now in preview. With this operator, you can check a search value for matches against several patterns, using one of these conditions:

  • LIKE ANY: Checks if at least one pattern matches.
  • LIKE SOME: Synonym for LIKE ANY.
  • LIKE ALL: Checks if every pattern matches.

The following JSON functions are now generally available (GA).

BigQuery now supports the ability to deny access to principals via deny policies for the following IAM permissions :

  • Managing reservations and capacity commitments: bigquery.googleapis.com/capacityCommitments.*, bigquery.googleapis.com/bireservations.*, bigquery.googleapis.com/reservationAssignments.*, bigquery.googleapis.com/reservations.*
  • Resource Deletion: bigquery.googleapis.com/[datasets, tables, models, routines, jobs, connections].delete
  • Dataset tag bindings: bigquery.googleapis.com/datasets.[createTagBinding, listTagBinding]
  • Row Access Policies: bigquery.rowAccessPolicies.[create, delete, update, setIamPolicy]

A weekly digest of client library updates from across the Cloud SDK.

Node.js

Changes for @google-cloud/bigquery

7.1.1 (2023-08-03)

Bug Fixes

7.1.0 (2023-08-01)

Features
  • Support and optionally parse JSON field (#1229) (cd11447)
  • Support create/list datasets on a different project (#1230) (86c63fb)

7.0.0 (2023-07-31)

⚠ BREAKING CHANGES
  • migrate to node 14 (#1236)
Bug Fixes
  • deps: Update dependency google-auth-library to v9 (#1228) (22cf786)
Miscellaneous Chores

Analytics Hub now supports the use of routines in linked datasets. This feature is now in preview.

August 04, 2023

BigQuery now supports using manifest files for external tables. This feature is now generally available (GA).

August 03, 2023

With Analytics Hub, you can now track usage metrics of your shared datasets. This feature is generally available (GA). The usage metrics include the following:

  • Jobs that run against your shared dataset.
  • The consumption details of your shared dataset by subscribers' projects and organizations.
  • The number of rows and bytes processed by the job.

Cloud console updates: The following features are now available in preview:

You can now GRANT or REVOKE access to materialized views with a SQL statement. This feature is generally available (GA).

July 31, 2023

A weekly digest of client library updates from across the Cloud SDK.

Go

Changes for bigquery/storage/apiv1beta1

1.53.0 (2023-07-24)

Features
  • bigquery/analyticshub: Promote to GA (130c571)
  • bigquery/connection: Add support for Salesforce connections, which are usable only by allowlisted partners (bac978a)
  • bigquery/datapolicies: Promote to GA (130c571)
  • bigquery/storage: Add ResourceExhausted to retryable error for Write API unary calls (#8214) (8ff13bf)
Bug Fixes
  • bigquery/storage/managedwriter: Context refactoring (#8275) (c4104ea)

Java

Changes for google-cloud-bigquery

2.31.0 (2023-07-25)

Features
  • Adds Exception handling to handle ALREADY EXISTS error (#2788) (67a07ea)
Dependencies
  • Update arrow.version to v12.0.1 (#2750) (f92bee5)
  • Update dependency com.google.api.grpc:proto-google-cloud-bigqueryconnection-v1 to v2.24.0 (#2811) (b660063)
  • Update dependency com.google.cloud:google-cloud-datacatalog-bom to v1.28.0 (#2812) (921716d)
  • Update dependency com.google.cloud:google-cloud-shared-dependencies to v3.13.1 (#2806) (df3cd76)
  • Update dependency org.junit.vintage:junit-vintage-engine to v5.10.0 (#2808) (694f711)
  • Update github/codeql-action action to v2.21.0 (#2803) (c6c536c)

BigQuery Omni is now available in the AWS - US West (Oregon) (aws-us-west-2) and the AWS - Europe (Ireland) (aws-eu-west-1) regions.

BigQuery Storage Write API multiplexing is now generally available (GA). You can use multiplexing in the default stream to write to multiple destination tables with shared connections.

July 28, 2023

Query queues are now generally available (GA). With query queues, BigQuery automatically determines your query concurrency based on available slots rather than a fixed limit. Once the maximum concurrency is reached, additional queries are queued until processing resources are available. Query queues are enabled by default and have been rolled out over the last several weeks; no user action is required and you shouldn't see any degradation in your query performance. You can optionally set the maximum concurrency target for a reservation. You can also control the interactive and batch query queue timeout by using default configurations.

July 24, 2023

A weekly digest of client library updates from across the Cloud SDK.

Java

Changes for google-cloud-bigquery

2.30.1 (2023-07-18)

Dependencies
  • Update dependency com.google.api.grpc:proto-google-cloud-bigqueryconnection-v1 to v2.23.0 (#2791) (940301b)
  • Update dependency com.google.cloud:google-cloud-datacatalog-bom to v1.27.0 (#2792) (c791066)

2.30.0 (2023-07-17)

Features
  • Add missing storage related fields to Table, TableInfo and StandardTableDefinition (#2673) (e3003f4)
  • Add support for Search statistics (#2787) (344f695)
Dependencies
  • Update dependency com.google.api.grpc:proto-google-cloud-bigqueryconnection-v1 to v2.22.0 (#2777) (078f244)
  • Update dependency com.google.cloud:google-cloud-datacatalog-bom to v1.26.0 (#2778) (2ee52c9)
  • Update dependency com.google.cloud:google-cloud-shared-dependencies to v3.13.0 (#2786) (dd14eee)
  • Update github/codeql-action action to v2.20.1 (#2766) (2014613)
  • Update github/codeql-action action to v2.20.4 (#2784) (e886f5f)
  • Update ossf/scorecard-action action to v2.2.0 (#2775) (688b2a0)

Python

Changes for google-cloud-bigquery

3.11.4 (2023-07-19)

Bug Fixes

July 20, 2023

Multivariate time series forecasting with the ARIMA_PLUS_XREG model in BigQuery ML is now generally available (GA). This feature lets you perform time series forecasting with extra feature columns. For more information, see the ARIMA_PLUS_XREG information in the end-to-end user journey topic, and try the multivariate time-series forecasting from Seattle air quality data tutorial.

BigQuery ML has introduced new Explainable AI capabilities for better model explainability:

Now you can also use Vertex Explainable AI on BigQuery ML models that you've registered to the Vertex AI Model Registry. To learn more, see Explainable AI for BigQuery ML models.

July 19, 2023

BigQuery can now use search indexes to optimize some queries that contain the equal operator (=), IN operator, LIKE operator, or STARTS_WITH function to compare string literals with indexed data. This feature is in preview.

July 17, 2023

Primary and foreign key table constraints are now generally available. In addition to the features available in preview, you can now also manage constraints through the BigQuery API and view constraints in the BigQuery console.

A weekly digest of client library updates from across the Cloud SDK.

Node.js

Changes for @google-cloud/bigquery

6.2.1 (2023-07-10)

Bug Fixes

The google.cloud.bigquery.storage.v1beta2 API package for BigQueryWrite operations is deprecated and will be removed on July 17, 2024. After that date, requests to that package version for use with the BigQuery Storage Write API will fail. Data written to BigQuery using the BigQuery Storage Write API is accessible by using the google.cloud.bigquery.storage.v1 package.

Next steps: If you call the API directly, switch to google.cloud.bigquery.storage.v1, the generally available (GA) version of the API, to prevent any impact on your workflow.

July 12, 2023

The following BigQuery ML feature preprocessing functionality is now generally available (GA).

You can export models that use the TRANSFORM clause for feature preprocessing to the TensorFlow SavedModel format. There are 13 data types supported for TRANSFORM clause input, and 127 SQL functions supported for use within the TRANSFORM clause.

You can also now deploy a model trained with the TRANSFORM clause to Vertex AI and locally.

Use the following functions to perform feature preprocessing:

Custom holiday modeling for time series forecasting is now in preview. This release offers the following features to improve the transparency, flexibility, and explainability of time series forecasting in BigQuery ML:

A new public table, bigquery-public-data.ml_datasets.holidays_and_events_for_forecasting, has also been added to provide easy look-up of the built-in holidays used in time series forecasting models.

Try these features with the Use custom holidays in a time-series forecasting model tutorial.

July 06, 2023

Spanner Data Boost lets you execute analytics queries and data exports with near-zero impact to existing workloads on your provisioned Spanner instance. This feature is now generally available (GA) in the following regions:

  • asia-northeast1 (Tokyo)
  • us-central1 (Iowa)
  • southamerica-east1 (São Paulo)
  • europe-west1 (Belgium)
  • europe-west2 (London)
  • europe-west3 (Frankfurt)

July 05, 2023

BigQuery is now available in the Turin (europe-west12) and Doha (me-central1) regions.

You can use the LOAD DATA SQL statement to load data from Avro, CSV, newline delimited JSON, JSON, ORC, or Parquet files into a table. This feature is generally available (GA).

The slot estimator now provides cost-optimal commitment and autoscale recommendations based on editions pricing and historical performance metrics. This feature is in preview.

The fail-safe period is now generally available (GA). The fail-safe period offers an additional seven days of data storage after the time travel window, so that the data is available for emergency recovery. Billed costs won't include the bytes used for fail-safe storage until July 17th, 2023.

The ability to use physical bytes for storage billing is now generally available (GA). When you set your dataset's storage billing model to use physical bytes, the total active storage costs you are billed for includes the bytes used for time travel and fail-safe storage. For more information, see Dataset storage billing models.

The ability to configure the time travel window is now generally available (GA). You can specify the duration of the time travel window from a minimum of two days to a maximum of seven days.

BigQuery capacity commitments have changed as follows:

  • Annual commitments are now only available in Enterprise or Enterprise Plus edition. Flat-rate annual commitments are no longer available. For more information about pricing, see Capacity compute (analysis) pricing.
  • Monthly and flex commitments are no longer available. For more information about commitment options, see Capacity commitment plans.

You can now restrict data egress on Analytics Hub listings. This feature is now in preview.

July 03, 2023

A weekly digest of client library updates from across the Cloud SDK.

Go

Changes for bigquery/storage/apiv1beta1

1.52.0 (2023-06-23)

Features
  • bigquery/storage: Add estimated physical file sizes to ReadAPI v1 (94ea341)
  • bigquery/storage: Add table sampling to ReadAPI v1 (ca94e27)
  • bigquery: Support for tables primary and foreign keys (#8055) (93d6a1a)
  • bigquery: Update all direct dependencies (b340d03)
Bug Fixes
  • bigquery/storage/managedwriter: Correct reconnection logic (#8164) (a67d53d)
  • bigquery: REST query UpdateMask bug (df52820)
  • bigquery: RowIterator.Schema not filled when using Storage Read API (#7671) (31040e8)

Python

Changes for google-cloud-bigquery

3.11.3 (2023-06-27)

Bug Fixes
  • Type annotations include Optional when None is accepted (#1554) (6c1ab80)

June 30, 2023

Metadata caching is now available for BigLake tables that reference Amazon S3 data. This feature is in preview. Using cached metadata might improve query performance for BigLake tables.

June 29, 2023

June 26, 2023

You can now create stored procedures for Apache Spark using Java or Scala. You can also use the Google Cloud console PySpark editor to add options for stored Python procedures for Apache Spark. This feature is in Preview.

A weekly digest of client library updates from across the Cloud SDK.

Java

Changes for google-cloud-bigquery

2.29.0 (2023-06-23)

Features
  • Increase default Read API timeout to 60s (#2764) (f606d0b)
Dependencies
  • Update dependency com.google.cloud:google-cloud-shared-dependencies to v3.12.0 (#2771) (7537e0f)
  • Update dependency org.graalvm.buildtools:junit-platform-native to v0.9.23 (#2759) (27ba48a)
  • Update dependency org.graalvm.buildtools:native-maven-plugin to v0.9.23 (#2760) (8cddf8f)

2.28.0 (2023-06-19)

Features

Python

Changes for google-cloud-bigquery

3.11.2 (2023-06-21)

Bug Fixes
  • Updates tests based on revised hacker_news tables (#1591) (d73cf49)

June 21, 2023

TRUNCATE TABLE is now supported for multi-statement transactions. This feature is now generally available (GA).

June 20, 2023

Metadata caching is now generally available (GA). Using cached metadata might improve query performance for BigLake tables and object tables that reference large numbers of objects, by allowing the query to avoid listing objects from Cloud Storage.

This release includes support for the following new features:

Metadata cache usage is billed going forward. For more information, see Costs.

BigQuery now supports querying Apache Iceberg tables that are created by open source engines. This feature is now generally available (GA).

June 19, 2023

A weekly digest of client library updates from across the Cloud SDK.

Java

Changes for google-cloud-bigquery

2.27.1 (2023-06-13)

Dependencies
  • Update actions/checkout action to v3.5.3 (#2746) (17f8438)
  • Update dependency com.google.api.grpc:proto-google-cloud-bigqueryconnection-v1 to v2.21.0 (#2741) (d665e52)
  • Update dependency com.google.cloud:google-cloud-datacatalog-bom to v1.25.0 (#2743) (5d38d23)
  • Update dependency com.google.cloud:google-cloud-shared-dependencies to v3.11.0 (#2738) (3b56445)
  • Update github/codeql-action action to v2.20.0 (#2751) (42ae181)
  • Update github/codeql-action action to v2.3.6 (#2712) (f043ed6)

Python

Changes for google-cloud-bigquery

3.11.1 (2023-06-09)

Documentation
  • Add/reformat return types for cloud RAD docs (#1582) (6efdce1)

June 15, 2023

The following Generative AI features are now in preview with allowlist:

Try these features with the Generate text by using a remote model and the ML.GENERATE_TEXT function tutorial.

June 14, 2023

BigQuery now provides information about the fail-safe period. The fail-safe period offers an additional seven days of data storage after the time travel window, so that the data is available for emergency recovery. This feature is in preview.

The INFORMATION_SCHEMA views that show table storage metadata are now generally available (GA):

BigLake Metastore is now generally available (GA). You can use BigLake Metastore to access and manage Iceberg table metadata from multiple sources.

June 12, 2023

The query execution graph is now generally available (GA). You can use the query execution graph to diagnose query performance issues, and to receive query performance insights.

June 05, 2023

A weekly digest of client library updates from across the Cloud SDK.

Java

Changes for google-cloud-bigquery

2.27.0 (2023-05-30)

Features
  • Add support for session id on TableDataWriteChannel (#2715) (42851d8)
Bug Fixes
  • Add support for repeated record query parameters (#2698) (51aff50)
Dependencies
  • Update dependency com.google.api.grpc:proto-google-cloud-bigqueryconnection-v1 to v2.20.0 (#2720) (4962cac)
  • Update dependency com.google.apis:google-api-services-bigquery to v2-rev20230506-2.0.0 (#2707) (4d2ec07)
  • Update dependency com.google.apis:google-api-services-bigquery to v2-rev20230520-2.0.0 (#2723) (5c64797)
  • Update dependency com.google.cloud:google-cloud-bigquerystorage-bom to v2.37.2 (#2726) (052c47a)
  • Update dependency com.google.cloud:google-cloud-datacatalog-bom to v1.24.0 (#2721) (7c357fb)
  • Update dependency com.google.cloud:google-cloud-shared-dependencies to v3.10.1 (#2713) (744e83a)

Python

Changes for google-cloud-bigquery

3.11.0 (2023-06-01)

Features
Bug Fixes
  • Filter None values from OpenTelemetry attributes (#1567) (9ea2e21)
  • Handle case when expirationMs is None (#1553) (fa6e13d)
  • Raise most recent exception when not able to fetch query job after starting the job (#1362) (09cc1df)

May 25, 2023

The BigQuery partitioning and clustering recommender is now in preview. The recommender analyzes your BigQuery tables to identify partitioning or clustering opportunities for potential cost savings. You can view partition or cluster recommendations through the BigQuery UI or recommender API. You can also apply recommendations directly to your BigQuery tables.

May 23, 2023

DML statements no longer count toward the number of table or partitioned tables modifications per day. The limit of table and partitioned table modifications has not changed.

May 22, 2023

A weekly digest of client library updates from across the Cloud SDK.

Java

Changes for google-cloud-bigquery

2.26.1 (2023-05-16)

Bug Fixes

2.26.0 (2023-05-15)

Features
  • Add field in HivePartitioningOptions (#2678) (4165e55)
  • Allow passing autodetect_schema on table update (#2661) (4c01698)
Bug Fixes
  • Move ratio calculation for whether to use read API to avoid NPE with setUseReadAPI(false) (#2509) (e1326c8)
Dependencies
  • Update arrow.version to v12 (major) (#2675) (7700cf5)
  • Update dependency com.google.api.grpc:proto-google-cloud-bigqueryconnection-v1 to v2.19.0 (#2691) (1939803)
  • Update dependency com.google.cloud:google-cloud-datacatalog-bom to v1.23.0 (#2692) (f56e541)
  • Update dependency com.google.cloud:google-cloud-shared-dependencies to v3.9.0 (#2685) (b74da29)
  • Update dependency org.graalvm.buildtools:junit-platform-native to v0.9.22 (#2687) (cf5d758)
  • Update dependency org.graalvm.buildtools:native-maven-plugin to v0.9.22 (#2688) (32ea8ab)
  • Update github/codeql-action action to v2.3.3 (#2658) (487f207)

May 19, 2023

EXTERNAL_QUERY SQL pushdown optimizes data retrieval from external sources like Cloud SQL or Cloud Spanner databases. Transferring less data reduces execution time and cost. SQL pushdown encompasses both column pruning (SELECT clauses) and filter pushdowns (WHERE clauses). SQL pushdown applies to SELECT * FROM T queries, a significant percentage of all federated queries. Not all data types are supported for filter pushdowns. This feature is generally available (GA).

May 18, 2023

You can now sort your query results by using the sort menu next to a column name. This feature is in preview.

May 16, 2023

The VPC Service Controls perimeter that protects the BigQuery API now also protects the BigQuery Reservation API. Customers who have already configured VPC Service Controls for the BigQuery API or the BigQuery Reservation API should update their configurations to reflect this change. For more information, see BigQuery Reservation API.

May 15, 2023

A weekly digest of client library updates from across the Cloud SDK.

Go

Changes for bigquery/storage/apiv1beta1

1.51.2 (2023-05-08)

Bug Fixes
  • bigquery: Update grpc to v1.55.0 (1147ce0)

BigQuery Omni is now available in the AWS - Asia Pacific (Seoul) (aws-ap-northeast-2) region.

May 11, 2023

Object tables are now generally available (GA).

Object tables are read-only tables containing metadata for unstructured data stored in Cloud Storage. They enable you to analyze and perform inference on images, audio files, documents and other file types by using BigQuery ML and BigQuery remote functions. Object tables extend the data security and governance best practices currently applied to structured data to unstructured data as well.

The GA release includes the following new and updated functions:

May 10, 2023

BigQuery is now available in the Dallas (us-south1) region.

May 09, 2023

You can now view BI Engine Top Tables Cached Bytes, BI Engine Query Fallback Count, and Query Execution Count as dashboard metrics for BigQuery. This feature is now generally available (GA).

EXTERNAL_QUERY SQL pushdown optimizes data retrieval from external sources like Cloud SQL or Cloud Spanner databases. Transferring less data reduces execution time and cost. SQL pushdown encompasses both column pruning (SELECT clauses) and filter pushdowns (WHERE clauses). SQL pushdown applies to SELECT * FROM T queries, a significant percentage of all federated queries. Pushdowns have limitations, for example not all data types are supported for filter pushdowns. This feature is generally available (GA).

May 08, 2023

Differential privacy is now in preview and includes four differential privacy aggregate functions that can be used to anonymize data: AVG, COUNT, SUM, and PERCENTILE_CONT. To learn more, see the following topics:

A weekly digest of client library updates from across the Cloud SDK.

Go

Changes for bigquery/storage/apiv1beta1

1.51.1 (2023-05-04)

Bug Fixes
  • bigquery/storage/managedwriter: Schema evolution improvements (#7838) (49a1621)
  • bigquery: Upgrade arrow to v12 to fix build error (#7870) (402e365)
Documentation
  • bigquery/storage/managedwriter: Improve tuning guidance (#7848) (ca2184c)

INFORMATION_SCHEMA.MATERIALIZED_VIEW view and enhanced job statistics now let you monitor materialized view usage and refresh jobs. This feature is in preview.

May 05, 2023

The INSERT INTO SELECT statement now lets you filter data from files in Amazon S3 and Azure Blob Storage and append it into BigQuery tables. This feature is in preview.

May 04, 2023

You can now use configuration YAML files to transform SQL code when you translate SQL queries from your source database. Configuration YAML files can be used with the batch SQL translator, the interactive SQL translator, and the batch translation Python client. This feature is now in preview.

May 03, 2023

The table clones feature of BigQuery is now generally available (GA).

May 01, 2023

A weekly digest of client library updates from across the Cloud SDK.

Java

Changes for google-cloud-bigquery

2.25.0 (2023-04-27)

Features
Dependencies
  • Update dependency com.google.api.grpc:proto-google-cloud-bigqueryconnection-v1 to v2.18.0 (#2648) (29bd415)
  • Update dependency com.google.apis:google-api-services-bigquery to v2-rev20230408-2.0.0 (#2650) (b9c2f60)
  • Update dependency com.google.cloud:google-cloud-bigquerystorage-bom to v2.36.1 (fea119b)
  • Update dependency com.google.cloud:google-cloud-datacatalog-bom to v1.22.0 (#2649) (b6326f3)
  • Update dependency com.google.cloud:google-cloud-shared-dependencies to v3.8.0 (#2659) (691a47a)
  • Update dependency org.junit.vintage:junit-vintage-engine to v5.9.3 (#2660) (319f98e)

You can now add descriptions to the columns of a view. To do this, use the CREATE VIEW or ALTER COLUMN DDL statements. This feature is in preview.

If you use query queues, then you can set the interactive and batch queue timeouts in your default configuration. This feature is in preview.

April 27, 2023

JSON data type mapping is now available for Cloud Spanner federated queries. This feature is generally available (GA).

April 26, 2023

April 25, 2023

Dynamic data masking has been updated to allow masking on RECORD columns that have been set to REPEATED mode. Previously, querying such columns when data masking had been applied would return internal errors. This feature is generally available (GA).

April 24, 2023

A weekly digest of client library updates from across the Cloud SDK.

Go

Changes for bigquery/storage/apiv1beta1

1.51.0 (2023-04-17)

Features
  • bigquery/storage/managedwriter: Expose connection multiplexing as experimental (#7673) (3b8bfb4)
  • bigquery: Add Iceberg as DataFormat option. (#7155) (7a9e211)
  • bigquery: Expose dataset MaxTimeTravelHours (#7706) (22a666c)
Bug Fixes
  • bigquery: Respect context during query execution (#7693) (56772f5)

Python

Changes for google-cloud-bigquery

3.10.0 (2023-04-18)

Features
  • Add date, datetime, time, timestamp dtype to to_dataframe (#1547) (64e913d)

The BigQuery Data Transfer Service for Google Ads supports the new Google Ads API. The Google Ads connector supports PMax and Discovery campaigns, a limit of 8000 leaf accounts per transfer, the --table_filter flag, and backwards compatibility. This feature is now generally available (GA).

April 19, 2023

Updates to preferred tables for existing BI engine reservations now take up to ten seconds to propagate, down from five minutes. This feature is generally available (GA).

April 17, 2023

A weekly digest of client library updates from across the Cloud SDK.

Java

Changes for google-cloud-bigquery

2.24.5 (2023-04-14)

Dependencies
  • Update actions/checkout action to v3.5.2 (#2630) (95e49fd)
  • Update dependency com.google.api.grpc:proto-google-cloud-bigqueryconnection-v1 to v2.16.0 (#2625) (594a7b4)
  • Update dependency com.google.apis:google-api-services-bigquery to v2-rev20230401-2.0.0 (#2631) (5d8d9a6)
  • Update dependency com.google.cloud:google-cloud-bigquerystorage-bom to v2.35.0 (8439020)
  • Update dependency com.google.cloud:google-cloud-datacatalog-bom to v1.20.0 (#2626) (f466b51)
  • Update dependency com.google.cloud:google-cloud-shared-dependencies to v3.7.0 (#2637) (e8f07d7)
  • Update dependency org.graalvm.buildtools:junit-platform-native to v0.9.21 (#2633) (3e376b1)
  • Update dependency org.graalvm.buildtools:native-maven-plugin to v0.9.21 (#2634) (000f720)
  • Update github/codeql-action action to v2.2.12 (#2635) (b2f97e9)
  • Update ossf/scorecard-action action to v2.1.3 (#2618) (d166401)

Python

Changes for google-cloud-bigquery

1.28.3 (2022-12-14)

Bug Fixes
  • Fix invalid version specification of pyarrow (#1403) (4812d82)

April 13, 2023

BigQuery supports setting the rounding mode to ROUND_HALF_EVEN or ROUND_HALF_AWAY_FROM_ZERO for parameterized NUMERIC or BIGNUMERIC columns at the column level. You can specify a default rounding mode at the table or dataset level that is automatically attached to any columns added within those entities. The ROUND() function also accepts the rounding mode as an optional argument. This feature is generally available GA.

April 10, 2023

The limit for maximum result size (20 GiB logical bytes) when querying Azure or Amazon Simple Storage service (S3) data is now generally available (GA). Querying Azure and Amazon S3 data are now subject to the following quotas and limitations:

  • The maximum row size is 10 MiB. For more information, see Quotas for query jobs.

  • If your query uses the ORDER BY clause and has a result size larger than 256 MB, then your query fails. Previously, this limit was 2 MB. For more information, see Limitations.

A weekly digest of client library updates from across the Cloud SDK.

Go

Changes for bigquery/storage/apiv1beta1

1.50.0 (2023-04-03)

Features
  • bigquery/connection: Add spark connection properties type (#7570) (499b489)
  • bigquery/migration: Add request_source field and update formatting (#7586) (c967961)
  • bigquery/reservation: Add edition/autoscale related fields (#7608) (2b7bb66)
  • bigquery/storage/managedwriter: Decouple connections and writers (#7314) (7d085b4)
  • bigquery/storage/managedwriter: Introduce location routing header (#7663) (cf06802)
Bug Fixes
  • bigquery/storage/managedwriter: Fix option propagation (#7669) (f684e16)
Documentation
  • bigquery/reservation: Mention that some fields are deprecated (597ea0f)

The results for queries against table snapshots can now be returned from cache.

April 06, 2023

The add data demo guide walks you through the process of adding data to BigQuery through popular sources and is now in preview.

April 05, 2023

Non-incremental materialized views support most SQL queries, including OUTER JOIN, UNION, and HAVING clauses, as well as analytic functions. This feature is in preview.

April 04, 2023

BigQuery is now available in the Israel (me-west1) region.

April 03, 2023

A weekly digest of client library updates from across the Cloud SDK.

Java

Changes for google-cloud-bigquery

2.24.4 (2023-03-30)

Bug Fixes
  • QueryWithStructsParameters sample mismatch (#2610) (71f9f55)
Dependencies
  • Update dependency com.google.apis:google-api-services-bigquery to v2-rev20230318-2.0.0 (#2607) (a328eb2)
  • Update dependency com.google.cloud:google-cloud-bigquerystorage-bom to v2.34.2 (#2619) (e4aa0fe)
  • Update dependency com.google.cloud:google-cloud-shared-dependencies to v3.6.0 (#2612) (eac97ac)
  • Update github/codeql-action action to v2.2.9 (#2608) (24aac14)

Python

Changes for google-cloud-bigquery

3.9.0 (2023-03-28)

Features
Bug Fixes
  • Keyerror when the load_table_from_dataframe accesses a unmapped dtype dataframe index (#1535) (a69348a)

3.8.0 (2023-03-24)

Features
  • Add bool, int, float, string dtype to to_dataframe (#1529) (5e4465d)
  • Add default LoadJobConfig to Client (#1526) (a2520ca)
  • Expose configuration property on CopyJob, ExtractJob, LoadJob, QueryJob (#1521) (8270a10)
Bug Fixes
  • Loosen ipywidgets restrictions further to address ipython compatibility issues (#1531) (50e5026)

March 30, 2023

BigQuery ML documentation is now integrated with BigQuery documentation to unify resources for data analysis and machine learning tasks such as inference. BigQuery ML documentation resources include:

BigQuery Partner Center, which can be used to discover and try validated partner applications, is now generally available (GA). In addition, the Google Cloud Ready - BigQuery initiative has added 14 new partners.

March 29, 2023

Compute (analysis) is now generally available (GA) in three new BigQuery editions: Standard, Enterprise, and Enterprise Plus. These editions support the slots autoscaling model to meet your organizations' needs and budgets.

Autoscaling slots are now generally available (GA). Autoscaling slot reservations and commitments created during the feature's preview have been set to BigQuery Enterprise edition.

March 28, 2023

You can now use the tf_version training option to specify the Tensorflow (TF) version during model training. By default, tf_version is set as '1.15'. If you want to use TF2 with Keras API, you can add tf_version = '2.8.0' when creating the model.

You can now use the xgboost_version training option to specify the XGBoost version during model training. By default, xgboost_version is set as '0.9'. You can choose XGBoost version 1.1 by specifying xgboost_version = '1.1'.

You can now use the instance_weight_col training option to identify the column containing weights for each data point in the training dataset. Currently the instance_weight_col option is only available for boosted tree and random forest models with non-array feature types.

You can now import model artifacts saved in ONNX, XGBoost, and TensorFlow Lite formats into BigQuery for inference, allowing you to leverage models built in popular frameworks directly within the BigQuery ML inference engine.

You can also host models remotely on Vertex AI Prediction and do inference with BigQuery ML, removing the need to build data pipelines manually.

You can do inference with Google Cloud's state of the art pretrained models using Cloud AI service table-valued functions (TVFs) to get insights from your data. The TVFs work with Cloud Vision API, Cloud Natural Language API and Cloud Translation API.

These features are in preview. To enroll to use this feature, complete the enrollment form.

March 27, 2023

A weekly digest of client library updates from across the Cloud SDK.

Node.js

Changes for @google-cloud/bigquery

6.2.0 (2023-03-22)

Features
Bug Fixes

Java

Changes for google-cloud-bigquery

2.24.3 (2023-03-24)

Dependencies
  • Update actions/checkout action to v3.5.0 (#2600) (f38d9f1)

2.24.2 (2023-03-22)

Dependencies
  • Update github/codeql-action action to v2.2.8 (#2593) (d306ad8)

2.24.1 (2023-03-21)

Dependencies
  • Update cloud client dependencies (7b07779)
  • Update dependency com.google.cloud:google-cloud-bigquery to v2.23.2 (7b07779)
  • Update dependency com.google.cloud:google-cloud-bigquery to v2.24.0 (7b07779)
  • Update dependency com.google.cloud:google-cloud-bigquerystorage-bom to v2.34.1 (7b07779)
  • Update dependency com.google.cloud:google-cloud-bigtable to v2.20.1 (7b07779)
  • Update dependency com.google.cloud:libraries-bom to v26.10.0 (7b07779)

2.24.0 (2023-03-21)

Features
Dependencies
  • Update actions/checkout action to v3.4.0 (#2575) (6935a1e)
  • Update actions/upload-artifact action to v3.1.2 (#2571) (aa0c70e)
  • Update cloud client dependencies (#2583) (dcacc31)
  • Update dependency com.google.api.grpc:proto-google-cloud-bigqueryconnection-v1 to v2.15.0 (#2577) (eaf09d6)
  • Update dependency com.google.apis:google-api-services-bigquery to v2-rev20230311-2.0.0 (#2578) (aab037c)
  • Update dependency com.google.cloud:google-cloud-shared-dependencies to v3.5.0 (#2580) (1764eeb)
  • Update dependency com.google.cloud:google-cloud-storage to v2.20.0 (#2559) (8a854db)
  • Update github/codeql-action action to v2.2.7 (#2572) (105f5ee)

BigQuery now supports change data capture (CDC) by processing and applying streamed changes in real-time to existing data using the BigQuery Storage Write API. This feature is in preview.

March 22, 2023

BigQuery now supports Unicode column naming using international character sets, alphanumeric and special characters. Existing columns can use these new capabilities using the RENAME command. This feature is now in preview.

March 20, 2023

A weekly digest of client library updates from across the Cloud SDK.

Go

Changes for bigquery/storage/apiv1beta1

1.49.0 (2023-03-16)

Features
  • bigquery: Add support for storage billing model (#7510) (0132ca9), refs #6978
  • bigquery: Update iam and longrunning deps (91a1f78)

Python

Changes for google-cloud-bigquery

3.7.0 (2023-03-06)

Features
  • Add connection_properties and create_session to LoadJobConfig (#1509) (cd0aaa1)
  • Add default_query_job_config property and property setter to BigQuery client (#1511) (a23092c)
Documentation

The following AutoML Tables model features are now generally available:

  • Availability in additional regions.
  • CMEK support in available regions except multi-regions US and EU.
  • OPTIMIZATION_OBJECTIVE now accepts two additional options:
    • MAXIMIZE_PRECISION_AT_RECALL
    • MAXIMIZE_RECALL_AT_PRECISION

March 14, 2023

The Lineage tab in the table properties page lets you track how your data moves and transforms through BigQuery. This feature is now generally available (GA).

March 13, 2023

A weekly digest of client library updates from across the Cloud SDK.

Java

Changes for google-cloud-bigquery

2.23.2 (2023-03-07)

Bug Fixes
  • External table definition parquet format options (#2535) (eb45973)
Documentation

You can now specify translation configurations in the BigQuery Interactive SQL Translator and use it to debug Batch SQL translator jobs. This feature is now in preview.

March 10, 2023

The CREATE TABLE AS SELECT statement now lets you filter data from files in Amazon S3 and Azure Blob Storage before transferring results into BigQuery tables This feature is in preview.

March 07, 2023

March 06, 2023

A weekly digest of client library updates from across the Cloud SDK.

Go

Changes for bigquery/storage/apiv1beta1

1.48.0 (2023-03-01)

Features
  • bigquery/connection: Add cloud spanner connection properties - serverless analytics (#7487) (14771b1)
  • bigquery/storage/managedwriter: Mark managedwriter as GA (#6804) (3d3eeda)

Java

Changes for google-cloud-bigquery

2.23.1 (2023-03-02)

Dependencies
  • Update dependency com.google.api.grpc:proto-google-cloud-bigqueryconnection-v1 to v2.14.0 (#2545) (ad78ebb)
  • Update dependency com.google.cloud:google-cloud-datacatalog-bom to v1.18.0 (#2546) (60e45e4)
  • Update dependency com.google.cloud:google-cloud-shared-dependencies to v3.4.0 (#2547) (2588582)

March 02, 2023

The WITH RECURSIVE clause is now generally available (GA). This clause lets you include one or more recursive common table expressions (CTEs) in a query.

February 27, 2023

A weekly digest of client library updates from across the Cloud SDK.

Go

Changes for bigquery/storage/apiv1beta1

1.47.0 (2023-02-22)

Features
  • bigquery/storage: Add default_value_expression to TableFieldSchema (#7400) (1244b3f)
  • bigquery: Add support for session in load jobs (#7418) (f9ff2ca)
Bug Fixes
  • bigquery: Avoid double-channel-close during context cancellation (#7467) (ca4b2ef)

Java

Changes for google-cloud-bigquery

2.23.0 (2023-02-22)

Features
Dependencies
  • Update cloud client dependencies (#2526) (4d88ccc)
  • Update dependency com.google.api.grpc:proto-google-cloud-bigqueryconnection-v1 to v2.13.0 (#2533) (ed2cb74)
  • Update dependency com.google.apis:google-api-services-bigquery to v2-rev20230210-2.0.0 (#2530) (62ff092)
  • Update dependency com.google.cloud:google-cloud-shared-dependencies to v3.3.0 (#2534) (f1bcc33)
  • Update dependency org.graalvm.buildtools:junit-platform-native to v0.9.20 (#2527) (5fe5e74)
  • Update dependency org.graalvm.buildtools:native-maven-plugin to v0.9.20 (#2528) (554e75d)

Python

Changes for google-cloud-bigquery

3.6.0 (2023-02-22)

Features
  • Adding preserveAsciiControlCharacter to CSVOptions (#1491) (f832e7a)
Bug Fixes
  • Annotate optional integer parameters with optional type (#1487) (a190aaa)
  • Loosen ipywidget dependency (#1504) (20d3276)
  • Removes scope to avoid unnecessary duplication (#1503) (665d7ba)
Dependencies
  • Update minimum google-cloud-core to 1.6.0 (a190aaa)

You can set default values on columns in your BigQuery tables. This feature is now generally available (GA).

The multivariate time-series forecasting model ARIMA_PLUS_XREG is now available to on-demand users.

February 23, 2023

Authorized stored procedures are now in preview. This feature lets you share stored procedures with users or groups without giving them direct access to the underlying tables.

February 22, 2023

Fixed linked datasets querying shared dataset that has data ingested through streaming inserts or the BigQuery Storage Write API.

February 21, 2023

February 15, 2023

You can now make a dataset and the tables in that dataset case-insensitive when you create a dataset or alter a dataset. This feature is generally available (GA).

In the Explorer pane, the resource corresponding to the focused tab is now selected. This feature is generally available (GA).

In the Explorer pane, you can now see all the resources in the searched resource's level by clicking Show more. This feature is generally available (GA).

February 13, 2023

A weekly digest of client library updates from across the Cloud SDK.

Go

Changes for bigquery/storage/apiv1beta1

1.46.0 (2023-02-06)

Features
Bug Fixes
  • bigquery: Create/update an isolated dataset for collation feature (#7256) (b371558)
  • bigquery: Fetch dst table for jobs when readings with Storage API (#7325) (0bf80d7), refs #7322

Java

Changes for google-cloud-bigquery

2.22.0 (2023-02-08)

Features
  • Add collation for Case sensitive string column (#2490) (3257737)
Dependencies
  • Update arrow.version to v11 (major) (#2495) (94ed060)
  • Update dependency com.google.api.grpc:proto-google-cloud-bigqueryconnection-v1 to v2.11.0 (#2482) (e6ffb9b)
  • Update dependency com.google.api.grpc:proto-google-cloud-bigqueryconnection-v1 to v2.12.0 (#2512) (09f280d)
  • Update dependency com.google.cloud:google-cloud-bigquerystorage-bom to v2.31.0 (#2499) (c0a393c)
  • Update dependency com.google.cloud:google-cloud-datacatalog-bom to v1.15.0 (#2483) (5c2bf69)
  • Update dependency com.google.cloud:google-cloud-shared-dependencies to v3.2.0 (#2513) (02832dd)

You can now create materialized views over BigLake metadata cache-enabled tables to reference structured data stored in Cloud Storage. This feature is in preview.

February 09, 2023

You can now apply four new types of dynamic data masking to table columns in BigQuery. These new data masking types include date year, email, first four characters, and last four characters masks. This feature is generally available (GA).

Autoscaling slot reservations are now available in preview. You can create autoscaling reservations and associated commitments using slots autoscaling.

Cloud console updates: In the Explorer pane, you can now refresh the contents of a resource (project or dataset). To refresh the contents of a resource, click View actions, and then click Refresh contents.

February 06, 2023

You can now view information related to query processing to monitor and optimize queries with the query_info column in INFORMATION_SCHEMA.JOBS, JOBS_BY_FOLDER and JOBS_BY_ORGANIZATION views. This feature is generally available (GA).

A weekly digest of client library updates from across the Cloud SDK.

Python

Changes for google-cloud-bigquery

3.5.0 (2023-01-31)

Features
  • Add str method to DatasetReference (#1477) (f32df1f)
  • Add preserveAsciiControlCharacter to LoadJobConfig (#1484) (bd1da9a)
Documentation
  • Adds snippet for creating table with external data config (#1420) (f0ace2a)
  • Revise delete label table code sample, add TODO to clean up sni… (#1466) (0dab7d2)
  • samples: Table variable fix (#1287) (a71888a)

February 01, 2023

The BigQuery Data Transfer Service can now transfer data from Azure Blob Storage into BigQuery. This feature is now in preview.

January 31, 2023

Azure workload identity federation is now generally available (GA) for BigQuery Omni connections. You can now create a connection for federated identity using Google Cloud console.

Cloud console updates: When you create datasets, select locations to run specific queries, or create exchanges in Analytics Hub, you now see separate options for multi-region and specific regions. Based on your selection, you see a list with more options.

January 30, 2023

You can search for BigQuery partners in the BigQuery Partner Center. This feature is in Preview.

A weekly digest of client library updates from across the Cloud SDK.

Node.js

Changes for @google-cloud/bigquery

6.1.0 (2023-01-25)

Features

Java

Changes for google-cloud-bigquery

2.21.0 (2023-01-23)

Features
  • Migrate from google-http-client-jackson2 to google-http-client-gson (#2471) (09a8382)
Bug Fixes
Dependencies
  • Update com.google.cloud:google-cloud-bigquerystorage-bom to v2.28.3 (5a20c32)
  • Update com.google.cloud:google-cloud-datacatalog-bom to v1.14.0 (606fc1e)
  • Update com.google.cloud:google-cloud-storage to v2.17.1 (#2465) (606fc1e)
  • Update com.google.cloud:google-cloud-storage to v2.17.2 (#2479) (5a20c32)
  • Update dependency com.google.api.grpc:proto-google-cloud-bigqueryconnection-v1 to v2.10.0 (#2463) (5bfa7ae)
  • Update dependency com.google.apis:google-api-services-bigquery to v2-rev20230114-2.0.0 (#2477) (b6409d5)
  • Update dependency com.google.cloud:google-cloud-shared-dependencies to v3.1.2 (#2476) (688b6a6)

January 23, 2023

A weekly digest of client library updates from across the Cloud SDK.

Python

Changes for google-cloud-bigquery

3.4.2 (2023-01-13)

Bug Fixes
  • Add support for python 3.11 (#1463) (730a1de)
  • Require grpcio >= 1.49.1 for python 3.11 (72b25c5)
Dependencies
  • Remove upper bound on packaging dependency (#1440) (6088129)
Documentation
  • Create sample to write schema file from table (#1439) (093cc68)
  • Created samples for load table and create table from schema file (#1436) (8ad2e5b)
  • Revise create table cmek sample (