This page documents production updates to BigQuery. We recommend that BigQuery developers periodically check this list for any new announcements. BigQuery automatically updates to the latest release and cannot be downgraded to a previous version.
You can see the latest product updates for all of Google Cloud on the Google Cloud page, browse and filter all release notes in the Google Cloud console, or programmatically access release notes in BigQuery.
To get the latest product updates delivered to you, add the URL of this page to your
feed
reader, or add the feed URL directly: https://cloud.google.com/feeds/bigquery-release-notes.xml
September 28, 2023
The following BigQuery ML point-in-time lookup functions are now in preview. These functions let you specify a point-in-time cutoff when retrieving features for training a model or running inference, in order to avoid data leakage.
- Use the
ML.FEATURES_AT_TIME
function to use the same point-in-time cutoff for all entities when retrieving features. - Use the
ML.ENTITY_FEATURES_AT_TIME
function to retrieve features from multiple points in time for multiple entities.
You can now use IAM conditions to control access to BigQuery resources. This feature is in preview.
September 25, 2023
Materialized views over BigLake metadata cache-enabled tables can reference structured data stored in Cloud Storage. These materialized views function like materialized views over BigQuery-managed storage tables, including the benefits of automatic refresh and smart tuning. This feature is now generally available (GA).
Authorized stored procedures are now generally available (GA). This feature lets you share stored procedures with users or groups without giving them direct access to the underlying tables.
Support for Google AdWords is now deprecated by the BigQuery Data Transfer Service. For information about transfers from Google Ads, see Google Ads transfers.
September 20, 2023
The maximum number of rows for results returned in Connected Sheets has increased as follows:
- Pivot tables increased from 30,000 to 50,000 rows
- Data extracts increased from 25,000 to 50,000 rows
September 18, 2023
A weekly digest of client library updates from across the Cloud SDK.
Go
Changes for bigquery/storage/apiv1beta1
1.55.0 (2023-09-11)
Features
- bigquery/datapolicies: Support using custom UDF in the data policy (20725c8)
- bigquery: Set PreferredMinStreamCount when creating read session (#8476) (22e095a), refs #8432
Bug Fixes
Java
Changes for google-cloud-bigquery
2.32.0 (2023-09-14)
Features
- Add support for converting interval fields to threeten PeriodDuration (#2838) (2294c2f)
- Add support for ExternalDatasetReference (#2871) (bbb86fd)
Dependencies
- Update actions/checkout action to v4 (#2862) (902e9b9)
- Update actions/upload-artifact action to v3.1.3 (#2867) (cbbf0fb)
- Update dependency com.google.api.grpc:proto-google-cloud-bigqueryconnection-v1 to v2.26.0 (#2873) (6196625)
- Update dependency com.google.cloud:google-cloud-datacatalog-bom to v1.30.0 (#2874) (6cafedf)
- Update dependency com.google.cloud:google-cloud-shared-dependencies to v3.15.0 (#2870) (f24439b)
- Update dependency org.graalvm.buildtools:junit-platform-native to v0.9.26 (#2868) (d01031c)
- Update dependency org.graalvm.buildtools:native-maven-plugin to v0.9.26 (#2869) (edd7141)
The BigQuery Data Transfer Service now supports transfers from Search Ads 360 using the new Search Ads 360 reporting API. This feature is in preview. Customers with existing Search Ads 360 transfers should migrate their workflows to be compatible with the new Search Ads 360. The BigQuery Data Transfer Service will stop its support for the old Search Ads 360 reporting API on May 31st, 2024.
September 13, 2023
You can now create a federated dataset in BigQuery that federates to an existing database in AWS Glue. This feature is in preview.
September 12, 2023
The array subscript operator now returns a value in an array directly by index. Previously, only offset and ordinal were available. This feature is generally available (GA).
The struct subscript operator has been added. With this operator, you can access a STRUCT
field by index, offset, or ordinal. This feature is generally available (GA).
September 11, 2023
A weekly digest of client library updates from across the Cloud SDK.
August 30, 2023
You can now use EXPORT DATA
statements to directly export BigQuery data to Bigtable. This feature is in preview.
August 29, 2023
Data clean rooms is now in preview. Data clean rooms provide a secure environment in which multiple parties can share, join, and analyze their data assets without moving or revealing the underlying data. To learn more, see the following topics:
Duet AI in BigQuery, an AI-powered collaborator in Google Cloud, can help you complete, generate, and explain SQL queries. This feature is in preview.
BigQuery Studio is now in preview. BigQuery Studio offers features to make it easier for you to discover, explore, analyze, and run inference on data in BigQuery, including:
- Python notebooks, powered by Colab Enterprise. Notebooks provide one-click Python development runtimes, and built-in support for BigQuery DataFrames.
- Asset management and version history for notebooks and saved queries, powered by Dataform.
BigQuery DataFrames is now in preview. BigQuery DataFrames is a Python API that you can use to analyze data and perform machine learning tasks in BigQuery. BigQuery DataFrames consists of the following parts:
bigframes.pandas
implements a DataFrame API (with partial Pandas compatibility) on top of BigQuery.bigframes.ml
implements a Python API for BigQuery ML (with partial scikit-learn compatibility).
Get started with BigQuery DataFrames by using the BigQuery DataFrames quickstart.
The following Generative AI features are now generally available (GA) in BigQuery ML:
- Creating a remote model based on the Vertex AI large language model (LLM) text-bison.
- Using the
ML.GENERATE_TEXT
function with an LLM-based remote model to perform generative natural language tasks on text stored in BigQuery tables.
Try these features with the Generate text by using a remote model and the ML.GENERATE_TEXT function tutorial.
August 24, 2023
The following text embedding features are now available in preview:
- Creating a BigQuery ML remote model that references the Vertex AI PaLM APIs for embeddings (
textembedding-gecko
). - Using the
ML.GENERATE_TEXT_EMBEDDING
function with the remote embedding model to embed text stored in BigQuery. - Using the
ARRAY<NUMERIC> type
as an input feature type to other models. - Generating text embeddings with the NNLM, SWIVEL, and BERT TensorFlow models.
For more information, see the tutorial for performing basic semantic search with text embeddings.
The following BigQuery ML inference features are now generally available (GA):
- Importing ONNX, XGBoost, and TensorFlow Lite models so that you can run them within the BigQuery ML inference engine.
- Hosting models remotely on Vertex AI Prediction so you can do inference with BigQuery ML, removing the need to build data pipelines manually.
- Using BigQuery ML functions to perform inference on Vertex AI pretrained models so that you can accomplish natural language processing, translation, and computer vision tasks in BigQuery. These functions work with the Cloud Vision, Cloud Natural Language, and Cloud Translation APIs.
August 22, 2023
BigQuery now allows you to create your own masking routines for your data. You can use the REGEX_REPLACE
scalar function to create custom masking rules to obfuscate your sensitive data. This feature is currently in preview.
August 21, 2023
You can now scan tables to create data profiles and monitor data quality. These features help you better understand your data and ensure it is accurate and reliable. These features are generally available (GA).
A weekly digest of client library updates from across the Cloud SDK.
Node.js
Changes for @google-cloud/bigquery
7.2.0 (2023-08-17)
Features
Bug Fixes
- deps: Update dependency @google-cloud/common to v5 (#1253) (0cbfdcd)
- deps: Update dependency @google-cloud/paginator to v5 (#1254) (658ff1a)
- deps: Update dependency @google-cloud/precise-date to v4 (#1261) (7f12f7d)
- deps: Update dependency @google-cloud/promisify to v4 (#1252) (6f04f39)
- Manual update for renovate-bot google cloud storage v7 (#1262) (b58fc29)
Go
Changes for bigquery/storage/apiv1beta1
1.54.0 (2023-08-17)
Features
- bigquery/biglake: Start generating apiv1 and apiv1alpha1 (#8433) (916f779)
- bigquery/datatransfer: Add EncryptionConfiguration to TransferConfig (e3f8c89)
- bigquery/storage/managedwriter: Refine connection metrics (#8324) (54fcf36)
- bigquery/storage: Add beta2 deprecation message (b3dbdde)
- bigquery/storage: Add default_missing_value_interpretation field; indicate KMS_SERVICE_ERROR is retryable (b3dbdde)
- bigquery: Add ProjectID to JobIDConfig (#8405) (7fafd80)
Bug Fixes
- bigquery/datapolicies: Sync the new PredefinedEpxression types to the client library (fcb41cc)
- bigquery: Use JobFromProject for storage iterator (#8372) (bfbf18c)
Documentation
Analytics Hub now lets you manage subscriptions. The subscription resource stores relevant information about the subscriber and represents the connection between the shared resource and linked dataset. This feature is now in preview.
August 17, 2023
You can now replicate a dataset from the source region to one or more other regions with cross-region dataset replication. This feature is now in preview.
August 14, 2023
A weekly digest of client library updates from across the Cloud SDK.
Java
Changes for google-cloud-bigquery
2.31.1 (2023-08-09)
Dependencies
- Update dependency com.google.api.grpc:proto-google-cloud-bigqueryconnection-v1 to v2.25.0 (#2845) (d940f8d)
- Update dependency com.google.cloud:google-cloud-datacatalog-bom to v1.29.0 (#2846) (87a0a10)
- Update dependency com.google.cloud:google-cloud-shared-dependencies to v3.14.0 (#2834) (79fe14c)
- Update dependency org.graalvm.buildtools:junit-platform-native to v0.9.24 (#2839) (ae752bc)
- Update dependency org.graalvm.buildtools:native-maven-plugin to v0.9.24 (#2840) (1ae6cb9)
- Update github/codeql-action action to v2.21.1 (#2824) (9978971)
- Update jmh.version to v1.37 (#2836) (4b3a3c2)
Starting September 15, 2023, prices will apply for network egress from a BigQuery Google Cloud region to another Google Cloud region on the same continent and between different continents. For more information, see BigQuery Network Egress Traffic Pricing Charges Announcement.
August 10, 2023
You can now see query performance insights about high cardinality joins. This feature is generally available (GA).
The September 14, 2022 release notes announced that you could configure the connector to authenticate the connection using an external account with workload identity federation for ODBC driver update release 2.5.0 1001, but workload identity federation is not supported. Workforce identity federation is still supported.
You can now use user-defined functions to export BigQuery data as Protocol Buffer (Protobuf) columns. This feature is generally-available.
August 08, 2023
The following features are now generally available (GA) in queries and materialized views:
August 07, 2023
The quantitive LIKE
operator is now in
preview. With this operator, you can check a search value for matches against several patterns, using one of these conditions:
LIKE ANY
: Checks if at least one pattern matches.LIKE SOME
: Synonym forLIKE ANY
.LIKE ALL
: Checks if every pattern matches.
The following JSON functions are now generally available (GA).
JSON_ARRAY
: Creates a JSON array.JSON_ARRAY_APPEND
: Appends JSON data to the end of a JSON array.JSON_ARRAY_INSERT
: Inserts JSON data into a JSON array.JSON_OBJECT
: Creates a JSON object.JSON_REMOVE
: Produces JSON with the specified JSON data removed.JSON_SET
: Inserts or replaces JSON data.JSON_STRIP_NULLS
: Removes JSON nulls.LAX_BOOL
: Attempts to convert a JSON value to a SQLBOOL
value.LAX_FLOAT64
: Attempts to convert a JSON value to a SQLFLOAT64
value.LAX_INT64
: Attempts to convert a JSON value to a SQLINT64
value.LAX_STRING
: Attempts to convert a JSON value to a SQLSTRING
value.
BigQuery now supports the ability to deny access to principals via deny policies for the following IAM permissions :
- Managing reservations and capacity commitments:
bigquery.googleapis.com/capacityCommitments.*, bigquery.googleapis.com/bireservations.*, bigquery.googleapis.com/reservationAssignments.*, bigquery.googleapis.com/reservations.*
- Resource Deletion:
bigquery.googleapis.com/[datasets, tables, models, routines, jobs, connections].delete
- Dataset tag bindings:
bigquery.googleapis.com/datasets.[createTagBinding, listTagBinding]
- Row Access Policies:
bigquery.rowAccessPolicies.[create, delete, update, setIamPolicy]
A weekly digest of client library updates from across the Cloud SDK.
Node.js
Changes for @google-cloud/bigquery
7.1.1 (2023-08-03)
Bug Fixes
7.1.0 (2023-08-01)
Features
- Support and optionally parse JSON field (#1229) (cd11447)
- Support create/list datasets on a different project (#1230) (86c63fb)
7.0.0 (2023-07-31)
⚠ BREAKING CHANGES
- migrate to node 14 (#1236)
Bug Fixes
Miscellaneous Chores
Analytics Hub now supports the use of routines in linked datasets. This feature is now in preview.
August 04, 2023
BigQuery now supports using manifest files for external tables. This feature is now generally available (GA).
August 03, 2023
With Analytics Hub, you can now track usage metrics of your shared datasets. This feature is generally available (GA). The usage metrics include the following:
- Jobs that run against your shared dataset.
- The consumption details of your shared dataset by subscribers' projects and organizations.
- The number of rows and bytes processed by the job.
Cloud console updates: The following features are now available in preview:
- On the Welcome page, in the Recently accessed section, you can view your 10 most recently accessed resources.
- After you run a query in the query editor, in the Chart tab, you can see the visualization of your query results.
You can now GRANT
or REVOKE
access to materialized views with a SQL statement. This feature is generally available (GA).
July 31, 2023
A weekly digest of client library updates from across the Cloud SDK.
Go
Changes for bigquery/storage/apiv1beta1
1.53.0 (2023-07-24)
Features
- bigquery/analyticshub: Promote to GA (130c571)
- bigquery/connection: Add support for Salesforce connections, which are usable only by allowlisted partners (bac978a)
- bigquery/datapolicies: Promote to GA (130c571)
- bigquery/storage: Add ResourceExhausted to retryable error for Write API unary calls (#8214) (8ff13bf)
Bug Fixes
Java
Changes for google-cloud-bigquery
2.31.0 (2023-07-25)
Features
Dependencies
- Update arrow.version to v12.0.1 (#2750) (f92bee5)
- Update dependency com.google.api.grpc:proto-google-cloud-bigqueryconnection-v1 to v2.24.0 (#2811) (b660063)
- Update dependency com.google.cloud:google-cloud-datacatalog-bom to v1.28.0 (#2812) (921716d)
- Update dependency com.google.cloud:google-cloud-shared-dependencies to v3.13.1 (#2806) (df3cd76)
- Update dependency org.junit.vintage:junit-vintage-engine to v5.10.0 (#2808) (694f711)
- Update github/codeql-action action to v2.21.0 (#2803) (c6c536c)
BigQuery Omni is now available in the AWS - US West (Oregon) (aws-us-west-2) and the AWS - Europe (Ireland) (aws-eu-west-1) regions.
BigQuery Storage Write API multiplexing is now generally available (GA). You can use multiplexing in the default stream to write to multiple destination tables with shared connections.
July 28, 2023
Query queues are now generally available (GA). With query queues, BigQuery automatically determines your query concurrency based on available slots rather than a fixed limit. Once the maximum concurrency is reached, additional queries are queued until processing resources are available. Query queues are enabled by default and have been rolled out over the last several weeks; no user action is required and you shouldn't see any degradation in your query performance. You can optionally set the maximum concurrency target for a reservation. You can also control the interactive and batch query queue timeout by using default configurations.
July 24, 2023
A weekly digest of client library updates from across the Cloud SDK.
Java
Changes for google-cloud-bigquery
2.30.1 (2023-07-18)
Dependencies
- Update dependency com.google.api.grpc:proto-google-cloud-bigqueryconnection-v1 to v2.23.0 (#2791) (940301b)
- Update dependency com.google.cloud:google-cloud-datacatalog-bom to v1.27.0 (#2792) (c791066)
2.30.0 (2023-07-17)
Features
- Add missing storage related fields to Table, TableInfo and StandardTableDefinition (#2673) (e3003f4)
- Add support for Search statistics (#2787) (344f695)
Dependencies
- Update dependency com.google.api.grpc:proto-google-cloud-bigqueryconnection-v1 to v2.22.0 (#2777) (078f244)
- Update dependency com.google.cloud:google-cloud-datacatalog-bom to v1.26.0 (#2778) (2ee52c9)
- Update dependency com.google.cloud:google-cloud-shared-dependencies to v3.13.0 (#2786) (dd14eee)
- Update github/codeql-action action to v2.20.1 (#2766) (2014613)
- Update github/codeql-action action to v2.20.4 (#2784) (e886f5f)
- Update ossf/scorecard-action action to v2.2.0 (#2775) (688b2a0)
Python
Changes for google-cloud-bigquery
3.11.4 (2023-07-19)
Bug Fixes
July 20, 2023
Multivariate time series forecasting with the
ARIMA_PLUS_XREG
model in BigQuery ML is now generally available
(GA). This feature lets you perform time series forecasting with extra feature columns. For more information, see the ARIMA_PLUS_XREG
information in the end-to-end user journey topic, and try the
multivariate time-series forecasting from Seattle air quality data tutorial.
BigQuery ML has introduced new Explainable AI capabilities for better model explainability:
- You can now use the
ML.EXPLAIN_FORECAST
function withARIMA_PLUS_XREG
models. - You can use the updated
ML.EXPLAIN_FORECAST
function to get explanations of the holiday effect for holidays in time series forecasting models (bothARIMA_PLUS
andARIMA_PLUS_XREG
). - You can now use the
ML.GLOBAL_EXPLAIN
function with AutoML Tables models for global model explainability. - For Boosted Tree and
Random Forest
models, you can now use the
approx_global_feature_contrib
training option to use fast approximation for global feature contribution computation in model training, and theapprox_feature_contrib
option in theML.EXPLAIN_PREDICT
function to use the fast approximation for local feature contribution computation in model inference.
Now you can also use Vertex Explainable AI on BigQuery ML models that you've registered to the Vertex AI Model Registry. To learn more, see Explainable AI for BigQuery ML models.
July 19, 2023
BigQuery can now use search indexes to optimize some queries that contain the equal operator (=
), IN
operator, LIKE
operator, or STARTS_WITH
function to compare string literals with indexed data. This feature is in preview.
July 17, 2023
Primary and foreign key table constraints are now generally available. In addition to the features available in preview, you can now also manage constraints through the BigQuery API and view constraints in the BigQuery console.
A weekly digest of client library updates from across the Cloud SDK.
The google.cloud.bigquery.storage.v1beta2 API package for BigQueryWrite operations is deprecated and will be removed on July 17, 2024. After that date, requests to that package version for use with the BigQuery Storage Write API will fail. Data written to BigQuery using the BigQuery Storage Write API is accessible by using the google.cloud.bigquery.storage.v1 package.
Next steps: If you call the API directly, switch to google.cloud.bigquery.storage.v1, the generally available (GA) version of the API, to prevent any impact on your workflow.
July 12, 2023
The following BigQuery ML feature preprocessing functionality is now generally available (GA).
You can export models that use the TRANSFORM clause for feature preprocessing to the TensorFlow SavedModel format. There are 13 data types supported for TRANSFORM clause input, and 127 SQL functions supported for use within the TRANSFORM clause.
You can also now deploy a model trained with the TRANSFORM clause to Vertex AI and locally.
Use the following functions to perform feature preprocessing:
Custom holiday modeling for time series forecasting is now in preview. This release offers the following features to improve the transparency, flexibility, and explainability of time series forecasting in BigQuery ML:
- New CREATE MODEL syntax to specify custom holiday modeling for time series models.
- The new ML.HOLIDAY_INFO function, which returns the list of holidays being modeled by an ARIMA_PLUS or ARIMA_PLUS_XREG time series forecasting model.
- An updated ML.EXPLAIN_FORECAST function, which includes an explanation of the holiday effect for each holiday included in the model.
A new public table, bigquery-public-data.ml_datasets.holidays_and_events_for_forecasting
, has also been added to provide easy look-up of the built-in holidays used in time series forecasting models.
Try these features with the Use custom holidays in a time-series forecasting model tutorial.
July 06, 2023
Spanner Data Boost lets you execute analytics queries and data exports with near-zero impact to existing workloads on your provisioned Spanner instance. This feature is now generally available (GA) in the following regions:
- asia-northeast1 (Tokyo)
- us-central1 (Iowa)
- southamerica-east1 (São Paulo)
- europe-west1 (Belgium)
- europe-west2 (London)
- europe-west3 (Frankfurt)
July 05, 2023
BigQuery is now available in the Turin (europe-west12) and Doha (me-central1) regions.
You can use the LOAD DATA
SQL statement to load data from Avro, CSV, newline delimited JSON, JSON, ORC, or Parquet files into a table. This feature is generally available (GA).
The slot estimator now provides cost-optimal commitment and autoscale recommendations based on editions pricing and historical performance metrics. This feature is in preview.
The fail-safe period is now generally available (GA). The fail-safe period offers an additional seven days of data storage after the time travel window, so that the data is available for emergency recovery. Billed costs won't include the bytes used for fail-safe storage until July 17th, 2023.
The ability to use physical bytes for storage billing is now generally available (GA). When you set your dataset's storage billing model to use physical bytes, the total active storage costs you are billed for includes the bytes used for time travel and fail-safe storage. For more information, see Dataset storage billing models.
The ability to configure the time travel window is now generally available (GA). You can specify the duration of the time travel window from a minimum of two days to a maximum of seven days.
BigQuery capacity commitments have changed as follows:
- Annual commitments are now only available in Enterprise or Enterprise Plus edition. Flat-rate annual commitments are no longer available. For more information about pricing, see Capacity compute (analysis) pricing.
- Monthly and flex commitments are no longer available. For more information about commitment options, see Capacity commitment plans.
You can now restrict data egress on Analytics Hub listings. This feature is now in preview.
July 03, 2023
A weekly digest of client library updates from across the Cloud SDK.
Go
Changes for bigquery/storage/apiv1beta1
1.52.0 (2023-06-23)
Features
- bigquery/storage: Add estimated physical file sizes to ReadAPI v1 (94ea341)
- bigquery/storage: Add table sampling to ReadAPI v1 (ca94e27)
- bigquery: Support for tables primary and foreign keys (#8055) (93d6a1a)
- bigquery: Update all direct dependencies (b340d03)
Bug Fixes
Python
Changes for google-cloud-bigquery
3.11.3 (2023-06-27)
Bug Fixes
June 30, 2023
Metadata caching is now available for BigLake tables that reference Amazon S3 data. This feature is in preview. Using cached metadata might improve query performance for BigLake tables.
June 29, 2023
Support for the following compliance programs is now generally available (GA):
June 26, 2023
You can now create stored procedures for Apache Spark using Java or Scala. You can also use the Google Cloud console PySpark editor to add options for stored Python procedures for Apache Spark. This feature is in Preview.
A weekly digest of client library updates from across the Cloud SDK.
Java
Changes for google-cloud-bigquery
2.29.0 (2023-06-23)
Features
Dependencies
- Update dependency com.google.cloud:google-cloud-shared-dependencies to v3.12.0 (#2771) (7537e0f)
- Update dependency org.graalvm.buildtools:junit-platform-native to v0.9.23 (#2759) (27ba48a)
- Update dependency org.graalvm.buildtools:native-maven-plugin to v0.9.23 (#2760) (8cddf8f)
2.28.0 (2023-06-19)
Features
Python
Changes for google-cloud-bigquery
3.11.2 (2023-06-21)
Bug Fixes
June 21, 2023
TRUNCATE TABLE
is now supported for multi-statement transactions. This feature is now generally available (GA).
June 20, 2023
Metadata caching is now generally available (GA). Using cached metadata might improve query performance for BigLake tables and object tables that reference large numbers of objects, by allowing the query to avoid listing objects from Cloud Storage.
This release includes support for the following new features:
- Protecting metadata cache data with customer-managed encryption keys.
- Statistics on metadata cache usage.
- Table statistics for better query plan performance.
Metadata cache usage is billed going forward. For more information, see Costs.
BigQuery now supports querying Apache Iceberg tables that are created by open source engines. This feature is now generally available (GA).
June 19, 2023
A weekly digest of client library updates from across the Cloud SDK.
Java
Changes for google-cloud-bigquery
2.27.1 (2023-06-13)
Dependencies
- Update actions/checkout action to v3.5.3 (#2746) (17f8438)
- Update dependency com.google.api.grpc:proto-google-cloud-bigqueryconnection-v1 to v2.21.0 (#2741) (d665e52)
- Update dependency com.google.cloud:google-cloud-datacatalog-bom to v1.25.0 (#2743) (5d38d23)
- Update dependency com.google.cloud:google-cloud-shared-dependencies to v3.11.0 (#2738) (3b56445)
- Update github/codeql-action action to v2.20.0 (#2751) (42ae181)
- Update github/codeql-action action to v2.3.6 (#2712) (f043ed6)
Python
Changes for google-cloud-bigquery
3.11.1 (2023-06-09)
Documentation
June 15, 2023
The following Generative AI features are now in preview with allowlist:
- Creating a remote model based on the Vertex AI large language model (LLM)
text-bison
. - Using the
ML.GENERATE_TEXT
function with an LLM-based remote model to perform generative natural language tasks on text stored in BigQuery tables.
Try these features with the Generate text by using a remote model and the ML.GENERATE_TEXT function tutorial.
June 14, 2023
BigQuery now provides information about the fail-safe period. The fail-safe period offers an additional seven days of data storage after the time travel window, so that the data is available for emergency recovery. This feature is in preview.
The INFORMATION_SCHEMA
views that show table storage metadata are now generally available (GA):
- Use the
TABLE_STORAGE
view to get a snapshot of current storage usage for tables at the project level. - Use the
TABLE_STORAGE_BY_ORGANIZATION
view to get a snapshot of current storage usage for tables at the organization level.
BigLake Metastore is now generally available (GA). You can use BigLake Metastore to access and manage Iceberg table metadata from multiple sources.
June 12, 2023
The query execution graph is now generally available (GA). You can use the query execution graph to diagnose query performance issues, and to receive query performance insights.
June 05, 2023
A weekly digest of client library updates from across the Cloud SDK.
Java
Changes for google-cloud-bigquery
2.27.0 (2023-05-30)
Features
Bug Fixes
Dependencies
- Update dependency com.google.api.grpc:proto-google-cloud-bigqueryconnection-v1 to v2.20.0 (#2720) (4962cac)
- Update dependency com.google.apis:google-api-services-bigquery to v2-rev20230506-2.0.0 (#2707) (4d2ec07)
- Update dependency com.google.apis:google-api-services-bigquery to v2-rev20230520-2.0.0 (#2723) (5c64797)
- Update dependency com.google.cloud:google-cloud-bigquerystorage-bom to v2.37.2 (#2726) (052c47a)
- Update dependency com.google.cloud:google-cloud-datacatalog-bom to v1.24.0 (#2721) (7c357fb)
- Update dependency com.google.cloud:google-cloud-shared-dependencies to v3.10.1 (#2713) (744e83a)
Python
Changes for google-cloud-bigquery
3.11.0 (2023-06-01)
Features
Bug Fixes
May 25, 2023
The BigQuery partitioning and clustering recommender is now in preview. The recommender analyzes your BigQuery tables to identify partitioning or clustering opportunities for potential cost savings. You can view partition or cluster recommendations through the BigQuery UI or recommender API. You can also apply recommendations directly to your BigQuery tables.
May 23, 2023
DML statements no longer count toward the number of table or partitioned tables modifications per day. The limit of table and partitioned table modifications has not changed.
May 22, 2023
A weekly digest of client library updates from across the Cloud SDK.
Java
Changes for google-cloud-bigquery
2.26.1 (2023-05-16)
Bug Fixes
2.26.0 (2023-05-15)
Features
- Add field in HivePartitioningOptions (#2678) (4165e55)
- Allow passing autodetect_schema on table update (#2661) (4c01698)
Bug Fixes
- Move ratio calculation for whether to use read API to avoid NPE with setUseReadAPI(false) (#2509) (e1326c8)
Dependencies
- Update arrow.version to v12 (major) (#2675) (7700cf5)
- Update dependency com.google.api.grpc:proto-google-cloud-bigqueryconnection-v1 to v2.19.0 (#2691) (1939803)
- Update dependency com.google.cloud:google-cloud-datacatalog-bom to v1.23.0 (#2692) (f56e541)
- Update dependency com.google.cloud:google-cloud-shared-dependencies to v3.9.0 (#2685) (b74da29)
- Update dependency org.graalvm.buildtools:junit-platform-native to v0.9.22 (#2687) (cf5d758)
- Update dependency org.graalvm.buildtools:native-maven-plugin to v0.9.22 (#2688) (32ea8ab)
- Update github/codeql-action action to v2.3.3 (#2658) (487f207)
May 19, 2023
EXTERNAL_QUERY
SQL pushdown optimizes data retrieval from external sources like Cloud SQL or Cloud Spanner databases. Transferring less data reduces execution time and cost. SQL pushdown encompasses both column pruning (SELECT
clauses) and filter pushdowns (WHERE
clauses). SQL pushdown applies to SELECT * FROM T
queries, a significant percentage of all federated queries. Not all data types are supported for filter pushdowns. This feature is generally available (GA).
May 18, 2023
You can now sort your query results by using the sort menu next to a column name. This feature is in preview.
May 16, 2023
The VPC Service Controls perimeter that protects the BigQuery API now also protects the BigQuery Reservation API. Customers who have already configured VPC Service Controls for the BigQuery API or the BigQuery Reservation API should update their configurations to reflect this change. For more information, see BigQuery Reservation API.
May 15, 2023
A weekly digest of client library updates from across the Cloud SDK.
Go
Changes for bigquery/storage/apiv1beta1
1.51.2 (2023-05-08)
Bug Fixes
- bigquery: Update grpc to v1.55.0 (1147ce0)
BigQuery Omni is now available in the AWS - Asia Pacific (Seoul) (aws-ap-northeast-2) region.
May 11, 2023
Object tables are now generally available (GA).
Object tables are read-only tables containing metadata for unstructured data stored in Cloud Storage. They enable you to analyze and perform inference on images, audio files, documents and other file types by using BigQuery ML and BigQuery remote functions. Object tables extend the data security and governance best practices currently applied to structured data to unstructured data as well.
The GA release includes the following new and updated functions:
ML.DECODE_IMAGE
: Decodes image data so that it can be interpreted by theML.PREDICT
function.ML.CONVERT_COLOR_SPACE
: Converts images with an RGB color space to a different color space.ML.CONVERT_IMAGE_TYPE
: Converts the data type of the pixel values in an image.ML.RESIZE_IMAGE
: Resizes images.ML.DISTANCE
: Computes the distance between two vectors.ML.LP_NORM
: Computes the Lᵖ norm for a vector, where ᵖ is the degree.
May 10, 2023
BigQuery is now available in the Dallas (us-south1) region.
May 09, 2023
You can now view BI Engine Top Tables Cached Bytes, BI Engine Query Fallback Count, and Query Execution Count as dashboard metrics for BigQuery. This feature is now generally available (GA).
EXTERNAL_QUERY SQL pushdown optimizes data retrieval from external sources like Cloud SQL or Cloud Spanner databases. Transferring less data reduces execution time and cost. SQL pushdown encompasses both column pruning (SELECT
clauses) and filter pushdowns (WHERE
clauses). SQL pushdown applies to SELECT * FROM T
queries, a significant percentage of all federated queries. Pushdowns have limitations, for example not all data types are supported for filter pushdowns. This feature is generally available (GA).
May 08, 2023
Differential privacy is now in preview and includes four differential privacy aggregate functions that can be used to anonymize data: AVG
, COUNT
, SUM
, and PERCENTILE_CONT
. To learn more, see the following topics:
A weekly digest of client library updates from across the Cloud SDK.
INFORMATION_SCHEMA.MATERIALIZED_VIEW view and enhanced job statistics now let you monitor materialized view usage and refresh jobs. This feature is in preview.
May 05, 2023
The INSERT INTO SELECT
statement now lets you filter data from files in Amazon S3 and Azure Blob Storage and append it into BigQuery tables. This feature is in preview.
May 04, 2023
You can now use configuration YAML files to transform SQL code when you translate SQL queries from your source database. Configuration YAML files can be used with the batch SQL translator, the interactive SQL translator, and the batch translation Python client. This feature is now in preview.
May 03, 2023
The table clones feature of BigQuery is now generally available (GA).
May 01, 2023
A weekly digest of client library updates from across the Cloud SDK.
Java
Changes for google-cloud-bigquery
2.25.0 (2023-04-27)
Features
Dependencies
- Update dependency com.google.api.grpc:proto-google-cloud-bigqueryconnection-v1 to v2.18.0 (#2648) (29bd415)
- Update dependency com.google.apis:google-api-services-bigquery to v2-rev20230408-2.0.0 (#2650) (b9c2f60)
- Update dependency com.google.cloud:google-cloud-bigquerystorage-bom to v2.36.1 (fea119b)
- Update dependency com.google.cloud:google-cloud-datacatalog-bom to v1.22.0 (#2649) (b6326f3)
- Update dependency com.google.cloud:google-cloud-shared-dependencies to v3.8.0 (#2659) (691a47a)
- Update dependency org.junit.vintage:junit-vintage-engine to v5.9.3 (#2660) (319f98e)
You can now add descriptions to the columns of a view. To do this, use the CREATE VIEW
or ALTER COLUMN
DDL statements. This feature is in preview.
If you use query queues, then you can set the interactive and batch queue timeouts in your default configuration. This feature is in preview.
April 27, 2023
JSON
data type mapping is now available for Cloud Spanner federated queries. This feature is generally available (GA).
April 26, 2023
BigLake and non-BigLake external tables now support Cloud Storage custom dual-regions. This feature is generally available (GA).
April 25, 2023
Dynamic data masking has been updated to allow masking on RECORD
columns that have been set to REPEATED
mode. Previously, querying such columns when data masking had been applied would return internal errors. This feature is generally available (GA).
April 24, 2023
A weekly digest of client library updates from across the Cloud SDK.
Go
Changes for bigquery/storage/apiv1beta1
1.51.0 (2023-04-17)
Features
- bigquery/storage/managedwriter: Expose connection multiplexing as experimental (#7673) (3b8bfb4)
- bigquery: Add Iceberg as DataFormat option. (#7155) (7a9e211)
- bigquery: Expose dataset MaxTimeTravelHours (#7706) (22a666c)
Bug Fixes
Python
Changes for google-cloud-bigquery
3.10.0 (2023-04-18)
Features
The BigQuery Data Transfer Service for Google Ads supports the new Google Ads API. The Google Ads connector supports PMax and Discovery campaigns, a limit of 8000 leaf accounts per transfer, the --table_filter
flag, and backwards compatibility. This feature is now generally available (GA).
April 19, 2023
Updates to preferred tables for existing BI engine reservations now take up to ten seconds to propagate, down from five minutes. This feature is generally available (GA).
April 17, 2023
A weekly digest of client library updates from across the Cloud SDK.
Java
Changes for google-cloud-bigquery
2.24.5 (2023-04-14)
Dependencies
- Update actions/checkout action to v3.5.2 (#2630) (95e49fd)
- Update dependency com.google.api.grpc:proto-google-cloud-bigqueryconnection-v1 to v2.16.0 (#2625) (594a7b4)
- Update dependency com.google.apis:google-api-services-bigquery to v2-rev20230401-2.0.0 (#2631) (5d8d9a6)
- Update dependency com.google.cloud:google-cloud-bigquerystorage-bom to v2.35.0 (8439020)
- Update dependency com.google.cloud:google-cloud-datacatalog-bom to v1.20.0 (#2626) (f466b51)
- Update dependency com.google.cloud:google-cloud-shared-dependencies to v3.7.0 (#2637) (e8f07d7)
- Update dependency org.graalvm.buildtools:junit-platform-native to v0.9.21 (#2633) (3e376b1)
- Update dependency org.graalvm.buildtools:native-maven-plugin to v0.9.21 (#2634) (000f720)
- Update github/codeql-action action to v2.2.12 (#2635) (b2f97e9)
- Update ossf/scorecard-action action to v2.1.3 (#2618) (d166401)
Python
Changes for google-cloud-bigquery
1.28.3 (2022-12-14)
Bug Fixes
April 13, 2023
BigQuery supports setting the rounding mode to ROUND_HALF_EVEN
or ROUND_HALF_AWAY_FROM_ZERO
for parameterized NUMERIC
or BIGNUMERIC
columns at the column level. You can specify a default rounding mode at the table or dataset level that is automatically attached to any columns added within those entities. The ROUND() function also accepts the rounding mode as an optional argument. This feature is generally available GA.
April 10, 2023
The limit for maximum result size (20 GiB logical bytes) when querying Azure or Amazon Simple Storage service (S3) data is now generally available (GA). Querying Azure and Amazon S3 data are now subject to the following quotas and limitations:
The maximum row size is 10 MiB. For more information, see Quotas for query jobs.
If your query uses the
ORDER BY
clause and has a result size larger than 256 MB, then your query fails. Previously, this limit was 2 MB. For more information, see Limitations.
A weekly digest of client library updates from across the Cloud SDK.
Go
Changes for bigquery/storage/apiv1beta1
1.50.0 (2023-04-03)
Features
- bigquery/connection: Add spark connection properties type (#7570) (499b489)
- bigquery/migration: Add request_source field and update formatting (#7586) (c967961)
- bigquery/reservation: Add edition/autoscale related fields (#7608) (2b7bb66)
- bigquery/storage/managedwriter: Decouple connections and writers (#7314) (7d085b4)
- bigquery/storage/managedwriter: Introduce location routing header (#7663) (cf06802)
Bug Fixes
Documentation
- bigquery/reservation: Mention that some fields are deprecated (597ea0f)
The results for queries against table snapshots can now be returned from cache.
April 06, 2023
The add data demo guide walks you through the process of adding data to BigQuery through popular sources and is now in preview.
April 05, 2023
Non-incremental materialized views support most SQL queries, including OUTER
JOIN
, UNION
, and HAVING
clauses, as well as analytic functions. This feature is in preview.
April 04, 2023
BigQuery is now available in the Israel (me-west1) region.
April 03, 2023
A weekly digest of client library updates from across the Cloud SDK.
Java
Changes for google-cloud-bigquery
2.24.4 (2023-03-30)
Bug Fixes
Dependencies
- Update dependency com.google.apis:google-api-services-bigquery to v2-rev20230318-2.0.0 (#2607) (a328eb2)
- Update dependency com.google.cloud:google-cloud-bigquerystorage-bom to v2.34.2 (#2619) (e4aa0fe)
- Update dependency com.google.cloud:google-cloud-shared-dependencies to v3.6.0 (#2612) (eac97ac)
- Update github/codeql-action action to v2.2.9 (#2608) (24aac14)
Python
Changes for google-cloud-bigquery
3.9.0 (2023-03-28)
Features
Bug Fixes
- Keyerror when the load_table_from_dataframe accesses a unmapped dtype dataframe index (#1535) (a69348a)
3.8.0 (2023-03-24)
Features
- Add bool, int, float, string dtype to to_dataframe (#1529) (5e4465d)
- Add default LoadJobConfig to Client (#1526) (a2520ca)
- Expose configuration property on CopyJob, ExtractJob, LoadJob, QueryJob (#1521) (8270a10)
Bug Fixes
March 30, 2023
BigQuery ML documentation is now integrated with BigQuery documentation to unify resources for data analysis and machine learning tasks such as inference. BigQuery ML documentation resources include:
BigQuery Partner Center, which can be used to discover and try validated partner applications, is now generally available (GA). In addition, the Google Cloud Ready - BigQuery initiative has added 14 new partners.
March 29, 2023
Compute (analysis) is now generally available (GA) in three new BigQuery editions: Standard, Enterprise, and Enterprise Plus. These editions support the slots autoscaling model to meet your organizations' needs and budgets.
Autoscaling slots are now generally available (GA). Autoscaling slot reservations and commitments created during the feature's preview have been set to BigQuery Enterprise edition.
March 28, 2023
You can now use the tf_version
training option to specify the Tensorflow (TF) version during model training. By default, tf_version
is set as '1.15'. If you want to use TF2 with Keras API, you can add tf_version
= '2.8.0' when creating the model.
You can now use the xgboost_version
training option to specify the XGBoost version during model training. By default, xgboost_version
is set as '0.9'. You can choose XGBoost version 1.1 by specifying xgboost_version
= '1.1'.
You can now use the instance_weight_col
training option to identify the column containing weights for each data point in the training dataset. Currently the instance_weight_col
option is only available for boosted tree and random forest models with non-array feature types.
You can now import model artifacts saved in ONNX, XGBoost, and TensorFlow Lite formats into BigQuery for inference, allowing you to leverage models built in popular frameworks directly within the BigQuery ML inference engine.
You can also host models remotely on Vertex AI Prediction and do inference with BigQuery ML, removing the need to build data pipelines manually.
You can do inference with Google Cloud's state of the art pretrained models using Cloud AI service table-valued functions (TVFs) to get insights from your data. The TVFs work with Cloud Vision API, Cloud Natural Language API and Cloud Translation API.
These features are in preview. To enroll to use this feature, complete the enrollment form.
March 27, 2023
A weekly digest of client library updates from across the Cloud SDK.
Node.js
Changes for @google-cloud/bigquery
6.2.0 (2023-03-22)
Features
- Add collation feature tests (#1188) (80d86ba)
- Add support for microseconds precision (#1192) (b5801a6)
Bug Fixes
Java
Changes for google-cloud-bigquery
2.24.3 (2023-03-24)
Dependencies
2.24.2 (2023-03-22)
Dependencies
2.24.1 (2023-03-21)
Dependencies
- Update cloud client dependencies (7b07779)
- Update dependency com.google.cloud:google-cloud-bigquery to v2.23.2 (7b07779)
- Update dependency com.google.cloud:google-cloud-bigquery to v2.24.0 (7b07779)
- Update dependency com.google.cloud:google-cloud-bigquerystorage-bom to v2.34.1 (7b07779)
- Update dependency com.google.cloud:google-cloud-bigtable to v2.20.1 (7b07779)
- Update dependency com.google.cloud:libraries-bom to v26.10.0 (7b07779)
2.24.0 (2023-03-21)
Features
Dependencies
- Update actions/checkout action to v3.4.0 (#2575) (6935a1e)
- Update actions/upload-artifact action to v3.1.2 (#2571) (aa0c70e)
- Update cloud client dependencies (#2583) (dcacc31)
- Update dependency com.google.api.grpc:proto-google-cloud-bigqueryconnection-v1 to v2.15.0 (#2577) (eaf09d6)
- Update dependency com.google.apis:google-api-services-bigquery to v2-rev20230311-2.0.0 (#2578) (aab037c)
- Update dependency com.google.cloud:google-cloud-shared-dependencies to v3.5.0 (#2580) (1764eeb)
- Update dependency com.google.cloud:google-cloud-storage to v2.20.0 (#2559) (8a854db)
- Update github/codeql-action action to v2.2.7 (#2572) (105f5ee)
BigQuery now supports change data capture (CDC) by processing and applying streamed changes in real-time to existing data using the BigQuery Storage Write API. This feature is in preview.
March 22, 2023
BigQuery now supports Unicode column naming using international character sets, alphanumeric and special characters. Existing columns can use these new capabilities using the RENAME
command. This feature is now in preview.
March 20, 2023
A weekly digest of client library updates from across the Cloud SDK.
Go
Changes for bigquery/storage/apiv1beta1
1.49.0 (2023-03-16)
Features
Python
Changes for google-cloud-bigquery
3.7.0 (2023-03-06)
Features
- Add
connection_properties
andcreate_session
toLoadJobConfig
(#1509) (cd0aaa1) - Add default_query_job_config property and property setter to BigQuery client (#1511) (a23092c)
Documentation
The following AutoML Tables model features are now generally available:
- Availability in additional regions.
- CMEK support in available regions except multi-regions US and EU.
- OPTIMIZATION_OBJECTIVE now accepts two additional options:
- MAXIMIZE_PRECISION_AT_RECALL
- MAXIMIZE_RECALL_AT_PRECISION
March 14, 2023
The Lineage tab in the table properties page lets you track how your data moves and transforms through BigQuery. This feature is now generally available (GA).
March 13, 2023
A weekly digest of client library updates from across the Cloud SDK.
You can now specify translation configurations in the BigQuery Interactive SQL Translator and use it to debug Batch SQL translator jobs. This feature is now in preview.
March 10, 2023
The CREATE TABLE AS SELECT
statement now lets you filter data from files in Amazon S3 and Azure Blob Storage before transferring results into BigQuery tables This feature is in preview.
March 07, 2023
Case-insensitive collation support is now generally available (GA). In addition to features available in the preview, the GA release includes:
- MIN, MAX, COUNT with DISTINCT, and PERCENTILE_DISC windows functions
- ORDER BY and PARTITION BY in the WINDOWS clause
- LIKE operator with limitations
- Views
- Materialized views with limitations
- Table functions with limitations
- BigQuery BI engine
March 06, 2023
A weekly digest of client library updates from across the Cloud SDK.
Go
Changes for bigquery/storage/apiv1beta1
1.48.0 (2023-03-01)
Features
Java
Changes for google-cloud-bigquery
2.23.1 (2023-03-02)
Dependencies
March 02, 2023
The WITH RECURSIVE
clause is now generally available (GA). This clause lets you include one or more recursive common table expressions (CTEs) in a query.
February 27, 2023
A weekly digest of client library updates from across the Cloud SDK.
Go
Changes for bigquery/storage/apiv1beta1
1.47.0 (2023-02-22)
Features
- bigquery/storage: Add default_value_expression to TableFieldSchema (#7400) (1244b3f)
- bigquery: Add support for session in load jobs (#7418) (f9ff2ca)
Bug Fixes
Java
Changes for google-cloud-bigquery
2.23.0 (2023-02-22)
Features
Dependencies
- Update cloud client dependencies (#2526) (4d88ccc)
- Update dependency com.google.api.grpc:proto-google-cloud-bigqueryconnection-v1 to v2.13.0 (#2533) (ed2cb74)
- Update dependency com.google.apis:google-api-services-bigquery to v2-rev20230210-2.0.0 (#2530) (62ff092)
- Update dependency com.google.cloud:google-cloud-shared-dependencies to v3.3.0 (#2534) (f1bcc33)
- Update dependency org.graalvm.buildtools:junit-platform-native to v0.9.20 (#2527) (5fe5e74)
- Update dependency org.graalvm.buildtools:native-maven-plugin to v0.9.20 (#2528) (554e75d)
Python
Changes for google-cloud-bigquery
3.6.0 (2023-02-22)
Features
Bug Fixes
- Annotate optional integer parameters with optional type (#1487) (a190aaa)
- Loosen ipywidget dependency (#1504) (20d3276)
- Removes scope to avoid unnecessary duplication (#1503) (665d7ba)
Dependencies
- Update minimum google-cloud-core to 1.6.0 (a190aaa)
You can set default values on columns in your BigQuery tables. This feature is now generally available (GA).
The multivariate time-series forecasting model ARIMA_PLUS_XREG
is now available to on-demand users.
February 23, 2023
Authorized stored procedures are now in preview. This feature lets you share stored procedures with users or groups without giving them direct access to the underlying tables.
February 22, 2023
Fixed linked datasets querying shared dataset that has data ingested through streaming inserts or the BigQuery Storage Write API.
Primary and foreign key table constraints are now available in preview. You can define table constraints using the CREATE TABLE
statement, the ALTER TABLE ADD PRIMARY KEY
statement, or the ALTER TABLE ADD CONSTRAINT
statement.
February 21, 2023
The ALTER TABLE RENAME COLUMN statement and the ALTER TABLE DROP COLUMN statement are now generally available (GA).
February 15, 2023
You can now make a dataset and the tables in that dataset case-insensitive when you create a dataset or alter a dataset. This feature is generally available (GA).
You can now run bq
commands using service account impersonation. This feature is generally available (GA).
In the Explorer pane, the resource corresponding to the focused tab is now selected. This feature is generally available (GA).
In the Explorer pane, you can now see all the resources in the searched resource's level by clicking Show more. This feature is generally available (GA).
February 13, 2023
A weekly digest of client library updates from across the Cloud SDK.
Go
Changes for bigquery/storage/apiv1beta1
1.46.0 (2023-02-06)
Features
- bigquery: Add dataset/table collation (#7235) (9f7bbeb)
- bigquery: Use storage api for query jobs (#6822) (26c04f4)
Bug Fixes
Java
Changes for google-cloud-bigquery
2.22.0 (2023-02-08)
Features
Dependencies
- Update arrow.version to v11 (major) (#2495) (94ed060)
- Update dependency com.google.api.grpc:proto-google-cloud-bigqueryconnection-v1 to v2.11.0 (#2482) (e6ffb9b)
- Update dependency com.google.api.grpc:proto-google-cloud-bigqueryconnection-v1 to v2.12.0 (#2512) (09f280d)
- Update dependency com.google.cloud:google-cloud-bigquerystorage-bom to v2.31.0 (#2499) (c0a393c)
- Update dependency com.google.cloud:google-cloud-datacatalog-bom to v1.15.0 (#2483) (5c2bf69)
- Update dependency com.google.cloud:google-cloud-shared-dependencies to v3.2.0 (#2513) (02832dd)
You can now create materialized views over BigLake metadata cache-enabled tables to reference structured data stored in Cloud Storage. This feature is in preview.
February 09, 2023
You can now apply four new types of dynamic data masking to table columns in BigQuery. These new data masking types include date year, email, first four characters, and last four characters masks. This feature is generally available (GA).
Autoscaling slot reservations are now available in preview. You can create autoscaling reservations and associated commitments using slots autoscaling.
Cloud console updates: In the Explorer pane, you can now refresh the contents of a resource (project or dataset). To refresh the contents of a resource, click
View actions, and then click Refresh contents.February 06, 2023
The HAVING MAX
and HAVING MIN
clause for the ANY_VALUE
function is now in preview.
You can now view information related to query processing to monitor and optimize queries with the query_info
column in INFORMATION_SCHEMA.JOBS
, JOBS_BY_FOLDER
and JOBS_BY_ORGANIZATION
views. This feature is generally available (GA).
A weekly digest of client library updates from across the Cloud SDK.
February 01, 2023
The BigQuery Data Transfer Service can now transfer data from Azure Blob Storage into BigQuery. This feature is now in preview.
January 31, 2023
Azure workload identity federation is now generally available (GA) for BigQuery Omni connections. You can now create a connection for federated identity using Google Cloud console.
Cloud console updates: When you create datasets, select locations to run specific queries, or create exchanges in Analytics Hub, you now see separate options for multi-region and specific regions. Based on your selection, you see a list with more options.
January 30, 2023
You can search for BigQuery partners in the BigQuery Partner Center. This feature is in Preview.
A weekly digest of client library updates from across the Cloud SDK.
Node.js
Changes for @google-cloud/bigquery
6.1.0 (2023-01-25)
Features
Java
Changes for google-cloud-bigquery
2.21.0 (2023-01-23)
Features
Bug Fixes
Dependencies
- Update com.google.cloud:google-cloud-bigquerystorage-bom to v2.28.3 (5a20c32)
- Update com.google.cloud:google-cloud-datacatalog-bom to v1.14.0 (606fc1e)
- Update com.google.cloud:google-cloud-storage to v2.17.1 (#2465) (606fc1e)
- Update com.google.cloud:google-cloud-storage to v2.17.2 (#2479) (5a20c32)
- Update dependency com.google.api.grpc:proto-google-cloud-bigqueryconnection-v1 to v2.10.0 (#2463) (5bfa7ae)
- Update dependency com.google.apis:google-api-services-bigquery to v2-rev20230114-2.0.0 (#2477) (b6409d5)
- Update dependency com.google.cloud:google-cloud-shared-dependencies to v3.1.2 (#2476) (688b6a6)
January 23, 2023
A weekly digest of client library updates from across the Cloud SDK.
Python
Changes for google-cloud-bigquery
3.4.2 (2023-01-13)
Bug Fixes
Dependencies
Documentation
- Create sample to write schema file from table (#1439) (093cc68)
- Created samples for load table and create table from schema file (#1436) (8ad2e5b)
- Revise create table cmek sample (#1452) (57740e4)
- Revise get table labels code sample, add TODO to clean up snipp… (#1464) (b5ccbfe)
- Revise label table code samples (#1451) (14ae1f2)
- Revise sample for nested schema (#1446) (a097631)
January 19, 2023
More than 20 BigQuery ML components for Vertex AI Managed Pipelines are now generally available. These components benefit AI/ML users for the following:
- Building pipelines using the KFP SDK and TFX SDK
- Linking and tracking metadata automatically
- Seamless integration with Vertex AI for online prediction
Major Google Cloud pipeline components available in Vertex AI are.
You can now attach Resource Manager tags to datasets, which let you conditionally apply Identity and Access Management (IAM) policies to your resources. This feature is generally available (GA).
You can now use the TRANSFORM
clause to train models which you can then export in the Tensorflow SavedModel format. This feature is now available in preview.
The following functions have been added for BigQuery ML:
These features are now available in preview.
January 17, 2023
BigQuery ML support for multivariate time-series forecasting with the ARIMA_PLUS_XREG
model is now available in preview. This feature lets you perform time-series forecasting with extra feature columns. For more information, see the ARIMA_PLUS_XREG
sections in the end-to-end user journey and the multivariate time-series forecasting from Seattle air quality data tutorial.
Sparse input support in BigQuery ML model training is now generally available (GA). This feature improves model training for data whose values are mostly zero or empty. For additional examples, see the sparse features support in BigQuery blog.
January 16, 2023
A weekly digest of client library updates from across the Cloud SDK.
Java
Changes for google-cloud-bigquery
2.20.2 (2023-01-12)
Dependencies
- Update com.google.cloud:google-cloud-bigquerystorage-bom to 2.28.1 (b62391d)
- Update dependency com.google.apis:google-api-services-bigquery to v2-rev20221209-2.0.0 (#2449) (9d0a107)
- Update dependency com.google.apis:google-api-services-bigquery to v2-rev20221217-2.0.0 (#2459) (08e2927)
- Update dependency com.google.cloud:google-cloud-shared-dependencies to v3.1.1 (#2455) (412710c)
- Update dependency org.junit.vintage:junit-vintage-engine to v5.9.2 (#2456) (04d47ad)
- Update dependency org.threeten:threeten-extra to v1.7.2 (#2450) (485be56)
January 10, 2023
The ALTER CAPACITY SET OPTIONS
statement and ALTER RESERVATION SET OPTIONS
statement are now generally available (GA). Additionally, the CREATE CAPACITY
, CREATE RESERVATION
, and CREATE ASSIGNMENT
statements now support the OPTIONS
clause.
January 09, 2023
The following generally available (GA) features have been added for sessions:
In a session, temporary functions are now maintained until the session ends.
In a session, statements that include the
TEMP
keyword can also include theOR REPLACE
andIF NOT EXISTS
keywords.
A weekly digest of client library updates from across the Cloud SDK.
Go
Changes for bigquery/storage/apiv1beta1
1.45.0 (2023-01-05)
Features
Java
Changes for google-cloud-bigquery
2.20.1 (2023-01-04)
Dependencies
January 04, 2023
BigQuery ML support for image analytics with vision models is available in preview. Customers can import vision models to perform inference modeling with images to detect objects, perform optical character recognition (OCR), and more. To request access to these features, complete the BigQuery ML interest sign up form.
This new capability uses BigQuery object tables to access image data stored in Cloud Storage and predict results from machine learning models. You can now generate insights from structured and unstructured data with the following steps:
- Create an object table to access images stored in Cloud Storage.
- Import vision models in TensorFlow vision models such as ImageNet or ResNet 50, or import your own models to detect objects from images, to annotate photos, and to perform OCR.
- Unify image data with structured data such as user activities or sales orders to train machine learning models. You can then use prediction results to extract insights from your data.
January 03, 2023
BigQuery ML integration with Vertex AI Model Registry is now generally available. With this integration, you can now use the following capabilities:
- Register and monitor BigQuery ML models with Vertex AI Model Registry
- Deploy BigQuery ML models directly from Vertex AI Model Registry to Vertex Deployment endpoints
- Use Vertex AI to compare and track evaluation metrics.
- Explainable AI for BigQuery ML models, including built-in XAI, inside Vertex AI
- The seamless integration between BigQuery ML and Vertex AI lets you use Vertex AI for MLOps.
Key features include:
- Model versioning for models registered with Vertex AI Model Registry
- Revision alias for different model versions, and User specified model ID
- List the models by type (custom model, BigQuery ML, AutoML)
- BigQuery ML models can be registered with Vertex AI Model Registry to help you explore, manage, and govern your BigQuery ML models
- Ability to deploy BigQuery ML models to Vertex AI end points
- BigQuery ML models deployed on Vertex AI endpoints can use MLOps features such as model monitoring
Customers can use BigQuery ML to train and run models on BigLake in Cloud Storage. See Data Cloud Blog and End to end unstructured data use cases demo.
December 22, 2022
The Lineage tab in the table properties page lets you track how your data moves and transforms through BigQuery. This feature is now in preview.
BigQuery now blocks saving query results to Google Drive from projects inside a VPC Service Controls protected perimeter.
December 19, 2022
A weekly digest of client library updates from across the Cloud SDK.
Java
Changes for google-cloud-bigquery
2.20.0 (2022-12-13)
Features
- Add fast query path support when empty jobId object is passed (#2349) (42c083a)
- Next release from main branch is 2.20.0 (#2405) (9297a43)
Dependencies
- Update arrow.version to v10.0.1 (#2426) (aff9019)
- Update cloud client dependencies (#2444) (7255357)
- Update dependency com.google.api.grpc:proto-google-cloud-bigqueryconnection-v1 to v2.8.0 (#2418) (1ac1653)
- Update dependency com.google.api.grpc:proto-google-cloud-bigqueryconnection-v1 to v2.9.0 (#2441) (01cc3c3)
- Update dependency com.google.apis:google-api-services-bigquery to v2-rev20221127-2.0.0 (#2437) (eb52002)
- Update dependency com.google.cloud:google-cloud-datacatalog-bom to v1.12.0 (#2419) (c449031)
- Update dependency com.google.cloud:google-cloud-shared-dependencies to v3.1.0 (#2435) (c99b215)
- Update dependency com.google.cloud:google-cloud-storage to v2.15.1 (#2420) (baf337a)
- Update dependency org.graalvm.buildtools:junit-platform-native to v0.9.18 (#2424) (63b5196)
- Update dependency org.graalvm.buildtools:junit-platform-native to v0.9.19 (#2432) (396c6dc)
- Update dependency org.graalvm.buildtools:native-maven-plugin to v0.9.18 (#2425) (cd2ae9f)
- Update dependency org.graalvm.buildtools:native-maven-plugin to v0.9.19 (#2433) (bf94087)
- Update jmh.version to v1.36 (#2415) (0676586)
Python
Changes for google-cloud-bigquery
3.4.1 (2022-12-09)
Documentation
Dependencies
- make
pyarrow
andBQ Storage
optional dependencies (e1aa921)
December 15, 2022
You can now access and query Cloud SQL data over a private connection. This feature is generally available (GA).
December 08, 2022
The demo query guide helps you query a public dataset from Google Trends and is now in preview.
December 05, 2022
A weekly digest of client library updates from across the Cloud SDK.
Go
Changes for bigquery/storage/apiv1beta1
1.44.0 (2022-11-30)
Features
- bigquery/datatransfer: Add Location API methods docs: updated comments (22ec3e3)
- bigquery/storage: Add missing_value_interpretations to AppendRowsRequest (2a0b1ae)
- bigquery: Add default partition expiration to Dataset (#7096) (601c77a), refs #7021
- bigquery: Remove code for reservation/apiv1beta1 (#7010) (451acc1)
- bigquery: Start generating proto stubs (#7026) (debc4c7)
- bigquery: Widen retry predicate (#6976) (753b751)
Bug Fixes
Documentation
- bigquery/storage: Remove stale header guidance for AppendRows (9c5d6c8)
December 01, 2022
BigQuery now supports querying Apache Iceberg tables that are created by open source engines. This feature is in preview.
November 28, 2022
BigQuery now supports the following features when you load data:
- ASCII control characters for CSV files.
- Reference file with the expected table schema for creating external tables with Avro, ORC, and Parquet files.
These features are generally available (GA).
November 21, 2022
A weekly digest of client library updates from across the Cloud SDK.
November 17, 2022
Object tables are now in preview. Object tables are read-only tables containing metadata for unstructured data stored in Cloud Storage. These tables enable you to analyze and perform inference on images, audio files, documents, and other file types by using BigQuery ML and BigQuery remote functions. Object tables extend structured data features such as data security and governance best practices to unstructured data.
Metadata caching is now in preview. Using cached metadata might improve query performance for BigLake tables and object tables that reference large numbers of objects, by allowing the query to avoid listing objects from Cloud Storage.
November 14, 2022
A weekly digest of client library updates from across the Cloud SDK.
Java
Changes for google-cloud-bigquery
2.19.1 (2022-11-08)
Dependencies
2.19.0 (2022-11-07)
Features
Dependencies
- Update dependency com.google.apis:google-api-services-bigquery to v2-rev20221028-2.0.0 (#2393) (d3f6a6b)
- Update dependency com.google.cloud:google-cloud-shared-dependencies to v3.0.6 (#2399) (9de9aa8)
- Update dependency org.graalvm.buildtools:junit-platform-native to v0.9.17 (#2396) (87f8cdd)
- Update dependency org.graalvm.buildtools:native-maven-plugin to v0.9.17 (#2397) (7927350)
The slot estimator helps you manage slot capacity based on historical performance metrics. This feature is now generally available (GA).
November 09, 2022
You can now transfer data from Amazon S3 and Azure Blob Storage to BigQuery using the LOAD DATA
statement. This feature is generally available (GA) and includes support for the following features:
- Transfer files that are hive partitioned.
- Load semi-structured JSON source data into BigQuery without providing a schema by using JSON columns in the destination table.
- Encrypt destination tables using customer managed encryption keys.
- Transfer data to
US
multi-region andUS-EAST-4
regions.
November 07, 2022
A weekly digest of client library updates from across the Cloud SDK.
Python
Changes for google-cloud-bigquery
3.3.6 (2022-11-02)
Features
Bug Fixes
- Corrects test for non-existent attribute (#1395) (a80f436)
- deps: Allow protobuf 3.19.5 (#1379) (3e4a074)
- deps: Allow pyarrow < 11 (#1393) (c898546)
- deps: Require requests>=2.21.0 (#1388) (e398336)
- Refactor to adapt to changes to shapely dependency (#1376) (2afd278)
Documentation
Miscellaneous Chores
- release 3.3.6 (4fce1d9)
In the Explorer pane, you can now star your projects, datasets, and tables. This feature replaces the pin feature, which formerly allowed you to pin projects to the Explorer pane. This feature is now generally available (GA).
In the Cloud console, the Add data feature lets you access popular ways to search for and ingest data sources that work with BigQuery. For an example, see viewing listings in Analytics Hub. This feature is now generally available (GA).
November 03, 2022
SQL functions for managing wrapped keysets are generally available (GA). You can now perform the following actions natively in BigQuery with fewer risks and steps:
- Create a wrapped keyset
- Rotate a wrapped keyset
- Rewrap a wrapped keyset
- Encrypt and decrypt a column with a wrapped keyset
Included with this release are the following new key management functions:
November 02, 2022
The query execution graph is now in preview. You can use the query execution graph to diagnose query performance issues, and to receive query performance insights.
November 01, 2022
The BigQuery migration assessment is now available for Amazon Redshift in preview. You can use this feature to assess the complexity of migrating from your Amazon Redshift data warehouse to BigQuery.
October 31, 2022
A weekly digest of client library updates from across the Cloud SDK.
Go
Changes for bigquery/storage/apiv1beta1
1.43.0 (2022-10-24)
Features
- bigquery/analyticshub: rename nodejs analyticshub library package name (52dddd1)
- bigquery/storage/managedwriter: Enable field name indirection (#6247) (1969273)
- bigquery/storage/managedwriter: Retry on FailedPrecondition (#6761) (d1a444d)
- bigquery/storage/managedwriter: Support append retries (#6695) (6ae9c67)
- bigquery/storage/managedwriter: Switch to opt-in retry (#6765) (a3e97a6)
- bigquery: Add remote function options to routine metadata (#6702) (d9a437d)
Bug Fixes
- bigquery/storage/managedwriter: Address possible resource leak (#6775) (979440b)
- bigquery: Avoid stack overflow on query param with recursive types (#6890) (854ccfc), refs #6884
- bigquery: Bq connection auth scopes (#6752) (8e09288), refs #6744
Documentation
Java
Changes for google-cloud-bigquery
2.18.2 (2022-10-28)
Dependencies
2.18.1 (2022-10-28)
Dependencies
2.18.0 (2022-10-27)
Features
Bug Fixes
- Add --add-opens arg to native-image command (#2369) (8e8b6d7)
- Properly handle external table schema on table update (#2236) (460ef31)
Dependencies
- Update arrow.version to v10 (major) (#2371) (b7873db)
- Update cloud client dependencies (#2362) (0936699)
- Update dependency com.google.api.grpc:proto-google-cloud-bigqueryconnection-v1 to v2.6.0 (#2355) (7bc59a7)
- Update dependency com.google.api.grpc:proto-google-cloud-bigqueryconnection-v1 to v2.7.0 (#2366) (02102d3)
- Update dependency com.google.apis:google-api-services-bigquery to v2-rev20221015-2.0.0 (#2370) (9b796cf)
- Update dependency com.google.cloud:google-cloud-datacatalog-bom to v1.10.0 (#2356) (edb2ca0)
- Update dependency com.google.cloud:google-cloud-shared-dependencies to v3.0.5 (#2361) (51b2258)
- Update dependency com.google.code.gson:gson to v2.10 (#2367) (82e3de5)
- Update dependency org.graalvm.buildtools:junit-platform-native to v0.9.15 (#2352) (b0f172c)
- Update dependency org.graalvm.buildtools:junit-platform-native to v0.9.16 (#2358) (f4e5fc5)
- Update dependency org.graalvm.buildtools:native-maven-plugin to v0.9.15 (#2353) (ac9226c)
- Update dependency org.graalvm.buildtools:native-maven-plugin to v0.9.16 (#2359) (52ec31a)
The max_staleness
materialized view option helps you achieve consistently high performance with controlled costs when processing large, frequently changing datasets. This feature is now in preview.
Column-level data masking is now generally available (GA). You can use data masking to selectively obscure column data for groups of users, while still allowing access to the column.
October 27, 2022
Search indexes and the SEARCH() function are now generally available (GA). These enable you to use Google Standard SQL to efficiently pinpoint specific data elements in unstructured text and semi-structured data.
October 26, 2022
The following geography functions are now generally available (GA):
ST_ISCLOSED
: ReturnsTRUE
for a non-empty geography, where each element in the geography has an empty boundary.ST_ISRING
: Checks if a geography is a linestring and if the linestring is both closed and simple.
October 24, 2022
You can now view BI Engine Top Tables Cached Bytes, BI Engine Query Fallback Count, and Query Execution Count as dashboard metrics for BigQuery. This feature is now in preview.
October 18, 2022
Remote functions, which let you invoke functions from Cloud Functions or Cloud Run in your Google Standard SQL queries, are now generally available (GA).
October 17, 2022
A weekly digest of client library updates from across the Cloud SDK.
Java
Changes for google-cloud-bigquery
2.17.1 (2022-10-10)
Dependencies
Python
Changes for google-cloud-bigquery
3.3.5 (2022-10-10)
Bug Fixes
3.3.5 (2022-10-10)
Bug Fixes
October 12, 2022
The reporting process for the tabledata.list bytes per minute
quota has been updated to more accurately reflect the enforced limit. The limit has not changed.
October 11, 2022
Analytics Hub is now generally available. As an Analytics Hub publisher, you can now view all subscriptions to your listing and remove a subscription from your listing.
You can now use stored procedures for Apache Spark. This feature is in preview.
October 10, 2022
Multi-statement transactions are now generally available (GA).
A weekly digest of client library updates from across the Cloud SDK.
Java
Changes for google-cloud-bigquery
2.17.0 (2022-10-03)
Features
Dependencies
- Update dependency com.google.api.grpc:proto-google-cloud-bigqueryconnection-v1 to v2.5.5 (#2328) (6e48ec2)
- Update dependency com.google.apis:google-api-services-bigquery to v2-rev20220913-2.0.0 (#2287) (fa33184)
- Update dependency com.google.apis:google-api-services-bigquery to v2-rev20220924-2.0.0 (#2325) (82c2097)
- Update dependency com.google.cloud:google-cloud-shared-dependencies to v3.0.4 (#2327) (188c779)
- Update dependency gcp-releasetool to v1.8.9 (#2326) (52dfd13)
- Update dependency importlib-metadata to v4.13.0 (#2323) (4c7e089)
- Update dependency importlib-metadata to v5 (#2324) (bd43cf4)
- Update dependency org.graalvm.buildtools:junit-platform-native to v0.9.14 (#2288) (959519c)
- Update dependency org.graalvm.buildtools:native-maven-plugin to v0.9.14 (#2289) (3cf7ef8)
- Update dependency org.junit.vintage:junit-vintage-engine to v5.9.1 (#2285) (65fac18)
Python
Changes for google-cloud-bigquery
3.3.4 (2022-09-29)
Bug Fixes
The ability to use physical bytes for storage billing is now in Preview. For more information, see Dataset storage billing models.
October 05, 2022
You can now explore query results in Colab using Python libraries. This feature is now in preview.
Concurrent connections quotas are now based on the project that initiates the Storage Write API request, not the project containing the BigQuery dataset resource.
October 03, 2022
A weekly digest of client library updates from across the Cloud SDK.
Python
Changes for google-cloud-bigquery
3.3.3 (2022-09-28)
Bug Fixes
- Refactors code to account for a tdqm code deprecation (#1357) (1369a9d)
- Validate opentelemetry span job attributes have values (#1327) (8287af1)
Documentation
- samples: uses function (create_job) more appropriate to the described sample intent (5aeedaa)
BigQuery supports JSON as a native column type. This feature is now generally available (GA).
September 29, 2022
In addition to standard rounding, BigQuery now supports the rounding mode ROUND_HALF_EVEN
for parameterized NUMERIC
or BIGNUMERIC
columns. The ROUND()
function also accepts the rounding mode as an optional argument. This feature is now in preview.
September 28, 2022
With Datastream for BigQuery, you can now replicate data and schema updates from operational databases directly into BigQuery. This feature is now in preview.
September 26, 2022
The totalItems
field returned by the projects.list
API method now returns the number of items per page, rather than an approximate total number of projects across all pages.
A weekly digest of client library updates from across the Cloud SDK.
Go
Changes for bigquery/storage/apiv1beta1
1.42.0 (2022-09-21)
Features
- bigquery/analyticshub: Start generating apiv1 (#6707) (feb7d7d)
- bigquery/datapolicies: Start generating apiv1beta1 (#6697) (f5443e8)
- bigquery/reservation/apiv1beta1: add REST transport (f7b0822)
- bigquery/storage/managedwriter: Define append retry predicate (#6650) (478b8dd)
- bigquery/storage: add proto annotation for non-ascii field mapping (ec1a190)
- bigquery: Add reference file schema option for federated formats (#6693) (3d26091)
- bigquery: Add support for explicit query parameter type (#6596) (d59b5b2), refs #4704
Bug Fixes
- bigquery/connection: integrate gapic-generator-python-1.4.1 and enable more py_test targets (ec1a190)
In the Explorer pane, you can now open tables in Connected Sheets. This feature is now generally available (GA).
September 21, 2022
BigQuery Omni has introduced support for on-demand pricing model (GA) for a limited duration. For more information, see BigQuery Omni Pricing.
You can now view shuffle usage ratios in the admin resource charts. This feature is now in preview.
September 19, 2022
A weekly digest of client library updates from across the Cloud SDK.
Go
Changes for bigquery/storage/apiv1beta1
1.41.0 (2022-09-14)
Features
- bigquery/storage: add location to WriteStream and add WriteStreamView support (6a0080a)
Bug Fixes
Java
Changes for google-cloud-bigquery
2.16.1 (2022-09-15)
Dependencies
2.16.0 (2022-09-12)
Features
- Add preserveAsciiControlCharacters to CsvOptions (#2143) (856893f)
- Add reference file schema option for federated formats (#2269) (8c488e6)
Bug Fixes
Dependencies
September 16, 2022
BigQuery Omni now supports the following quota and limit:
- The quota for total query result sizes for a project is now 1 TB per day. For more information, see Query jobs.
- The limit for maximum result size for a query has been increased from 2 MB to 10 GB (preview).
For more information, see Limitations.
September 15, 2022
The BigQuery Data Transfer Service for Google Ads now supports the new Google Ads API. This feature is now in preview.
BigQuery is now available in the Madrid (europe-southwest1), Milan (europe-west8), and Paris (europe-southwest1) regions. The Madrid and Paris regions have the lowest carbon impact.
BigQuery ML is now available in the Madrid (europe-southwest1), Milan (europe-west8), and [Paris (europe-southwest1)]https://cloud.google.com/bigquery/docs/locations#regional-locations) regions. The Madrid and Paris regions have the lowest carbon impact.
September 14, 2022
The Merge is coming! You may experience disruptions in the Ethereum public datasets in BigQuery.
The is_case_insensitive
schema option, which allows you to make a dataset and its table names case-insensitive, is now in preview.
JDBC driver update, release 1.3.0 1001
- You can now configure the connector to authenticate the connection using an external account (workforce or workload identity federation).
- You can now configure the connector to use Private Service Connect URLs.
- The connector now supports JDBC transaction APIs. BigQuery supports multi-statement transactions inside a single query, or across multiple queries, when using sessions.
- The connector is now verified to use a default project for datasets. To do this, set the
dataset_project_id
property inQueryProperties
of the connection string to the desired project. MATERIALIZED_VIEW
has been added to the list of table types when using thegetTableTypes
function.- The connector now supports the JSON data type.
ODBC driver update, release 2.5.0 1001
- You can now configure the connector to authenticate the connection using an external account (workforce or workload identity federation), with limited support, using Azure AD and Okta identity providers.
- You can now configure the connector to use Private Service Connect URLs.
- The connector now supports ODBC transaction APIs. BigQuery supports multi-statement transactions inside a single query, or across multiple queries, when using sessions.
- The connector is now verified to use a default project for datasets. To do this, set the
dataset_project_id
property inQueryProperties
of the connection string to the desired project. MATERIALIZED_VIEW
has been added to the list of table types. To retrieve these table types, configureSQLTables
toTABLE_TYPES_ONLY
.- The connector now supports the JSON data type.
September 13, 2022
In Cloud Monitoring, you can view metrics for quota usage and limits of the Storage Write API's concurrent connections and throughput quotas. This feature is now generally available (GA).
September 12, 2022
A weekly digest of client library updates from across the Cloud SDK.
Node.js
Changes for @google-cloud/bigquery
6.0.3 (2022-08-23)
Bug Fixes
Go
Changes for bigquery/storage/apiv1beta1
1.40.0 (2022-09-08)
⚠ BREAKING CHANGES
- bigquery/dataexchange: update BigQuery Analytics Hub API v1beta1 client BREAKING CHANGE: refresh current dataexchange/v1beta1/* directory to include recent change in protos. Removed common directory and use local enum Category fix!: refactor references to Category message docs: improve proto documentation.
Features
- bigquery/dataexchange: update BigQuery Analytics Hub API v1beta1 client BREAKING CHANGE: refresh current dataexchange/v1beta1/* directory to include recent change in protos. Removed common directory and use local enum Category fix!: refactor references to Category message docs: improve proto documentation. (e45ad9a)
- bigquery/storage/managedwriter: Augment reconnection logic (#6609) (6b0ac0c)
- bigquery: Add trace instrumentation support for individual rpcs (#6493) (eedc632)
- bigquery: Improve error when reading null values (#6566) (e9a94c2), refs #2612
Documentation
Miscellaneous Chores
September 09, 2022
The following features are now generally available for ARIMA_PLUS
models:
- The
HOLIDAY_REGION
option can now take more than one region string as input. If you include more than one region string, the union of the holidays in all of the provided regions will be taken into the modeling. - You can use the new
TREND_SMOOTHING_WINDOW_SIZE
option to smooth the trend component of the time series by applying a center moving average.
September 06, 2022
Cloud console updates: Improvements that are related to query execution include the following:
For long-running queries, the Execution details tab is automatically displayed with the timing details of each stage of the query.
In the query editor, you can now see the query validation message when your query is completed or canceled.
September 01, 2022
Customer-managed encryption keys are now integrated with CMEK organization policies. This feature is generally available (GA).
The slot recommender creates recommendations for customers using on-demand billing and is now generally available (GA).
The Random Forest model is now generally available (GA). For more information, see the random forest sections in the end-to-end user journey page.
August 29, 2022
A weekly digest of client library updates from across the Cloud SDK.
Go
Changes for bigquery/storage/apiv1beta1
1.39.0 (2022-08-23)
Features
- bigquery/storage: allow users to set Apache Avro output format options through avro_serialization_options param in TableReadOptions message Through AvroSerializationOptions, users can set enable_display_name_attribute, which populates displayName for every avro field with the original column name Improved documentation for selected_fields, added example for clarity. (41ab4ec)
- bigquery: add PreserveAsciiControlCharacters support for CSV (#6448) (b7bac2f)
- bigquery: add preview support for default values (#6464) (edc3be5)
Java
Changes for google-cloud-bigquery
2.14.7 (2022-08-23)
Bug Fixes
2.15.0 (2022-08-25)
Features
Community contributed UDFs are now generally available in the bigquery-utils
GitHub repository and the bigquery-public-data.persistent_udfs
public dataset.
Cloud console updates: In the query editor, when you select a function signature from the autocomplete list, you can remove the parameter names quickly by pressing the Backspace or Delete key.
August 22, 2022
A weekly digest of client library updates from across the Cloud SDK.
Go
Changes for bigquery/storage/apiv1beta1
1.38.0 (2022-08-12)
Features
- bigquery/migration: Add MySQL dialect to bigquerymigration v2 client library (370e23e)
- bigquery/storage/managedwriter: improve error communication (#6360) (b30d89d)
Bug Fixes
Python
Changes for google-cloud-bigquery
3.3.2 (2022-08-16)
Bug Fixes
August 19, 2022
The ALTER TABLE RENAME COLUMN
DDL statement, which allows you to rename the columns of a table, is now in preview.
August 17, 2022
You can now set default values on columns in your BigQuery tables. This feature is now in preview.
Cloud console updates: You can now copy BigQuery metadata to your clipboard by using the following options:
In the Schema view, to copy a table's schema, select any fields, and then click
Copy.In the Explorer pane, to copy the ID of a resource, click
View actions, and then click Copy ID.
Cloud console updates: Improvements include the following:
Query results are now displayed in resizable columns.
Tab titles now expand when space is available for longer names.
Tooltips no longer display text immediately when you hold the pointer over them, avoiding unnecessary distraction.
In the Explorer pane, you can now access saved queries by expanding your project. The Saved Queries pane is no longer at the bottom of the console.
In the Explorer pane, you can now find a table by searching for
mydataset.mytable
.In the query editor, you can now press the F1 shortcut key to view more editor shortcuts.
August 16, 2022
Workforce identity federation lets you authenticate and authorize users from external identity providers to access supported Google Cloud products, including BigQuery resources. This feature is now in preview.
August 15, 2022
A weekly digest of client library updates from across the Cloud SDK.
Python
Changes for google-cloud-bigquery
3.3.1 (2022-08-09)
Bug Fixes
An issue was identified in the max_staleness
materialized view option. This feature is not available.
BigQuery Omni now supports reservation DDL and access control DCL. This feature is now generally available (GA).
Previously, you could commit up to 100 GB in streaming bytes for every Storage Write API pending mode commit that you triggered in regions other than the US and EU multi-regions. This limit is now 1 TB. For more information, see Storage Write API quotas.
A weekly digest of client library updates from across the Cloud SDK.
August 10, 2022
You can now set default configurations at a project or organization level. This feature is now generally available (GA).
You can now manage query execution priority for Cloud Spanner federated queries. This feature is now generally available (GA).
August 09, 2022
August 08, 2022
A weekly digest of client library updates from across the Cloud SDK.
Go
Changes for bigquery/storage/apiv1beta1
1.37.0 (2022-08-04)
Features
- bigquery/connection: Add service_account_id output field to CloudSQL properties (1d6fbcc)
- bigquery/storage/managedwriter: refactor AppendResponse (#6402) (c07bca2)
- bigquery: support JSON as a data type (#5986) (835fe4f)
Bug Fixes
Documentation
- bigquery/storage: clarify size limitations for AppendRowsRequest chore: add preferred_min_stream_count to CreateReadSessionRequest chore: add write_stream to AppendRowsResponse (1d6fbcc)
A weekly digest of client library updates from across the Cloud SDK.
Java
Changes for google-cloud-bigquery
2.14.3 (2022-08-05)
Dependencies
- update dependency com.google.apis:google-api-services-bigquery to v2-rev20220730-2.0.0 (#2208) (5165e2b)
- update dependency com.google.cloud:google-cloud-storage to v2.11.2 (#2207) (da5389d)
2.14.2 (2022-08-04)
Dependencies
- update arrow.version to v9 (major) (#2201) (3ec5ef9)
- update dependency com.google.apis:google-api-services-bigquery to v2-rev20220716-2.0.0 (#2202) (c1ca09e)
- update dependency com.google.cloud:google-cloud-bigquerystorage-bom to 2.18.0 (c1ca09e)
- update dependency com.google.cloud:google-cloud-datacatalog-bom to 1.9.1 (c1ca09e)
- update dependency com.google.cloud:google-cloud-shared-dependencies to v3 (c1ca09e)
- update dependency com.google.cloud:google-cloud-storage to 2.11.0 (c1ca09e)
- update dependency com.google.cloud:google-cloud-storage to v2.11.1 (#2194) (45be001)
- update dependency com.google.code.gson:gson to v2.9.1 (#2190) (4bd4539)
- update dependency org.threeten:threeten-extra to v1.7.1 (c1ca09e)
Documentation
August 03, 2022
The max_staleness
materialized view option helps you achieve consistently high performance with controlled costs when processing large, frequently changing datasets. This feature is