BigQuery release notes

This page documents production updates to BigQuery. We recommend that BigQuery developers periodically check this list for any new announcements. BigQuery automatically updates to the latest release and cannot be downgraded to a previous version.

You can see the latest product updates for all of Google Cloud on the Google Cloud page, browse and filter all release notes in the Google Cloud console, or programmatically access release notes in BigQuery.

To get the latest product updates delivered to you, add the URL of this page to your feed reader, or add the feed URL directly.

April 24, 2025

You can now work with a Gemini powered assistant in a BigQuery data canvas. The data canvas assistant is an agent-like tool, capable of constructing and modifying a data canvas to answer data analytics questions from user prompting. This feature is now in Preview.

April 23, 2025

You can now set a maximum slot limit for a reservation. You can configure the maximum reservation size when creating or updating a reservation. This feature is in public preview.

You can now specify which reservation a query uses at runtime, and set IAM policies directly on reservations. This provides more flexibility and fine-grained control over resource management. This feature is in public preview.

You can now allocate idle slots fairly across reservations within a single admin project. This ensures each reservation receives an approximately equal share of available capacity. This feature is in public preview.

April 21, 2025

A weekly digest of client library updates from across the Cloud SDK.

Node.js

Changes for @google-cloud/bigquery

7.9.4 (2025-04-02)

Bug Fixes

• MergeSchemaWithRows can be called with empty schema if result set is empty (#1455) (e608601)

BigQuery now provides spend-based committed use discounts (CUDs). Spend-based committed use discounts provide a discount in exchange for your commitment to spend a minimum amount per hour on PAYG compute resources listed here. You can purchase CUDs with a one or three year commitment period.

You can now enable fine-grained access control on BigQuery metastore Iceberg tables. This feature is generally available (GA).

You can get the required permissions to use BigQuery data preparation through the BigQuery Studio User (roles/bigquery.studioUser) and Gemini for Google Cloud User (roles/cloudaicompanion.user) roles, and permission to access the data you're preparing.

BigQuery data preparation no longer requires that you have the permissions granted by the following IAM roles:

• BigQuery Data Editor (roles/bigquery.dataEditor)
• Service Usage Consumer (roles/serviceusage.serviceUsageConsumer)

For more information about the required roles, see Manage data preparations.

April 17, 2025

You can now use BigQuery DataFrames version 2.0, which makes security and performance improvements to the BigQuery DataFrames API, adds new features, and introduces breaking changes.

You can use partial ordering mode in BigQuery DataFrames to generate efficient queries. This feature is generally available (GA).

April 09, 2025

Updated pricing, packaging, and setup guidance is now available for Gemini in BigQuery.

You can now combine raster and vector data with the ST_REGIONSTATS geography function to perform geospatial analysis in BigQuery. For more information, see Work with raster data and try the tutorial that shows you how to use raster data to analyze global temperature by country. This feature is in preview.

You can now use the Apache Arrow format to stream data to BigQuery with the Storage Write API. This feature is available in preview.

Analytics Hub has been renamed BigQuery sharing. You'll see this new name in the documentation set and the marketing collateral. The product functionality and endpoints remain the same. For more information, see Introduction to data governance in BigQuery.

Dataplex Catalog has been renamed BigQuery universal catalog. You'll see this new name in the product page of the Google Cloud console, the documentation set, and the marketing collateral. Universal catalog brings together the data catalog capabilities of Dataplex Catalog and the runtime metastore capabilities of BigQuery metastore. For more information, see Introduction to data governance in BigQuery.

April 08, 2025

BigQuery ML now offers a built-in TimesFM univariate time series forecasting model that implements Google Research's open source TimesFM model. You can use BigQuery ML's built-in TimesFM model with the AI.FORECAST function to perform forecasting without having to create and train your own model. This lets you avoid the need for model management.

To try using a TimesFM model with the AI.FORECAST function, see Forecast a time series with a TimesFM univariate model.

This feature is in preview.

You can now create, view, modify, and delete Apache Iceberg resources in BigQuery metastore. This feature is generally available (GA).

You can now connect BigQuery metastore to Apache Flink. This feature is generally available (GA).

April 07, 2025

BigQuery data preparation is generally available (GA). It offers AI-powered suggestions from Gemini for data cleansing, transformation, and enrichment. BigQuery supports visual data preparation pipelines and pipeline scheduling with Dataform.

You can now create remote models in BigQuery ML based on Llama and Mistral AI models in Vertex AI.

Use the ML.GENERATE_TEXT function with these remote models to perform generative natural language tasks for text stored in BigQuery tables. Try this feature with the Generate text by using the ML.GENERATE_TEXT function tutorial.

This feature is generally available (GA).

An updated version of JDBC driver for BigQuery is now available.

Smart-tuning is now supported for materialized views when they are in the same project as one of their base tables, or when they are in the project running the query. This feature is generally available (GA).

BigQuery ML now uses dynamic token-based batching for embedding generation requests. Dynamic token-based batching puts as many rows as possible into one request. This change boosts per-request utilization and improves scalability for any queries per minute (QPM) quota. Actual performance varies based on the embedding content length, with an average 10x improvement.

April 04, 2025

BigQuery ML now supports the following generative AI functions, which let you analyze text using a Vertex AI Gemini model. The function output includes a response that matches the type in the function name:

• AI.GENERATE
• AI.GENERATE_BOOL
• AI.GENERATE_INT
• AI.GENERATE_DOUBLE

This feature is in preview.

April 03, 2025

BigQuery migration assessment now includes support for Amazon Redshift Serverless. This feature is in preview.

You can now generate structured data by using BigQuery ML's AI.GENERATE_TABLE function with Gemini 1.5 Pro, Gemini 1.5 Flash, and Gemini 2.0 Flash models. You can use the AI.GENERATE_TABLE function's output_schema argument to more easily format the model's response. The output_schema argument lets you specify a SQL schema for formatting, similar to the schema used in the CREATE TABLE statement. By creating structured output, you can more easily convert the function output into a BigQuery table.

Try this feature with the Generate structured data by using the AI.GENERATE_TABLE function tutorial.

This feature is in preview.

April 02, 2025

You can now create and use Python user-defined functions (UDFs) in BigQuery. Python UDFs support the use of additional libraries and external APIs. This feature is in preview.

The Python code that you generate using Gemini in BigQuery Notebooks is now much more likely to leverage your data. With this change, BigQuery Notebooks can intelligently pull relevant table names directly from your BigQuery project, resulting in personalized, executable Python code.

You can now generate Dataframes code in BigQuery Notebooks that use BigFrames libraries. In your code generation prompt, include the word BigFrames to generate code that uses BigQuery DataFrames. This feature is in preview.

April 01, 2025

Pipe syntax supports a linear query structure designed to make your queries easier to read, write, and maintain. This feature is generally available (GA).

You can use a CREATE MODEL statement to create a contribution analysis model in BigQuery ML. The top_k_insights_by_apriori_support and pruning_method model options are now supported. You can use a contribution analysis model with the ML.GET_INSIGHTS function to generate insights about changes to key metrics in your multi-dimensional data. The following metric types are supported:

• Summable metric
• Summable ratio metric
• Summable by category metric

This feature is generally available (GA).

March 31, 2025

Iceberg external tables now support merge-on-read. You can query Iceberg tables with position deletes and equality deletes. This feature is generally available (GA).

A weekly digest of client library updates from across the Cloud SDK.

Python

Changes for google-cloud-bigquery

3.31.0 (2025-03-20)

Features

• Add query text and total bytes processed to RowIterator (#2140) (2d5f932)
• Add support for Python 3.13 (0842aa1)

Bug Fixes

• Add property setter for table constraints, #1990 (#2092) (f8572dd)
• Allow protobuf 6.x (0842aa1)
• Avoid "Unable to determine type" warning with JSON columns in to_dataframe (#1876) (968020d)
• Remove setup.cfg configuration for creating universal wheels (#2146) (d7f7685)

Dependencies

• Remove Python 3.7 and 3.8 as supported runtimes (#2133) (fb7de39)

On the Scheduling page, you can now view existing schedules, create new schedules, and perform other actions for data preparations, notebooks, BigQuery pipelines, and scheduled queries. For more information, see Create a pipeline schedule. This feature is generally available (GA).

You can build BigQuery pipelines (formerly workflows), composed of SQL queries or notebooks, in BigQuery Studio. You can then run these pipelines on a schedule. You can also configure notebook runtimes for a pipeline, share a pipeline, or share a pipeline link. This feature is generally available (GA).

You can now define a _CHANGE_SEQUENCE_NUMBER for BigQuery change data capture (CDC) to manage streaming UPSERT ordering for BigQuery. This feature is generally available (GA).

BigQuery now supports subqueries in row level access policies. It also includes support for BigLake managed tables and the BigQuery Storage Read API. This feature is now generally available (GA).

You can now use BigQuery Data Transfer Service for Search Ads to view Performance Max (PMax) campaign data for the following tables:

• CartDataSalesStats
• ProductAdvertised
• ProductAdvertisedDeviceStats
• ProductAdvertisedConversionActionAndDeviceStats

This feature is generally available (GA).

You can now configure the repeat frequency of BigQuery Data Transfer Service for Google Ad Manager. This option has a default of every 8 hours and a minimum of every 4 hours. This feature is generally available (GA).

You can now skip loading match tables for BigQuery Data Transfer Service for Google Ad Manager. If match tables are not needed, you can set parameter load_match_tables to FALSE. This feature is generally available (GA).

You can include data preparation tasks in BigQuery pipelines that execute your code assets in sequence at a scheduled time. This feature is in Preview.

March 27, 2025

You can now enable metadata caching for SQL translation, which can significantly reduce latency for subsequent translation requests. This feature is in preview.

March 26, 2025

You can now set the column granularity when you create a search index, which stores additional column information in your search index to further optimize your search query performance. This feature is in preview.

March 25, 2025

BigQuery ML now supports visualization of model monitoring metrics. This feature lets you use charts and graphs to analyze model monitoring function output. The following functions support metric visualization:

• ML.VALIDATE_DATA_SKEW: compute the statistics for a set of serving data, and then compare them to the statistics for the data used to train a BigQuery ML model in order to identify anomalous differences between the two data sets.
• ML.VALIDATE_DATA_DRIFT: compute and compare the statistics for two sets of serving data in order to identify anomalous differences between the two data sets.

This feature is in preview.

March 24, 2025

A weekly digest of client library updates from across the Cloud SDK.

Node.js

Changes for @google-cloud/bigquery

7.9.3 (2025-03-17)

Bug Fixes

• Make sure to pass selectedFields to tabledata.list method (#1449) (206aff9)

Go

Changes for bigquery/storage/apiv1beta1

1.67.0 (2025-03-14)

Features

• bigquery/reservation: Add a new field enable_gemini_in_bigquery to .google.cloud.bigquery.reservation.v1.Assignment that indicates if "Gemini in BigQuery" (601e742)
• bigquery/reservation: Add a new field replication_status to .google.cloud.bigquery.reservation.v1.Reservation to provide visibility into errors that could arise during Disaster Recovery(DR) replication (#11666) (601e742)
• bigquery/reservation: Add the CONTINUOUS Job type to .google.cloud.bigquery.reservation.v1.Assignment.JobType for continuous SQL jobs (601e742)
• bigquery: Support MetadataCacheMode for ExternalDataConfig (#11803) (af5174d), refs #11802

Bug Fixes

• bigquery: Increase timeout for storage api test and remove usage of deprecated pkg (#11810) (f47e038), refs #11801
• bigquery: Update golang.org/x/net to 0.37.0 (1144978)

Documentation

• bigquery/reservation: Remove the section about EDITION_UNSPECIFIED in the comment for slot_capacity in .google.cloud.bigquery.reservation.v1.Reservation to clarify that (601e742)
• bigquery/reservation: Update the google.api.field_behavior for the .google.cloud.bigquery.reservation.v1.Reservation.primary_location and .google.cloud.bigquery.reservation.v1.Reservation.original_primary_location fields to clarify that they are OUTPUT_ONLY (601e742)

Java

Changes for google-cloud-bigquery

2.49.0 (2025-03-20)

Features

• bigquery: Implement getArray in BigQueryResultImpl (#3693) (e2a3f2c)
• Next release from main branch is 2.49.0 (#3706) (b46a6cc)

Bug Fixes

• Retry ExceptionHandler not retrying on IOException (#3668) (83245b9)

Dependencies

• Exclude io.netty:netty-common from org.apache.arrow:arrow-memor… (#3715) (11b5809)
• Update actions/upload-artifact action to v4.6.2 (#3724) (426a59b)
• Update actions/upload-artifact action to v4.6.2 (#3724) (483f930)
• Update dependency com.google.api.grpc:proto-google-cloud-bigqueryconnection-v1 to v2.61.0 (#3703) (53b07b0)
• Update dependency com.google.api.grpc:proto-google-cloud-bigqueryconnection-v1 to v2.62.0 (#3726) (38e004b)
• Update dependency com.google.apis:google-api-services-bigquery to v2-rev20250302-2.0.0 (#3720) (c0b3902)
• Update dependency com.google.apis:google-api-services-bigquery to v2-rev20250313-2.0.0 (#3723) (b8875a8)
• Update dependency com.google.cloud:google-cloud-datacatalog-bom to v1.65.0 (#3704) (53b68b1)
• Update dependency com.google.cloud:google-cloud-datacatalog-bom to v1.66.0 (#3727) (7339f94)
• Update dependency com.google.cloud:sdk-platform-java-config to v3.45.1 (#3714) (e4512aa)
• Update dependency com.google.oauth-client:google-oauth-client-java6 to v1.39.0 (#3710) (c0c6352)
• Update dependency com.google.oauth-client:google-oauth-client-jetty to v1.39.0 (#3711) (43b86e9)
• Update dependency node to v22 (#3713) (251def5)
• Update netty.version to v4.1.119.final (#3717) (08a290a)

Documentation

• Update error handling comment to be more precise in samples (#3712) (9eb555f)

You can now use KLL quantile functions to efficiently compute approximate quantiles. This feature is in preview.

You can now set labels on reservations. These labels can be used to organize your reservations and for billing analysis. This feature is in preview.

The BigQuery Data Transfer Service can now transfer reporting and configuration data from Google Analytics 4 into BigQuery. This feature is in preview.

We have redesigned the Add Data dialog to guide you through loading data into BigQuery with a source-first experience and enhanced search and filtering capabilities. This feature is generally available (GA).

March 20, 2025

BigQuery workflows have been renamed to BigQuery pipelines in the Google Cloud console. For more information, see Introduction to BigQuery pipelines.

You can now use repositories and workspaces in BigQuery to perform version control.

Repositories perform version control on files by using Git to record changes and manage file versions. You can use workspaces within repositories to edit the code stored in the repository.

You can have a repository use Git directly on BigQuery, or you can connect a repository to a third-party Git provider.

This feature is in preview.

You can now create remote models in BigQuery ML based on the Anthropic Claude model in Vertex AI.

Use the ML.GENERATE_TEXT function with these remote models to perform generative natural language tasks for text stored in BigQuery tables. Try this feature with the Generate text by using the ML.GENERATE_TEXT function tutorial.

You can also evaluate Claude models by using the ML.EVALUATE function.

This feature is generally available (GA).

March 17, 2025

You can now use EXPORT DATA statements to reverse ETL BigQuery data to Spanner. This feature is generally available (GA).

You can now create an external dataset in BigQuery that links to an existing database in Spanner. This feature is generally available (GA).

You can now use the TYPEOF function to determine the data type of an expression. This feature is generally available (GA).

March 13, 2025

Dataform now supports the CMEK organization policy.

You can now use Gemini Cloud Assist chat to generate SQL queries and Python code. This feature is in preview.

March 12, 2025

You can configure reusable, default Cloud resource connections in a project. Default connections are available in Preview.

An updated version of ODBC driver for BigQuery is now available.

March 10, 2025

Analytics Hub egress controls and data clean room subscriptions are now available in all BigQuery editions and on-demand pricing.

March 06, 2025

BigQuery Data Transfer Service now supports custom reports for Google Ads. You can use Google Ads Query Language (GAQL) queries in your transfer configuration to ingest custom Google Ads reports and fields beyond those available in the standard reports and fields. This feature is now generally available (GA).

March 04, 2025

BigQuery is now available in the Stockholm (europe-north2) region.

March 03, 2025

A weekly digest of client library updates from across the Cloud SDK.

Java

Changes for google-cloud-bigquery

2.48.1 (2025-02-26)

Dependencies

• Update actions/upload-artifact action to v4.6.1 (#3691) (9c0edea)
• Update dependency com.google.api.grpc:proto-google-cloud-bigqueryconnection-v1 to v2.60.0 (#3680) (6d9a40d)
• Update dependency com.google.apis:google-api-services-bigquery to v2-rev20250216-2.0.0 (#3688) (e3beb6f)
• Update dependency com.google.cloud:google-cloud-datacatalog-bom to v1.64.0 (#3681) (9e4e261)
• Update dependency com.google.cloud:sdk-platform-java-config to v3.44.0 (#3694) (f69fbd3)
• Update dependency com.google.oauth-client:google-oauth-client-java6 to v1.38.0 (#3685) (53bd7af)
• Update dependency com.google.oauth-client:google-oauth-client-jetty to v1.38.0 (#3686) (d71b2a3)
• Update ossf/scorecard-action action to v2.4.1 (#3690) (cdb61fe)

Python

Changes for google-cloud-bigquery

3.30.0 (2025-02-26)

Features

• Add roundingmode enum, wiring, and tests (#2121) (3a48948)
• Adds foreign_type_info attribute to table class and adds unit tests. (#2126) (2c19681)
• Support resource_tags for table (#2093) (d4070ca)

Bug Fixes

• Avoid blocking in download thread when using BQ Storage API (#2034) (54c8d07)
• Retry 404 errors in Client.query(...) (#2135) (c6d5f8a)

Dependencies

• Updates required checks list in github (#2136) (fea49ff)
• Use pandas-gbq to determine schema in load_table_from_dataframe (#2095) (7603bd7)

Documentation

• Update magics.rst (#2125) (b5bcfb3)

You can create a SQL user-defined aggregate function by using the CREATE AGGREGATE FUNCTION statement. This feature is generally available (GA).

Gemini in BigQuery can help you complete Python code with contextually appropriate recommendations that are based on content in the query editor. This feature is now generally available (GA).

February 25, 2025

You can now see a list of BigQuery API and service dependencies. You can also review the effects of disabling an API or service.

You can use the best sellers and price competitiveness migration guides to transition to the newer version of the reports. This feature is in preview.

BigQuery resource utilization charts provide metrics views and more chart configuration options in Preview.

February 24, 2025

You can now use the @@location system variable to set the location in which to run a query. This feature is in preview.

February 17, 2025

A weekly digest of client library updates from across the Cloud SDK.

Node.js

Changes for @google-cloud/bigquery

7.9.2 (2025-02-12)

Bug Fixes

• Avoid schema field mutation when passing selectedFields opt (#1437) (27044d5)

Java

Changes for google-cloud-bigquery

2.48.0 (2025-02-13)

Features

• Implement wasNull for BigQueryResultSet (#3650) (c7ef94b)

Dependencies

• Update dependency com.google.api.grpc:proto-google-cloud-bigqueryconnection-v1 to v2.59.0 (#3660) (3a6228b)
• Update dependency com.google.apis:google-api-services-bigquery to v2-rev20250128-2.0.0 (#3667) (0b92af6)
• Update dependency com.google.cloud:google-cloud-datacatalog-bom to v1.63.0 (#3661) (9bc8c01)
• Update dependency com.google.cloud:sdk-platform-java-config to v3.43.0 (#3669) (4d9e0ff)

Documentation

• Update CONTRIBUTING.md for users without branch permissions (#3670) (009b9a2)

Subscriber email logging lets you log the principal identifiers of users who execute jobs and queries against linked datasets. You can enable logging at the listing level and the data exchange level (for all the listings in the data exchange). Once you enable and save subscriber email logging, this setting cannot be edited. This feature is in preview.

February 10, 2025

A weekly digest of client library updates from across the Cloud SDK.

Go

Changes for bigquery/storage/apiv1beta1

1.66.2 (2025-02-04)

Bug Fixes

• bigquery: Broken github.com/envoyproxy/go-control-plane/envoy dep (#11556) (e70d63b), refs #11542

1.66.1 (2025-02-03)

Bug Fixes

• bigquery: Move MaxStaleness field to table level (#10066) (164492d)

BigQuery data preparation provides context-aware join operation recommendations from Gemini. Data preparation is available in Preview.

February 06, 2025

You can create a JavaScript user-defined aggregate function by using the CREATE AGGREGATE FUNCTION statement. This feature is generally available (GA).

February 03, 2025

A weekly digest of client library updates from across the Cloud SDK.

Java

Changes for google-cloud-bigquery

2.47.0 (2025-01-29)

Features

• bigquery: Support resource tags for datasets in java client (#3647) (01e0b74)

Bug Fixes

• bigquery: Remove ReadAPI bypass in executeSelect() (#3624) (fadd992)
• Close bq read client (#3644) (8833c97)

Dependencies

• Update dependency com.google.apis:google-api-services-bigquery to v2-rev20250112-2.0.0 (#3651) (fd06100)
• Update dependency com.google.cloud:sdk-platform-java-config to v3.42.0 (#3653) (1a14342)
• Update github/codeql-action action to v2.28.1 (#3637) (858e517)

The BigQuery ML ML.BUCKETIZE and ML.QUANTILE_BUCKETIZE functions now support formatting of the function output. You can use the output_format argument to format the function output as one of the following:

• A string in the format bin_<bucket_index>
• A string in interval notation
• A JSON-formatted string

You can now use the BY NAME and CORRESPONDING modifiers with set operations to match columns by name instead of by position. This feature is generally available (GA).

January 28, 2025

You can now view stored column usage information for a query job that performs vector search using stored columns. This feature is generally available (GA).

January 27, 2025

A weekly digest of client library updates from across the Cloud SDK.

Go

Changes for bigquery/storage/apiv1beta1

1.66.0 (2025-01-20)

Features

• bigquery/storage/managedwriter: Graceful connection drains (#11463) (b29912f)

Bug Fixes

• bigquery: Update golang.org/x/net to v0.33.0 (e9b0b69)

Python

Changes for google-cloud-bigquery

3.29.0 (2025-01-21)

Features

• Add ExternalCatalogTableOptions class and tests (#2116) (cdc1a6e)

Bug Fixes

• Add default value in SchemaField.from_api_repr() (#2115) (7de6822)

The following BigQuery ML generative AI features are now available:

• Creating a remote model based on an open model from Vertex Model Garden or Hugging Face that is deployed to Vertex AI. Options include Llama, Gemma, and other leading open text generation models.
• Using the ML.GENERATE_TEXT function with this remote model to perform a broad range of generative AI tasks.
• Using the ML.EVALUATE function to evaluate the remote model.

Try these features with the Generate text by using the ML.GENERATE_TEXT function how-to topic and the
Generate text by using a Gemma open model and the ML.GENERATE_TEXT function tutorial.

These features are generally available (GA).

We previously communicated that after January 27, 2025, a purchase would be required to use Gemini in BigQuery features. We are temporarily delaying enforcement of these procurement methods, and no purchase is required at this time. For more information, see Gemini for Google Cloud pricing.

You can now set conditional IAM access on BigQuery datasets with access control lists (ACLs). This feature is generally available (GA).

January 22, 2025

BigQuery metastore lets you access and manage metadata from a variety of processing engines, including BigQuery and Apache Spark. BigQuery metastore supports BigQuery tables and open formats such as Apache Iceberg. This feature is in preview.

January 21, 2025

In BigQuery ML, you can now evaluate Anthropic Claude models by using the ML.EVALUATE function. The quotas for use of Anthropic Claude models in BigQuery ML have also been brought into parity with Vertex AI quotas.

This feature is in preview.

You can use natural language to prepare data with Gemini in BigQuery.

Data preparation in BigQuery lets you test data preparations you're developing before you deploy and schedule runs in production. For more information, see Develop a data preparation.

January 20, 2025

A weekly digest of client library updates from across the Cloud SDK.

Java

Changes for google-cloud-bigquery

2.46.0 (2025-01-11)

Features

• bigquery: Support IAM conditions in datasets in Java client. (#3602) (6696a9c)

Bug Fixes

• NPE when reading BigQueryResultSet from empty tables (#3627) (9a0b05a)
• test: Force usage of ReadAPI (#3625) (5ca7d4a)

Dependencies

• Update actions/upload-artifact action to v4.5.0 (#3620) (cc25099)
• Update actions/upload-artifact action to v4.6.0 (#3633) (ca20aa4)
• Update dependency com.google.api.grpc:proto-google-cloud-bigqueryconnection-v1 to v2.57.0 (#3617) (51370a9)
• Update dependency com.google.api.grpc:proto-google-cloud-bigqueryconnection-v1 to v2.58.0 (#3631) (b0ea0d5)
• Update dependency com.google.apis:google-api-services-bigquery to v2-rev20241222-2.0.0 (#3623) (4061922)
• Update dependency com.google.cloud:google-cloud-datacatalog-bom to v1.61.0 (#3618) (6cba626)
• Update dependency com.google.cloud:google-cloud-datacatalog-bom to v1.62.0 (#3632) (e9ff265)
• Update dependency com.google.cloud:sdk-platform-java-config to v3.41.1 (#3628) (442d217)
• Update dependency com.google.oauth-client:google-oauth-client-java6 to v1.37.0 (#3614) (f5faa69)
• Update dependency com.google.oauth-client:google-oauth-client-jetty to v1.37.0 (#3615) (a6c7944)
• Update github/codeql-action action to v2.27.9 (#3608) (567ce01)
• Update github/codeql-action action to v2.28.0 (#3621) (e0e09ec)

Python

Changes for google-cloud-bigquery

3.28.0 (2025-01-15) - YANKED

Reason this release was yanked:

This turned out to be incompatible with pandas-gbq. For more details, see issue.

Features

• Add property for allowNonIncrementalDefinition for materialized view (#2084) (3359ef3)
• Add property for maxStaleness in table definitions (#2087) (729322c)
• Add type hints to Client (#2044) (40529de)
• Adds ExternalCatalogDatasetOptions and tests (#2111) (b929a90)
• Adds ForeignTypeInfo class and tests (#2110) (55ca63c)
• Adds new input validation function similar to isinstance. (#2107) (a2bebb9)
• Adds StorageDescriptor and tests (#2109) (6be0272)
• Adds the SerDeInfo class and tests (#2108) (62960f2)
• Migrate to pyproject.toml (#2041) (1061611)
• Preserve unknown fields from the REST API representation in SchemaField (#2097) (aaf1eb8)
• Resource tags in dataset (#2090) (3e13016)
• Support setting max_stream_count when fetching query result (#2051) (d461297)

Bug Fixes

• Allow geopandas 1.x (#2065) (f2ab8cb)

Documentation

• Render fields correctly for update calls (#2055) (a4d9534)

January 17, 2025

The BigQuery Data Transfer Service can now transfer data from the following data sources:

• MySQL
• PostgreSQL

Transfers from these data sources are supported in Preview.

In the navigation menu, you can now go to the Settings page to set default settings that are applied when you start a session in BigQuery Studio. This feature is in preview.

January 16, 2025

The BigQuery migration assessment for Oracle now includes a total cost of ownership (TCO) calculator that provides an estimation of compute and storage costs for migrating your Oracle data warehouse to BigQuery. This feature is in preview.

We have rearranged the navigation menu into new categories. This feature is generally available (GA).

January 13, 2025

In BigQuery ML, you can now forecast multiple time series at once by using the new TIME_SERIES_ID_COL option that is available in ARIMA_PLUS_XREG multivariate time series models. Try this feature with the Forecast multiple time series with a multivariate model tutorial.

This feature is in preview.

You can now use BigQuery Omni Virtual Private Cloud (VPC) allowlists to restrict access to AWS S3 buckets and Azure Blob Storage from specific BigQuery Omni VPCs. This feature is generally available (GA).

January 02, 2025

An updated version of JDBC driver for BigQuery is now available.

December 23, 2024

BigQuery is available in the Mexico (northamerica-south1) region.

December 19, 2024

The Sovereign Controls for EU control package now supports BigQuery Data Transfer Service. For more information, see Supported products by control package. This feature is generally available (GA).

You can now manage data canvases, data preparations, notebooks, saved queries, and workflows in Dataplex. Metadata of data canvases, data preparations, notebooks, saved queries, and workflows is automatically available in Dataplex, without additional configuration. This feature is generally available (GA).

You can now search for and view the metadata of data canvases, data preparations, notebooks, saved queries, and workflows in the Dataplex console. This feature is in preview.

December 16, 2024

A weekly digest of client library updates from across the Cloud SDK.

Java

Changes for google-cloud-bigquery

2.45.0 (2024-12-13)

Features

• Enable Lossless Timestamps in BQ java client lib (#3589) (c0b874a)
• Introduce java.time methods and variables (#3586) (31fb15f)

Bug Fixes

• test: Update schema for broken ConnImplBenchmark test (#3574) (8cf4387)

Dependencies

• Update dependency com.google.api.grpc:proto-google-cloud-bigqueryconnection-v1 to v2.56.0 (#3582) (616ee2a)
• Update dependency com.google.apis:google-api-services-bigquery to v2-rev20241111-2.0.0 (#3591) (3eef3a9)
• Update dependency com.google.apis:google-api-services-bigquery to v2-rev20241115-2.0.0 (#3601) (41f9adb)
• Update dependency com.google.cloud:google-cloud-datacatalog-bom to v1.60.0 (#3583) (34dd8bc)
• Update dependency com.google.cloud:sdk-platform-java-config to v3.41.0 (#3607) (11499d1)
• Update github/codeql-action action to v2.27.5 (#3588) (3f94075)
• Update github/codeql-action action to v2.27.6 (#3597) (bc1f3b9)
• Update github/codeql-action action to v2.27.7 (#3603) (528426b)

Documentation

• bigquery: Add javadoc description of timestamp() parameter. (#3604) (6ee0c10)

You can now use the Google Cloud Code extension for VS Code to work with BigQuery datasets and notebooks in your VS Code environment. This feature is in preview.

December 12, 2024

Regional endpoints, which help you run your workloads in compliance with data residency and data sovereignty requirements, are now generally available (GA). With regional endpoints, your request traffic is routed directly to the region specified in the endpoint. For more information, see BigQuery regional endpoints.

You can now discover, procure, and commercialize your Analytics Hub listings on Google Cloud Marketplace to share data offerings at scale. This feature is in preview.

December 11, 2024

You can now replicate a dataset from the source region to one or more other regions with cross-region dataset replication. This feature is now generally available (GA).

BigQuery Managed Disaster Recovery provides managed failover and redundant compute capacity for business-critical workloads. It is intended for use in the case of a total region outage and is supported with the BigQuery Enterprise Plus edition only. This feature is now generally available (GA).

You can now create remote models in BigQuery ML based on the gemini-2.0-flash-exp model in Vertex AI. To create remote models, you can use either SQL or BigQuery DataFrames.

You can use the ML.GENERATE_TEXT function with these remote models to perform generative natural language tasks for text stored in BigQuery tables. You can also use the ML.GENERATE_TEXT function with these remote models to perform generative AI tasks, for example audio transcription or document classification, using image, video, audio, PDF, or text content stored in BigQuery object tables.

Try this feature by using either the Generate text by using the ML.GENERATE_TEXT function how-to topic, or the BigFrames Gemini 2.0 Text Generation Simple Example notebook.

This feature is in preview.

December 09, 2024

A weekly digest of client library updates from across the Cloud SDK.

Go

Changes for bigquery/storage/apiv1beta1

1.65.0 (2024-12-05)

Features

• bigquery/reservation: Add a new field is_flat_rate to .google.cloud.bigquery.reservation.v1.CapacityCommitment to distinguish between flat rate and edition commitments (8dedb87)
• bigquery/reservation: Add the managed disaster recovery API(https (8dedb87)
• bigquery: Expose IsCaseInsensitive for dataset metadata (#11216) (364b639)
• bigquery: Support IAM conditions in datasets (#11123) (d93c2d9)

Bug Fixes

• bigquery: Do not override ctx within inner span creation (#11119) (627c69a), refs #10961

Documentation

• bigquery/reservation: Clarify that Autoscale.current_slots in message .google.cloud.bigquery.reservation.v1.Reservation can temporarily be larger than Autoscale.max_slots if users reduce Autoscale.max_slots (8dedb87)
• bigquery/reservation: Update comment for slot_capacity in message .google.cloud.bigquery.reservation.v1.Reservation to provide more clarity about reservation baselines, committed slots and autoscaler SKU charges when the baseline exceeds committed slots (8dedb87)
• bigquery/reservation: Update comments for commitment_start_time and commitment_end_time in message .google.cloud.bigquery.reservation.v1.CapacityCommitment to provide details on how these values are affected by commitment renewal (8dedb87)

November 25, 2024

A weekly digest of client library updates from across the Cloud SDK.

Java

Changes for google-cloud-bigquery

2.44.0 (2024-11-17)

Features

• Enable maxTimeTravelHours in BigQuery java client library (#3555) (bd24fd8)

Bug Fixes

• Update experimental methods documentation to @internalapi (#3552) (20826f1)

Dependencies

• Update dependency com.google.api.grpc:proto-google-cloud-bigqueryconnection-v1 to v2.55.0 (#3559) (950ad0c)
• Update dependency com.google.apis:google-api-services-bigquery to v2-rev20241027-2.0.0 (#3568) (b5ccfcc)
• Update dependency com.google.cloud:google-cloud-datacatalog-bom to v1.59.0 (#3561) (1bd24a1)
• Update dependency com.google.cloud:sdk-platform-java-config to v3.40.0 (#3576) (d5fa951)
• Update github/codeql-action action to v2.27.1 (#3567) (e154ee3)
• Update github/codeql-action action to v2.27.3 (#3569) (3707a40)
• Update github/codeql-action action to v2.27.4 (#3572) (2c7b4f7)

Documentation

• Fix BigQuery documentation formating (#3565) (552f491)

November 19, 2024

You can create a search index on columns containing INT64 or TIMESTAMP data and BigQuery can optimize predicates that use those columns. This feature is generally available (GA).

November 14, 2024

The following BigQuery ML features are now available:

• Creating remote models based on the Vertex AI gemini-1.5-flash and gemini-1.5-pro models.
• Using the ML.GENERATE_TEXT function with these remote models to perform generative natural language tasks for text stored in BigQuery tables.
• Using the ML.GENERATE_TEXT function with these remote models to perform generative AI tasks, for example audio transcription or document classification, using image, video, audio, PDF, or text content stored in BigQuery object tables.

Try these features with the Generate text by using the ML.GENERATE_TEXT function how-to topic.

These features are now generally available (GA).

You can try Gemini in BigQuery at no charge until January 27, 2025. After that date, to continue to use Gemini in BigQuery you must do one of the following:

• Purchase and assign BigQuery Enterprise Plus edition reservations to projects that use Gemini in BigQuery.
• Purchase Gemini Code Assist Enterprise.

To learn more, see Purchase Gemini in BigQuery. These purchase options are now generally available (GA).

November 11, 2024

A weekly digest of client library updates from across the Cloud SDK.

Python

Changes for google-cloud-bigquery

3.27.0 (2024-11-01)

Features

• Updates to allow users to set max_stream_count (#2039) (7372ad6)

The following BigQuery ML features are now available:

• You can perform supervised tuning on a remote model based on a Vertex AI Gemini 1.5 flash or Gemini 1.5 pro model.
• You can evaluate a Vertex AI LLM using the ML.EVALUATE function. Pre-trained PaLM and Gemini models and tuned Gemini models are supported for evaluation.

Try tuning and evaluating an LLM with the Customize an LLM by using supervised fine tuning how-to topic or the Use tuning and evaluation to improve model performance tutorial.

These BigQuery ML features are generally available (GA).

November 06, 2024

BigQuery now offers the following Gemini-enhanced SQL translation features:

• In interactive translation mode, you can use Gemini-enhanced SQL translations to customize translated GoogleSQL queries. This feature is generally available (GA).

• You can generate AI suggestions for batch translations using the Gemini model. The suggestions are based on a Gemini-based configuration YAML file. This feature is in Preview.

• After running an interactive SQL translation, you can request a Gemini-generated text explanation that includes a summary of the translated SQL query. This feature is in Preview.

November 05, 2024

Dataplex automatic discovery lets you scan data in Cloud Storage buckets to extract and catalog metadata. Automatic discovery creates BigLake or external tables and object tables you can use for analytics and AI, and catalogs that data in Dataplex Catalog. This feature is available in public preview.

The BigQuery Data Transfer Service data source change log provides details about upcoming changes to data source schemas and schema mappings.

November 04, 2024

A weekly digest of client library updates from across the Cloud SDK.

Go

Changes for bigquery/storage/apiv1beta1

1.64.0 (2024-10-30)

Features

• bigquery/datatransfer: Add scheduleOptionsV2 and Error fields for TransferConfig (78d8513)
• bigquery/storage: Add experimental ArrowData type and arrow_data field within AppendRowsRequest (f0b05e2)

Bug Fixes

• bigquery: Handle null RANGE (#11058) (9979e72), refs #11047
• bigquery: Parse negative NUMERIC from arrow (#11052) (83352c4)
• bigquery: Update google.golang.org/api to v0.203.0 (8bb87d5)
• bigquery: WARNING: On approximately Dec 1, 2024, an update to Protobuf will change service registration function signatures to use an interface instead of a concrete type in generated .pb.go files. This change is expected to affect very few if any users of this client library. For more information, see https://togithub.com/googleapis/google-cloud-go/issues/11020. (8bb87d5)

Documentation

• bigquery: Link types on package docs (#11036) (c4af6fe)

Java

Changes for google-cloud-bigquery

2.43.3 (2024-10-29)

Dependencies

• Update dependency com.google.cloud:google-cloud-bigquerystorage-bom to v3.10.2 (19fc184)

2.43.2 (2024-10-27)

Dependencies

• Update actions/checkout action to v4.2.2 (#3541) (c36c123)
• Update actions/upload-artifact action to v4.4.2 (#3524) (776a554)
• Update actions/upload-artifact action to v4.4.3 (#3530) (2f87fd9)
• Update dependency com.google.api.grpc:proto-google-cloud-bigqueryconnection-v1 to v2.54.0 (#3532) (25be311)
• Update dependency com.google.apis:google-api-services-bigquery to v2-rev20241013-2.0.0 (#3544) (0c42092)
• Update dependency com.google.cloud:google-cloud-bigquerystorage-bom to v3.10.0 (0bd3c86)
• Update dependency com.google.cloud:google-cloud-bigquerystorage-bom to v3.10.1 (c03a63a)
• Update dependency com.google.cloud:google-cloud-datacatalog-bom to v1.58.0 (#3533) (cad2643)
• Update dependency com.google.cloud:sdk-platform-java-config to v3.38.0 (#3542) (16448ee)
• Update dependency com.google.cloud:sdk-platform-java-config to v3.39.0 (#3548) (616b2f6)
• Update github/codeql-action action to v2.26.13 (#3536) (844744f)
• Update github/codeql-action action to v2.27.0 (#3540) (1616a0f)

Documentation

• Reformat javadoc (#3545) (4763f73)
• Update SimpleApp to explicitly set project id (#3534) (903a0f7)

October 24, 2024

BigQuery provides context-aware transformation recommendations from Gemini for cleansing data for analysis. Data preparation is available in Preview.

October 21, 2024

You can now view, trigger, and pause Airflow DAGs in BigQuery. This feature is in Preview.

You can now manage notebook schedules on the Orchestration page. Notebook scheduling is in Preview.

Custom organization policies let you allow or deny specific operations on BigQuery Data Transfer Service transfer configurations to meet your organization's compliance and security requirements. This feature is generally available (GA).

October 14, 2024

A weekly digest of client library updates from across the Cloud SDK.

Java

Changes for google-cloud-bigquery

2.43.1 (2024-10-09)

Dependencies

• Update actions/checkout action to v4.2.1 (#3520) (ad8175a)
• Update actions/upload-artifact action to v4.4.1 (#3521) (dc21975)
• Update dependency com.google.apis:google-api-services-bigquery to v2-rev20240919-2.0.0 (#3514) (9fe3829)
• Update dependency com.google.cloud:sdk-platform-java-config to v3.37.0 (bf4d37a)
• Update github/codeql-action action to v2.26.11 (#3517) (ac736bb)
• Update github/codeql-action action to v2.26.12 (#3522) (fdf8dc4)

You can now use fine-grained DML to optimize the execution of UPDATE, DELETE, and MERGE statements on tables. This feature is in Preview.

October 11, 2024

Use the BigQuery migration assessment for Oracle to assess the complexity of migrating data from your Oracle data warehouse to BigQuery. This feature is in preview.

October 10, 2024

BigQuery tables for Apache Iceberg bring the convenience of BigQuery storage optimization to Apache Iceberg tables that reside in your own cloud buckets. BigQuery tables for Apache Iceberg let you use BigQuery without moving data out of buckets that you control. This feature is now in preview.

You can now export and load Parquet files that include GeoParquet metadata. This feature is generally available (GA).

October 08, 2024

You can now use pipe syntax anywhere you write GoogleSQL. Pipe syntax supports a linear query structure designed to make your queries easier to read, write, and maintain. This feature is in Preview.

October 07, 2024

A weekly digest of client library updates from across the Cloud SDK.

Go

Changes for bigquery/storage/apiv1beta1

1.63.1 (2024-10-01)

Bug Fixes

• bigquery/storage: Increase method timeout to 240s for BigQuery Metastore Partition Service API version v1alpha (fdb4ea9)
• bigquery: Create read session with client or job projectID (#10932) (f98396e)
• bigquery: Missing schema for empty result set on stateless queries (#10935) (28a069a)

Java

Changes for google-cloud-bigquery

2.43.0 (2024-10-01)

Features

• Add max staleness to ExternalTableDefinition (#3499) (f1ebd5b)

2.42.4 (2024-09-30)

Dependencies

• Update actions/checkout action to v4.2.0 (#3495) (b57fefb)
• Update dependency com.google.api.grpc:proto-google-cloud-bigqueryconnection-v1 to v2.51.0 (#3480) (986b036)
• Update dependency com.google.api.grpc:proto-google-cloud-bigqueryconnection-v1 to v2.53.0 (#3504) (57ce901)
• Update dependency com.google.apis:google-api-services-bigquery to v2-rev20240905-2.0.0 (#3483) (a6508a2)
• Update dependency com.google.cloud:google-cloud-datacatalog-bom to v1.55.0 (#3481) (8908cfd)
• Update dependency com.google.cloud:google-cloud-datacatalog-bom to v1.57.0 (#3505) (6e78f56)
• Update dependency com.google.cloud:sdk-platform-java-config to v3.36.0 (#3490) (a72c582)
• Update dependency com.google.cloud:sdk-platform-java-config to v3.36.1 (#3496) (8f2e5c5)
• Update dependency ubuntu to v24 (#3498) (4f87ade)
• Update github/codeql-action action to v2.26.10 (#3506) (ca71294)
• Update github/codeql-action action to v2.26.7 (#3482) (e2c94b6)
• Update github/codeql-action action to v2.26.8 (#3488) (a6d75de)
• Update github/codeql-action action to v2.26.9 (#3494) (8154043)

October 03, 2024

You can now create an external dataset in BigQuery that links to an existing database in Spanner. This feature is in preview.

ODBC driver update, release 3.0.7 1016

• [New] Connector authentication on Google Cloud VMs: The connector now supports authentication through Application Default Credentials using the Google internal metadata server, eliminating the need for a keyfile. This feature works only on Google Cloud Compute Engine VMs.
• [Resolved] The output for PrimaryKeys previously denoted the Key Sequence as a 0-indexed value. This has been corrected to a 1-indexed value, indicating the sequential order of the primary key's column within the primary key itself.

September 30, 2024

A weekly digest of client library updates from across the Cloud SDK.

Node.js

Changes for @google-cloud/bigquery

7.9.1 (2024-09-23)

Bug Fixes

• Throw timeout error when using jobs.query (#1402) (cf962a5)

Python

Changes for google-cloud-bigquery

3.26.0 (2024-09-25)

Features

• Include LegacyPandasError in init imports (#2014) (3ab5e95)
• Use bigquery-magics package for the %%bigquery magic (#1965) (60128a5)

Bug Fixes

• Add docfx to the presubmit configuration and delete docs-presubmit (#1995) (bd83cfd)
• Add warning when encountering unknown field types (#1989) (8f5a41d)
• Allow protobuf 5.x; require protobuf >=3.20.2; proto-plus >=1.22.3 (#1976) (57bf873)
• Do not set job timeout extra property if None (#1987) (edcb79c)
• Set pyarrow field nullable to False for a BigQuery field in REPEATED mode (#1999) (5352870)

Dependencies

• Bump min version of google-api-core and google-cloud-core to 2.x (#1972) (a958732)

Documentation

• Add short mode query sample & test (#1978) (ba61a8a)
• Improve QueryJobConfig.destination docstring (#2016) (1b4cca0)

You can now enable, disable, and analyze history-based optimizations for queries. This feature is generally available (GA).

You can now use flexible column names with BigQuery tables and views for extracting, loading, streaming, and querying data. This feature is generally available (GA).

You can now use the operational health dashboard to get a single-pane view of key metrics such as slot usage, shuffle usage, errors, and total storage in real time. This feature is generally available (GA).

You can now create a materialized view replica directly from the Google Cloud console. This feature is generally available (GA).

September 26, 2024

Cloud console updates: You can now use keyboard shortcuts to control tab navigation in the details pane. This feature is generally available (GA).

September 24, 2024

You can now use Cloud KMS Autokey to automate the creation and use of customer-managed encryption keys (CMEKs), including the Cloud HSM service. This feature is generally available (GA).

BigQuery ML now offers the following expanded embedding support features:

• Using the ML.GENERATE_EMBEDDING function with a remote model based on a Vertex AI multimodalembedding large language model (LLM) to create multimodal embeddings, which embed text, image, and video into the same semantic space.
• Using the ML.GENERATE_EMBEDDING function with a principal component analysis (PCA) model or autoencoder model to create embeddings for structured independent and identically distributed random variables (IID) data.
• Using the ML.GENERATE_EMBEDDING function with a matrix factorization model to create embeddings for user or item data.

Try these capabilities with the following tutorials:

• Generate image embeddings by using the ML.GENERATE_EMBEDDING function
• Generate video embeddings by using the ML.GENERATE_EMBEDDING function
• Generate text embeddings by using the ML.GENERATE_EMBEDDING function
• Generate and search multimodal embeddings

These features are generally available (GA).

BigQuery ML now offers the following AI features:

• You can process documents from BigQuery object tables by doing the following:

  1. Creating a remote model based on the Document AI API, including specifying a document processor to use.
  2. Using the ML.PROCESS_DOCUMENT function with a Document AI-based remote model to process the documents.

  Try this feature with the Process documents with the ML.PROCESS_DOCUMENT function how-to.

• You can transcribe audio files from BigQuery object tables by doing the following:

  1. Creating a remote model based on the Speech-to-Text API, including specifying a speech recognizer to use.
  2. Using the ML.TRANSCRIBE function with a Speech-to-Text-based remote model to transcribe the audio files.

  Try this feature with the Transcribe audio files with the ML.TRANSCRIBE function how-to.

These BigQuery ML feature are generally available (GA).

September 23, 2024

A weekly digest of client library updates from across the Cloud SDK.

Go

Changes for bigquery/storage/apiv1beta1

1.63.0 (2024-09-18)

Features

• bigquery/migration: Update MS API stubs with Unified API (1bb4c84)
• bigquery: Add support for Go 1.23 iterators (84461c0)
• bigquery: New client(s) (#10774) (9638b8d)

Bug Fixes

• bigquery: Bump dependencies (2ddeb15)
• bigquery: ProcessStream check ctx done when queuing non retryable err (#10675) (60ad7f3)
• bigquery: Properly handle RANGE type arrays (#10883) (ce3d492)
• bigquery: Remove retry on FailedPrecondition (#10671) (ab9a961)
• bigquery: Update dependencies (257c40b)
• bigquery: Update google.golang.org/api to v0.191.0 (5b32644)

Documentation

• bigquery/datatransfer: Add a note to the CreateTransferConfigRequest and UpdateTransferConfigRequest to disable restricting service account usage (2710d0f)
• bigquery/datatransfer: Deprecate authorization_code (84461c0)
• bigquery/migration: A comment for field name in message .google.cloud.bigquery.migration.v2.MigrationWorkflow is changed to include 'Identifier' (1bb4c84)
• bigquery/migration: A comment for field translation_config_details in message .google.cloud.bigquery.migration.v2.MigrationTask is changed (1bb4c84)
• bigquery/migration: A comment for field type in message .google.cloud.bigquery.migration.v2.MigrationTask is changed to include new supported types (1bb4c84)
• bigquery/storage: A comment for field location_uri in message .google.cloud.bigquery.storage.v1alpha.StorageDescriptor is changed (2710d0f)
• bigquery/storage: A comment for message StreamMetastorePartitionsRequest is changed (2710d0f)
• bigquery/storage: A comment for message StreamMetastorePartitionsResponse is changed (2710d0f)

You can now create workflows to execute code assets in sequence at a scheduled time. This feature is in Preview.

September 19, 2024

You can perform model monitoring in BigQuery ML. The following model monitoring functions are now generally available (GA):

• ML.DESCRIBE_DATA: compute descriptive statistics for a set of training or serving data.
• ML.VALIDATE_DATA_SKEW: compute the statistics for a set of serving data, and then compare them to the statistics for the data used to train a BigQuery ML model in order to identify anomalous differences between the two data sets.
• ML.VALIDATE_DATA_DRIFT: compute and compare the statistics for two sets of serving data in order to identify anomalous differences between the two data sets.
• ML.TFDV_DESCRIBE: compute fine-grained descriptive statistics for a set of training or serving data. This function provides the same behavior as the TensorFlow tfdv.generate_statistics_from_csv API.
• ML.TFDV_VALIDATE: compute and compare the statistics for training and serving data, or two sets of serving data, in order to identify anomalous differences between the two data sets. This function provides the same behavior as the TensorFlow tfdv.validate_statistics API.

September 16, 2024

You can now batch migrate classic saved queries to saved queries. This feature is in Preview for projects that have fewer than 2500 classic saved queries.

A weekly digest of client library updates from across the Cloud SDK.

Java

Changes for google-cloud-bigquery

2.42.3 (2024-09-12)

Dependencies

• Update actions/upload-artifact action to v4.4.0 (#3467) (08b28c5)
• Update dependency com.google.cloud:sdk-platform-java-config to v3.35.0 (#3472) (fa9ac5d)

You can now use a CREATE MODEL statement to create a contribution analysis model in BigQuery ML. You can use a contribution analysis model with the ML.GET_INSIGHTS function to generate insights about changes to key metrics in your multi-dimensional data.

Try this feature with the Get data insights from a contribution analysis model tutorial.

This feature is in preview.

You can store columns in your vector indexes and pre-filter data in your vector searches to improve query efficiency. This feature is Generally Available.

September 12, 2024

You can now use the partial ordering mode in BigQuery DataFrames to generate more efficient queries. This feature is in Preview.

September 11, 2024

You can now use Terraform to manage IAM tags on datasets and tables. This feature is generally available (GA).

September 09, 2024

The BigQuery Data Transfer Service can now transfer campaign reporting and configuration data from Display & Video 360 into BigQuery, including Creative, Partner, and Advertiser tables. This feature is generally available (GA).

September 04, 2024

You can now use vector search and vector index features in BigQuery.

You can use the VECTOR_SEARCH function to search embeddings in order to identify semantically similar entities.

You can use vector indexes to make VECTOR_SEARCH more efficient, with the trade-off of returning more approximate results.

You can try the vector search and vector index capabilities by using the Search embeddings with vector search tutorial.

The BigQuery vector search and vector index features are generally available (GA).

September 02, 2024

A weekly digest of client library updates from across the Cloud SDK.

Java

Changes for google-cloud-bigquery

2.42.2 (2024-08-29)

Bug Fixes

• ExecuteSelect now use provided credentials instead of GOOGLE_APP… (#3465) (cd82235)

Dependencies

• Update actions/upload-artifact action to v4.3.5 (#3456) (f00977c)
• Update actions/upload-artifact action to v4.3.5 (#3462) (e1c6e92)
• Update actions/upload-artifact action to v4.3.6 (#3463) (ba91227)
• Update github/codeql-action action to v2.26.6 (#3464) (2aeb44d)

2.42.1 (2024-08-27)

Bug Fixes

• NPE for executeSelect nonFast path with empty result (#3445) (d0d758a)

Dependencies

• Update actions/upload-artifact action to v4.3.5 (#3420) (d5ec87d)
• Update actions/upload-artifact action to v4.3.5 (#3422) (c7d07b3)
• Update actions/upload-artifact action to v4.3.5 (#3424) (a9d6869)
• Update actions/upload-artifact action to v4.3.5 (#3427) (022eb57)
• Update actions/upload-artifact action to v4.3.5 (#3430) (c7aacba)
• Update actions/upload-artifact action to v4.3.5 (#3432) (b7e8244)
• Update actions/upload-artifact action to v4.3.5 (#3436) (ccefd6e)
• Update actions/upload-artifact action to v4.3.5 (#3440) (916fe9a)
• Update actions/upload-artifact action to v4.3.5 (#3443) (187f099)
• Update actions/upload-artifact action to v4.3.5 (#3444) (04aea5e)
• Update actions/upload-artifact action to v4.3.5 (#3449) (c6e93cd)
• Update actions/upload-artifact action to v4.3.5 (#3455) (fbfc106)
• Update dependency com.google.api.grpc:proto-google-cloud-bigqueryconnection-v1 to v2.49.0 (#3417) (66336a8)
• Update dependency com.google.api.grpc:proto-google-cloud-bigqueryconnection-v1 to v2.50.0 (#3448) (2c12839)
• Update dependency com.google.apis:google-api-services-bigquery to v2-rev20240714-2.0.0 (#3412) (8a48fd1)
• Update dependency com.google.apis:google-api-services-bigquery to v2-rev20240727-2.0.0 (#3421) (91d780b)
• Update dependency com.google.apis:google-api-services-bigquery to v2-rev20240727-2.0.0 (#3423) (16f350c)
• Update dependency com.google.apis:google-api-services-bigquery to v2-rev20240727-2.0.0 (#3428) (9ae6eca)
• Update dependency com.google.apis:google-api-services-bigquery to v2-rev20240803-2.0.0 (#3435) (b4e20db)
• Update dependency com.google.apis:google-api-services-bigquery to v2-rev20240815-2.0.0 (#3454) (8796aee)
• Update dependency com.google.cloud:google-cloud-bigquerystorage-bom to v3.9.0 (c4afbef)
• Update dependency com.google.cloud:google-cloud-datacatalog-bom to v1.53.0 (#3418) (6cff7f0)
• Update dependency com.google.cloud:google-cloud-datacatalog-bom to v1.54.0 (#3450) (cc9da95)
• Update dependency com.google.cloud:sdk-platform-java-config to v3.34.0 (#3433) (801f441)
• Update github/codeql-action action to v2.26.2 (#3426) (0a6574f)
• Update github/codeql-action action to v2.26.3 (#3438) (390e182)
• Update github/codeql-action action to v2.26.5 (#3446) (58aacc5)

Documentation

• Update iam policy sample user to be consistent with other languages (#3429) (2fc15b3)

August 29, 2024

The BigQuery Data Transfer Service now supports incremental transfers when you migrate your data from your Teradata data warehouses to BigQuery. This feature is generally available (GA).

Delta Lake BigLake tables are now generally available (GA). Delta Lake is an open source, tabular data storage format that supports petabyte scale data tables.

August 28, 2024

You can now use the GROUP BY clause and the SELECT DISTINCT clause with the ARRAY and STRUCT data types. This feature is in Preview.

You can now query data in AlloyDB using a federated query. This feature is now generally available (GA).

The following Gemini in BigQuery features are now generally available (GA):

• Data insights
• Data canvas
• SQL and Python code assistance features:
  • Use the SQL generation tool
  • Prompt to generate SQL queries
  • Explain a SQL query
  • Generate Python code
• Partitioning and clustering recommendations

To learn how to enable and activate Gemini in BigQuery features, see Set up Gemini in BigQuery.

Phrase support for the SEARCH function is now generally available (GA).

August 26, 2024

You can now create remote models in BigQuery ML based on the Anthropic Claude model in Vertex AI.

Use the ML.GENERATE_TEXT function with these remote models to perform generative natural language tasks for text stored in BigQuery tables. Try this feature with the Generate text by using the ML.GENERATE_TEXT function how-to topic.

This feature is in preview.

You can now use EXPORT DATA statements to directly export BigQuery data to Bigtable (reverse ETL). This feature is generally available (GA).

August 21, 2024

Python code completion is now available for all BigQuery projects. This feature is available in preview. To learn how to enable and activate Gemini in BigQuery features, see Set up Gemini in BigQuery.

August 20, 2024

You can now perform anomaly detection with BigQuery ML multivariate time series (ARIMA_PLUS_XREG) models. This feature lets you detect anomalies in historical time series data or in new data with multiple feature columns. You can try this feature by using the Perform anomaly detection with a multivariate time-series forecasting model tutorial. This feature is generally available (GA).

August 19, 2024

You can now view your BigQuery insights and recommendations using the Recommendations page in the Google Cloud console. You can also view your BigQuery insights and recommendations using the following INFORMATION_SCHEMA views:

• INSIGHTS
• RECOMMENDATIONS
• RECOMMENDATIONS_BY_ORGANIZATION

These features are now in preview.

August 14, 2024

You can now get lower latency for small queries with the new short query optimized mode. BigQuery automatically determines which queries may be accelerated while other queries continue to run like before. This feature is now in preview.

August 12, 2024

A weekly digest of client library updates from across the Cloud SDK.

Node.js

Changes for @google-cloud/bigquery

7.9.0 (2024-08-01)

Features

• Add retryOptions passing to underlying Service class (#1390) (a7cd3af)
• Clean cached rows and responses after conversion (#1393) (3fd28b8)

Bug Fixes

• Add missing alias for valid types (#1388) (7195951)

You can now use time series and range functions to support time series analysis. This feature is now generally available (GA).

August 08, 2024

The JSON_KEYS function, which extracts unique JSON keys from a JSON expression, is in Preview.

Some JSON functions that take a JSONPath let you specify a mode that allows flexibility in how the JSONPath matches the JSON data structure. This feature is in Preview.

August 07, 2024

An updated version of JDBC driver for BigQuery is now available.

You can now create a materialized view over Apache Iceberg table that is partition aligned with the base table. The materialized view only supports time-based partition transformation, for example, YEAR, MONTH, DAY, and HOUR. This feature is in preview.

July 31, 2024

When you translate SQL queries from your source database, you can use configuration YAML files to optimize and improve the performance of your translated SQL. This feature is in preview.

Workload management now provides the following benefits:

• The autoscaler now scales up immediately.
• The autoscaler now scales more precisely.
• The autoscaler scales to the nearest multiple of 50 slots, instead of 100.
• You can now purchase capacity commitments, set baseline slots, and set autoscale max slots in incremental steps of 50 slots.
• If one minute or more has passed since the most recent increase in capacity, you can now reduce capacity without resetting the one minute minimum. This allows for multiple consecutive decreases without a one minute delay between them.

These features are now generally available (GA).

July 30, 2024

You can now use the output_dimensionality argument of the ML.GENERATE_EMBEDDING function when you use the function with a remote model based on a Vertex AI multimodalembedding model. The output_dimensionality argument lets you specify the number of dimensions to use when generating embeddings. This feature is in Preview.

July 29, 2024

The RANGE data type is now a supported JSON encoding. This feature is Generally Available (GA).

A weekly digest of client library updates from across the Cloud SDK.

Go

Changes for bigquery/storage/apiv1beta1

1.62.0 (2024-07-22)

Features

• bigquery/analyticshub: Support Direct Table Access Toggle (Egress GA) (b660d68)
• bigquery/analyticshub: Support public directory self service for Listings/Exchanges (#10485) (b660d68)
• bigquery: Add rounding mode to FieldSchema (#10328) (1a9e204)
• bigquery: Json support on managedwriter/adapt pkg (#10542) (978d4a1)
• bigquery: Support column name character map in load jobs (#10425) (b829327)

Bug Fixes

• bigquery/storage/managedwriter: Faster context failure on send (#10169) (1fb0e64)
• bigquery: Bump google.golang.org/api@v0.187.0 (8fa9e39)
• bigquery: Bump google.golang.org/grpc@v1.64.1 (8ecc4e9)
• bigquery: Empty slice instead of nil slice for primitive repeated fields (#7315) (b371210)
• bigquery: Reduce default backoffs (#10558) (037e9ef)

Documentation

• bigquery/analyticshub: A comment for message DataExchange is changed (b660d68)
• bigquery/analyticshub: A comment for message Listing is changed (b660d68)
• bigquery/datatransfer: Update OAuth links in CreateTransferConfigRequest and UpdateTransferConfigRequest (3df3c04)
• bigquery: Improve Inserter and StructSaver godoc (#10170) (c1cffb6)
• bigquery: Update description of query preview feature (#10554) (25c5cbe)

Java

Changes for google-cloud-bigquery

2.42.0 (2024-07-28)

Features

• Add ability to specify RetryOptions and BigQueryRetryConfig when create job and waitFor (#3398) (1f91ae7)
• Add additional parameters to CsvOptions and ParquetOptions (#3370) (34f16fb)
• Add remaining Statement Types (#3381) (5f39b19)

Bug Fixes

• Null field mode inconsistency (#2863) (b9e96e3)

Dependencies

• Update actions/upload-artifact action to v4.3.4 (#3382) (efa1aef)
• Update dependency com.google.api.grpc:proto-google-cloud-bigqueryconnection-v1 to v2.48.0 (#3374) (45b7f20)
• Update dependency com.google.apis:google-api-services-bigquery to v2-rev20240616-2.0.0 (#3368) (ceb270c)
• Update dependency com.google.apis:google-api-services-bigquery to v2-rev20240623-2.0.0 (#3384) (e1de34f)
• Update dependency com.google.apis:google-api-services-bigquery to v2-rev20240629-2.0.0 (#3392) (352562d)
• Update dependency com.google.cloud:google-cloud-datacatalog-bom to v1.52.0 (#3375) (2115c04)
• Update dependency com.google.cloud:sdk-platform-java-config to v3.33.0 (#3405) (a4a9999)
• Update dependency org.junit.vintage:junit-vintage-engine to v5.10.3 (#3371) (2e804c5)
• Update github/codeql-action action to v2.25.11 (#3376) (f1e0014)
• Update github/codeql-action action to v2.25.12 (#3387) (af60b30)
• Update github/codeql-action action to v2.25.13 (#3395) (95c8d6f)
• Update github/codeql-action action to v2.25.15 (#3402) (a61ce7d)
• Update ossf/scorecard-action action to v2.4.0 (#3408) (66777a2)

Documentation

• Add short mode query sample (#3397) (6dca6ff)
• Add simple query connection read api sample (#3394) (d407baa)

You can now use the administrative jobs explorer to help you quickly monitor jobs activity across your organization. This feature is generally available (GA).

Vector indexes support the TreeAH index type, which uses Google's ScaNN algorithm. The TreeAH index is optimized for batch queries that process hundreds or more query vectors. This feature is in Preview.

July 25, 2024

You can now use table explorer to examine table data and create data exploration queries. This feature is in preview.

IAM deny policies now support additional permissions, including bigquery.tables.getData which can deny permission to read tables. Consider special cases when you create deny policies for bigquery.tables.getData and other BigQuery permissions. This feature is in preview.

July 23, 2024

Starting September 17, 2024, the bigquery.datasets.update permission check when creating or updating authorized datasets will be removed. For more information, see Required permissions and roles for authorized datasets.

You can now configure SAP Datasphere connections with network attachments to help secure connections. SAP Datasphere connections are in preview.

Manifest files are now supported for Amazon S3 and Azure Blob Storage. This feature is generally available (GA).

July 22, 2024

The CHANGES change history function is now in preview. This table-valued function provides a history of table changes over a window of time and captures the following operations:

• CREATE TABLE DDL statement
• INSERT DML statement
• Data appended or changed as part of a MERGE DML statement
• UPDATE DML statement
• DELETE DML statement
• Loading data into BigQuery
• Streaming ingestion
• TRUNCATE TABLE DML statement
• Jobs configured with a writeDisposition of WRITE_TRUNCATE
• Individual table partition deletions

You can use data manipulation language (DML) to modify rows that have been recently written to a BigQuery table by the Storage Write API. This is now generally available (GA).

The BigQuery continuous queries feature is now in preview.

Continuous queries let you build long-lived, continuously processing SQL statements that can analyze, process, and perform machine learning (ML) inference on incoming data in BigQuery in real time. You can configure continuous queries to replicate query results to a Pub/Sub topic, Bigtable instance, or another BigQuery table, a process also known as Reverse ETL.

You can use continuous queries to perform the following tasks, using the accessible language of SQL:

• Transform incoming data and act immediately on insights.
• Use Vertex AI to apply real time ML insights.
• Build automated event-driven data pipelines.
• Replicate real-time events to downstream operational systems like Bigtable.

To try BigQuery continuous queries, see Create continuous queries.

You can now use BigQuery Omni Virtual Private Cloud (VPC) allowlists to restrict access to AWS S3 buckets and Azure Blob Storage from specific BigQuery Omni VPCs. This feature is in preview.

July 18, 2024

The following BigQuery migration assessment features are now generally available (GA):

• When you run a migration assessment, the migration assessment now automatically creates a BigQuery dataset to store the assessment results. You can also choose to store assessment results in an existing empty dataset or manually create a dataset with a custom name.
• While a migration assessment is running, you can view the assessment report with partial data. You can also view its progress and estimated completion time in the status icon tooltip.
• You can view more information and errors about a migration assessment in the assessment details page.

July 17, 2024

You can now configure the default storage billing model for new datasets. This feature is generally available (GA).

July 16, 2024

When you run a migration assessment for Amazon Redshift, Teradata, or Snowflake, the service also creates a dataset containing only highly aggregated assessment results. This aggregated dataset doesn't contain any query logs; therefore, no personally identifiable information (PII) or business-sensitive information is visible. You can share this dataset with users that are not in your project. This feature is in preview.

July 11, 2024

You can now use EXPORT DATA statements to reverse ETL BigQuery data to Spanner. This feature is in preview.

July 01, 2024

A weekly digest of client library updates from across the Cloud SDK.

Node.js

Changes for @google-cloud/bigquery

7.8.0 (2024-06-19)

Features

• Update types from Discovery (#1374) (e416b6e)

Java

Changes for google-cloud-bigquery

2.41.0 (2024-06-25)

Features

• Add columnNameCharacterMap to LoadJobConfiguration (#3356) (2f3cbe3)
• Add MetadataCacheMode to ExternalTableDefinition (#3351) (2814dc4)

Bug Fixes

• Add clustering value to ListTables result (#3359) (5d52bc9)

Dependencies

• Update actions/checkout action to v4.1.7 (#3349) (0857234)
• Update dependency com.google.apis:google-api-services-bigquery to v2-rev20240602-2.0.0 (#3273) (7b7e52b)
• Update dependency com.google.cloud:sdk-platform-java-config to v3.32.0 (#3360) (4420996)
• Update github/codeql-action action to v2.25.10 (#3348) (8b6feff)

Cloud console updates: You can now drag a tab in the details pane to open a new column and compare tabs. You can also drag the tab to a new position in the current or an adjacent column. This feature is in preview.

The following Analytics Hub features are now generally available:

• Making exchanges and listings publicly discoverable.
• Highlighting listings in the Featured section of the Analytics Hub catalog.
• Generating unauthenticated URLs for public listings.

Data publishers can now share Pub/Sub topics and manage subscriptions in Analytics Hub. This feature is in preview.

June 27, 2024

You can now use tags on BigQuery tables to conditionally grant or deny access with Identity and Access Management (IAM) policies. This feature is generally available (GA). You can also attach tags to BigQuery datasets during dataset creation to conditionally grant or deny access with IAM policies.

June 25, 2024

You can now use the BigQuery JupyterLab plugin to explore your data, use BigQuery DataFrames in a Jupyter notebook, and deploy a BigQuery DataFrames notebook to Cloud Composer. This feature is in preview.

June 24, 2024

A weekly digest of client library updates from across the Cloud SDK.

Python

Changes for google-cloud-bigquery

3.25.0 (2024-06-17)

Features

• Add prefer_bqstorage_client option for Connection (#1945) (bfdeb3f)
• Support load job option ColumnNameCharacterMap (#1952) (7e522ee)

Bug Fixes

• Do not overwrite page_size with max_results when start_index is set (#1956) (7d0fcee)

June 21, 2024

The BigQuery migration assessment for Amazon Redshift is now generally available (GA). You can use this feature to assess the complexity of migrating from your Amazon Redshift data warehouse to BigQuery.

June 18, 2024

Additional collation support for the NULLIF conditional expression has been added. The NULLIF conditional expression is now affected by collation and can be used in collation-supported comparisons with the STRUCT data type. This feature is generally available (GA).

June 17, 2024

A weekly digest of client library updates from across the Cloud SDK.

Java

Changes for google-cloud-bigquery

2.40.3 (2024-06-12)

Dependencies

• Update actions/checkout action to v4.1.6 (#3309) (c7d6362)
• Update dependency com.google.api.grpc:proto-google-cloud-bigqueryconnection-v1 to v2.46.0 (#3328) (a6661ad)
• Update dependency com.google.api.grpc:proto-google-cloud-bigqueryconnection-v1 to v2.47.0 (#3342) (79e34c2)
• Update dependency com.google.cloud:google-cloud-datacatalog-bom to v1.50.0 (#3330) (cabb0ab)
• Update dependency com.google.cloud:google-cloud-datacatalog-bom to v1.51.0 (#3343) (e3b934f)
• Update dependency com.google.cloud:sdk-platform-java-config to v3.31.0 (#3335) (0623455)
• Update dependency com.google.oauth-client:google-oauth-client-java6 to v1.36.0 (#3305) (d05e554)
• Update dependency com.google.oauth-client:google-oauth-client-jetty to v1.36.0 (#3306) (0eeed66)
• Update dependency org.graalvm.buildtools:junit-platform-native to v0.10.2 (#3311) (3912a92)
• Update dependency org.graalvm.buildtools:native-maven-plugin to v0.10.2 (#3312) (9737a5d)
• Update github/codeql-action action to v2.25.6 (#3307) (8999d33)
• Update github/codeql-action action to v2.25.7 (#3334) (768342d)
• Update github/codeql-action action to v2.25.8 (#3338) (8673fe5)

You can now perform supervised tuning on a BigQuery ML remote model based on a gemini-1.0-pro-002 model. This feature is in preview. To try this feature, see Tune a model using your data.

You can also perform supervised tuning by using the BigQuery DataFrames Python API. Use the fit() and score() methods in the bigframes.ml.llm.GeminiTextGenerator model class to perform supervised tuning.

Global rate limits on BigQuery Omni connection creation and use have replaced the regional limits on AWS and Azure connections.

June 13, 2024

You can now schedule notebooks. This feature is available in preview.

June 10, 2024

A weekly digest of client library updates from across the Cloud SDK.

Python

Changes for google-cloud-bigquery

3.24.0 (2024-06-04)

Features

• Add default timeout for Client.get_job() (#1935) (9fbad76)
• Add support for map target type in Parquet options (#1919) (c3f7b23)

Bug Fixes

• Create query job in job.result() if doesn't exist (#1944) (8f5b4b7)
• Retry is_job_done on ConnectionError (#1930) (4f72723)

Performance Improvements

• If page_size or max_results is set on QueryJob.result(), use to download first page of results (#1942) (3e7a48d)

June 05, 2024

The BigQuery ML ML.GENERATE_EMBEDDING function now supports the output_dimensionality argument for text-embedding and text-multilingual-embedding models. The output_dimensionality argument lets you specify the number of dimensions to use when generating embeddings.

Analytics Hub data egress controls are now generally available (GA). Publishers can now enforce egress restrictions on Analytics Hub listings to prevent subscribers from copying or exporting the shared data.

The slot recommender for editions analyzes historical usage data to recommend optimal capacity purchasing for edition and on-demand workloads. This feature is generally available (GA).

June 03, 2024

A weekly digest of client library updates from across the Cloud SDK.

Node.js

Changes for @google-cloud/bigquery

7.7.1 (2024-05-31)

Bug Fixes

• Return extra args when autoPaginate is on (#1365) (9d77cd8), closes #1362

Java

Changes for google-cloud-bigquery

2.40.2 (2024-05-26)

Bug Fixes

• Fixing NPE bug by adding to if clause (#3290) (127cff9)

Dependencies

• Update dependency com.google.api.grpc:proto-google-cloud-bigqueryconnection-v1 to v2.45.0 (#3295) (c659523)
• Update dependency com.google.cloud:google-cloud-datacatalog-bom to v1.49.0 (#3296) (7d148d5)
• Update dependency com.google.cloud:sdk-platform-java-config to v3.30.1 (#3310) (641f1a8)
• Update github/codeql-action action to v2.25.4 (#3291) (13bb5aa)
• Update ossf/scorecard-action action to v2.3.3 (#3304) (d096082)

May 31, 2024

You can now use IAM conditions to control access to BigQuery resources. This feature is generally available (GA).

May 30, 2024

You can now define a _CHANGE_SEQUENCE_NUMBER for BigQuery change data capture (CDC) to manage streaming UPSERT ordering for BigQuery. This feature is in preview.

May 29, 2024

The maximum number of partitions per partitioned table limit has changed from 4,000 to 10,000.

May 28, 2024

The following Generative AI features are now in preview:

• Creating remote models based on the Vertex AI gemini-1.5-flash foundation model.
• Using the ML.GENERATE_TEXT function with these remote models to perform generative natural language tasks for text stored in BigQuery tables.
• Using the ML.GENERATE_TEXT function with these remote models to perform generative AI tasks, for example audio transcription or document classification, using image, video, audio, PDF, or text content stored in BigQuery object tables.

Try these features with the Generate text by using the ML.GENERATE_TEXT function how-to topic.

May 27, 2024

A weekly digest of client library updates from across the Google Cloud SDK.

Python

Changes for google-cloud-bigquery

3.23.1 (2024-05-21)

Performance Improvements

• Decrease the threshold in which we use the BigQuery Storage Read API (#1925) (eaa1a52)

May 23, 2024

In BigQuery ML univariate time series models, the FORECAST_LIMIT_LOWER_BOUND and FORECAST_LIMIT_UPPER_BOUND parameters now work with the TIME_SERIES_ID_COL parameter. The FORECAST_LIMIT_LOWER_BOUND and FORECAST_LIMIT_UPPER_BOUND arguments let you set the lower and upper bounds of the forecasted values returned by the model. Try this feature with the Limit forecasted values for a time series model tutorial.

BigQuery ML now offers the following Generative AI features:

• Grounding and safety attributes when you use Vertex AI Gemini models with the ML.GENERATE_TEXT function:

  • Use the ground_with_google_search argument to perform grounding. Grounding lets the Gemini model use additional information from the internet when generating a response, in order to make model responses more specific and factual.
  • Use the safety_settings argument to configure safety attributes.The Gemini model filters the responses it returns based on the attributes you specify.

• Video embedding (Preview). You can use the ML.GENERATE_EMBEDDING function with a remote model based on a Vertex AI multimodalembedding model to create multimodal embeddings that include video embeddings.

  To try the new video embedding functionality, see Generate video embeddings by using the ML.GENERATE_EMBEDDING function.

May 22, 2024

The interactive SQL translator, the translation API, and the batch SQL translator features let you translate the following SQL dialects into GoogleSQL:

• IBM DB2 SQL
• Greenplum SQL
• SQLite

These features are in preview.

You can now query data in AlloyDB using a federated query. This feature is in preview.

May 21, 2024

The following Generative AI features are now in preview:

• Creating remote models based on the Vertex AI gemini-1.5-pro foundation model.
• Using the ML.GENERATE_TEXT function with these remote models to perform generative natural language tasks for text stored in BigQuery tables.
• Using the ML.GENERATE_TEXT function with these remote models to perform generative AI tasks, for example audio transcription or document classification, using image, video, audio, PDF, or text content stored in BigQuery object tables.

Try these features with the Generate text by using the ML.GENERATE_TEXT function how-to topic.

May 20, 2024

A weekly digest of client library updates from across the Cloud SDK.

Python

Changes for google-cloud-bigquery

3.23.0 (2024-05-16)

Features

• Adds timer decorator to facilitate debugging (#1917) (ea750e0)
• Support insertAll for range (#1909) (74e75e8)

Bug Fixes

• Add pyarrow version check for range support (#1914) (a86d7b9)
• Edit presubmit for to simplify configuration (#1915) (b739596)

You can now use a search index to optimize lookups on the INT64 and TIMESTAMP data types. The feature is in preview.

You can use DLP functions to support encryption and decryption between BigQuery and Sensitive Data Protection, using AES-SIV. This feature is now generally available (GA).

May 14, 2024

You can now create Gemini-enhanced translation rules to use with the interactive SQL translator. Translation rules let you customize and adjust the results of the interactive translator according to your SQL migration needs. This feature is in preview.

May 13, 2024

A weekly digest of client library updates from across the Cloud SDK.

Java

Changes for google-cloud-bigquery

2.40.1 (2024-05-06)

Dependencies

• Update actions/checkout action (#3286) (4d8f3fb)

2.40.0 (2024-05-06)

Features

• Add getStringOrDefault method to FieldValue (#3255) (8bac33a)

Dependencies

• Update dependency com.google.cloud:sdk-platform-java-config to v3.30.0 (#3279) (67f2ea4)

Python

Changes for google-cloud-bigquery

3.22.0 (2024-04-19)

Features

• Support RANGE in queries Part 2: Arrow (#1868) (5251b5d)

Phrase support for the SEARCH function is in preview.

May 07, 2024

JavaScript user-defined aggregate functions (UDAFs) are in preview. You can create a JavaScript UDAF with the CREATE AGGREGATE FUNCTION statement.

You can now store columns in your vector indexes and pre-filter data in your vector searches to improve query efficiency. This feature is in preview.

May 06, 2024

A weekly digest of client library updates from across the Cloud SDK.

Node.js

Changes for @google-cloud/bigquery

7.7.0 (2024-05-03)

Features

• Add support for RANGE type (#1352) (#1360) (961a802)

Java

Changes for google-cloud-bigquery

2.39.1 (2024-04-29)

Bug Fixes

• @Nullable annotations on builder methods (#3222) (0c5eed1)

Dependencies

• Update actions/checkout action (#3267) (c297ed2)
• Update actions/upload-artifact action to v4.3.3 (#3258) (5215235)
• Update dependency com.google.api.grpc:proto-google-cloud-bigqueryconnection-v1 to v2.44.0 (#3270) (ee09ab6)
• Update dependency com.google.cloud:google-cloud-bigquerystorage-bom to v3.5.0 (e7c6201)
• Update dependency com.google.cloud:google-cloud-datacatalog-bom to v1.48.0 (#3271) (3b6e0d5)
• Update github/codeql-action action to v2.25.2 (#3260) (3302dc4)
• Update github/codeql-action action to v2.25.3 (#3268) (1cf2377)

BigQuery Managed Disaster Recovery provides managed failover and redundant compute capacity for business critical workloads. It is intended for use in the case of a total region outage and is supported with the BigQuery Enterprise Plus edition only. This feature is now available in preview.

You can now create AWS Glue federated datasets using the the Google Cloud console. This feature is generally available (GA).

May 02, 2024

Analytics Hub Subscription Management is generally available (GA). Data Publishers can now manage their subscriptions, view information about their subscribers, and revoke access to their data at any time.

Analytics Hub Provider Usage Metrics is now generally available (GA). The usage metrics include the following:

• Jobs that run against your shared data.
• The consumption details of your shared data by subscribers' projects and organizations.
• The number of rows and bytes processed by the job.

April 30, 2024

AWS Glue federated datasets are now generally available (GA).

An AWS Glue federated dataset is a connection at the dataset level between BigQuery and an existing database in AWS Glue.

You can now specify translation configurations in the BigQuery interactive SQL translator and use it to debug batch SQL translator jobs. This feature is generally available (GA).

The following BigQuery ML data preprocessing features are now generally available (GA):

• The ML.TRANSFORM function, which you can use to preprocess feature data. This function processes input data by applying the data transformations captured in the TRANSFORM clause of an existing model.
• Transform-only models, which you can use to apply preprocessing functions to input data and return the preprocessed data. Transform-only models decouple data preprocessing from model training, making it easier for you to capture and reuse a set of data preprocessing rules.

You can now reference Iceberg tables in materialized views instead of migrating that data to BigQuery-managed storage. This feature is in preview.

April 29, 2024

A weekly digest of client library updates from across the Cloud SDK.

Node.js

Changes for @google-cloud/bigquery

7.6.1 (2024-04-26)

Bug Fixes

• Should not override queryParameters if params not informed (#1359) (2b66d1e)

Go

Changes for bigquery/storage/apiv1beta1

1.61.0 (2024-04-24)

Features

• bigquery/storage/managedwriter/adapt: Add RANGE support to adapt (#9836) (ae25253)
• bigquery: RANGE support for basic data movement (#9762) (07f0806)
• bigquery: RANGE support when reading Arrow format (#9795) (da245fa)
• bigquery: RANGE type StandardSQLDataType support (#9754) (33666cf)

Bug Fixes

• bigquery/datatransfer: Mark parent/name fields with the REQUIRED field_behavior annotation (8892943)
• bigquery: Bump x/net to v0.24.0 (ba31ed5)

Java

Changes for google-cloud-bigquery

2.39.0 (2024-04-22)

Features

• Add ExportDataStats to QueryStatistics (#3244) (e91be80)
• Add new fields to copy job statistics (#3205) (64bdda8)
• Add Range object to allow reading range value (#3236) (2c3399d)
• Add support for inserting Range values (#3246) (ff1ebc6)
• Add support for ObjectMetadata (#3217) (975df05)
• Add totalSlotMs to JobStatistics (#3250) (75ea095)

Bug Fixes

• Fix BigQuery#listDatasets to include dataset location in the response (#3238) (c50c17b)
• Remove @InternalApi from TableResult (#3257) (19d92a1)

Dependencies

• Update actions/checkout action (#3256) (6df3a32)
• Update actions/upload-artifact action to v4.3.2 (#3248) (066b51f)
• Update dependency com.google.api.grpc:proto-google-cloud-bigqueryconnection-v1 to v2.40.0 (#3210) (bf7e97e)
• Update dependency com.google.api.grpc:proto-google-cloud-bigqueryconnection-v1 to v2.41.0 (#3219) (9d71b8b)
• Update dependency com.google.api.grpc:proto-google-cloud-bigqueryconnection-v1 to v2.43.0 (#3225) (a897306)
• Update dependency com.google.apis:google-api-services-bigquery to v2-rev20240323-2.0.0 (#3239) (2c0f48f)
• Update dependency com.google.cloud:google-cloud-datacatalog-bom to v1.44.0 (#3211) (6993b51)
• Update dependency com.google.cloud:google-cloud-datacatalog-bom to v1.45.0 (#3220) (21ae09c)
• Update dependency com.google.cloud:google-cloud-datacatalog-bom to v1.47.0 (#3226) (d45d168)
• Update dependency com.google.cloud:sdk-platform-java-config to v3.28.1 (#3207) (6204331)
• Update dependency org.threeten:threeten-extra to v1.8.0 (#3242) (66d5efd)
• Update github/codeql-action action to v2.24.9 (#3204) (7a24d3e)
• Update github/codeql-action action to v2.25.1 (#3229) (aeedf29)

You can now let users that are in Microsoft Entra groups access BigQuery data in Power BI by using Workforce Identity Federation. This feature is generally available.

April 26, 2024

SQL code generation is now available for all BigQuery projects. This feature is available in preview. To learn how to enable and activate Gemini in BigQuery features, see Set up Gemini in BigQuery.

April 25, 2024

BigQuery Studio is now available in the following regions:

• Johannesburg (africa-south1)
• Hong Kong (asia-east2)
• Seoul (asia-northeast3)
• Jakarta (asia-southeast2)
• Sydney (australia-southeast1)
• Madrid (europe-southwest1)
• Turin (europe-west12)
• Doha (me-central1)
• Dammam (me-central2)
• Montréal (northamerica-northeast1)
• N. Virginia (us-east4)
• Columbus (us-east5)
• Dallas (us-south1)
• Los Angeles (us-west2)
• Las Vegas (us-west4)

For more information, see BigQuery Studio locations.

The BigQuery Data Transfer Service for Google Merchant Center supports the Product Targeting report.

April 24, 2024

User-defined aggregate functions (UDAFs) that support SQL expressions are in preview. You can create a UDAF with the CREATE AGGREGATE FUNCTION statement.

April 22, 2024

A weekly digest of client library updates from across the Cloud SDK.

Node.js

Changes for @google-cloud/bigquery

7.6.0 (2024-04-09)

Features

• Integrate jobs.query and stateless query for faster queries (#1337) (74aa150)

Bug Fixes

• Parsing zero value timestamp (#1355) (d433711)

Python

Changes for google-cloud-bigquery

3.21.0 (2024-04-18)

Features

• Add compression option ZSTD. (#1890) (5ed9cce)
• Adds billing to opentel (#1889) (38697fb)
• Support RANGE in queries Part 1: JSON (#1884) (3634405)

Bug Fixes

• Add types to DatasetReference constructor (#1601) (bf8861c)
• Creates linting-typing.cfg in presubmit (#1881) (c852c15)
• Remove duplicate key time_partitioning from Table._PROPERTY_TO_A… (#1898) (82ae908)
• Retry query jobs that fail even with ambiguous jobs.getQueryResults REST errors (#1903, #1900) (1367b58)

Performance Improvements

• Avoid unnecessary API call in QueryJob.result() when job is already finished (#1900) (1367b58)

April 18, 2024

The quantified LIKE operator is generally available (GA). With this operator, you can check a search value for matches against a list of patterns or an array of patterns, using one of these conditions:

• LIKE ANY: Checks if at least one pattern matches.
• LIKE SOME: Synonym for LIKE ANY.
• LIKE ALL: Checks if every pattern matches.

April 17, 2024

More permissions are now supported by deny policies. This feature is in preview.

April 16, 2024

BigQuery now supports subqueries in row level access policies. This feature is now in public preview.

April 09, 2024

BigQuery ML now offers the following expanded embedding support features in preview:

• Using the ML.GENERATE_EMBEDDING function with a remote model based on a Vertex AI multimodalembedding large language model (LLM) to create multimodal embeddings, which embed text and images into the same semantic space.
• Using the ML.GENERATE_EMBEDDING function with a principal component analysis (PCA) model or autoencoder model to create embeddings for structured independent and identically distributed random variables (IID) data.
• Using the ML.GENERATE_EMBEDDING function with a matrix factorization model to create embeddings for user or item data.

Try the new multimodal embedding functionality:

• Generate image embeddings by using the ML.GENERATE_EMBEDDING function
• Generate text embeddings by using the ML.GENERATE_EMBEDDING function
• Generate and search multimodal embeddings

You can now create a data canvas in BigQuery Studio. A data canvas lets you discover, transform, query, and visualize data using natural language. It provides a graphic interface for your analysis that lets you work with data sources, queries, and visualizations in a directed acyclic graph (DAG), giving you a graphical view of your analysis workflow that maps to your mental model. You can iterate on query results and work with multiple branches of inquiry in a single place. This feature is in preview and access can be requested here.

The following Gemini in BigQuery features are now available in Public Preview:

• Data insights: an automated and intuitive way to explore and understand your data.
• Data canvas: a graphic interface that lets you discover, transform, query, and visualize data using natural language.
• SQL and Python code assistance: Gemini-assisted code generation, completion, and explanation.
• Materialized views, partitioning, and clustering recommendations: recommendations to reduce cost and improve performance.
• Autotune and troubleshoot serverless Spark: optimize and explain Spark workloads.

To learn how to enable and activate Gemini in BigQuery features, see Set up Gemini in BigQuery.

April 08, 2024

BigQuery Studio is generally available (GA).

BigQuery Studio lets you save, share, and manage versions of code assets such as notebooks and saved queries.

BigQuery DataFrames is generally available (GA).

BigQuery DataFrames is a set of open source Python libraries that implements the pandas and scikit-learn APIs with server-side processing. To get started, you can try BigQuery DataFrames.

A weekly digest of client library updates from across the Cloud SDK.

Python

Changes for google-cloud-bigquery

3.20.1 (2024-04-01)

Bug Fixes

• Make pyarrow an optional dependency post-3.20.0 yanked release (#1879) (21714e1)

The BigQuery materialized view recommender analyzes your past query jobs to identify opportunities to apply materialized views to your queries for potential cost savings. You can view all available materialized view recommendations through the BigQuery UI or Recommender API. This feature is in preview.

April 05, 2024

Differential privacy is now generally available (GA).

You can now use BigLake to access Delta Lake tables. For more information, see Create Delta Lake BigLake tables. This feature is available in preview.

April 04, 2024

The allow_non_incremental_definition option and max_staleness option for materialized views are now generally available (GA). The allow_non_incremental_definition option supports an expanded range of SQL queries to create materialized views, and the max_staleness option provides consistently high performance with controlled costs when processing large, frequently changing datasets.

You can now perform model monitoring in BigQuery ML. The following model monitoring functions are now in preview:

• ML.DESCRIBE_DATA: compute descriptive statistics for a set of training or serving data.
• ML.VALIDATE_DATA_SKEW: compute the statistics for a set of serving data, and then compare them to the statistics for the data used to train a BigQuery ML model in order to identify anomalous differences between the two data sets.
• ML.VALIDATE_DATA_DRIFT: compute and compare the statistics for two sets of serving data in order to identify anomalous differences between the two data sets.
• ML.TFDV_DESCRIBE: compute fine-grained descriptive statistics for a set of training or serving data. This function provides the same behavior as the TensorFlow tfdv.generate_statistics_from_csv API.
• ML.TFDV_VALIDATE: compute and compare the statistics for training and serving data, or two sets of serving data, in order to identify anomalous differences between the two data sets. This function provides the same behavior as the TensorFlow validate_statistics API.

BigQuery data clean rooms with analysis rules and enhanced usage metrics are now generally available (GA). Data clean rooms provide a security-enhanced and privacy-preserving environment for multiple parties to share and augment data without moving or revealing the underlying data.

Join restrictions, list overlap, differential privacy with privacy budgeting, and aggregation thresholding are now enforceable in BigQuery data clean rooms using analysis rules.

April 03, 2024

Collation now supports the following generally available (GA) features:

• The underscore in the LIKE operator.
• Comparison support for the STRUCT data type with the following operators and conditional expressions: =, !=, IN and CASE.

You can now configure materialized views with tables enabled for change data capture (CDC) streaming update and delete operations.

April 02, 2024

The following BigQuery ML features are now in preview:

• Performing supervised tuning on a remote model based on a Vertex AI text-bison large language model (LLM).
• Evaluate a Vertex AI LLM using the ML.EVALUATE function. Pre-trained text-bison, text unicorn, or gemini-pro models and tuned text-bison models are supported for evaluation.

Try tuning and evaluating an LLM with the Customize an LLM by using supervised fine tuning how-to topic.

April 01, 2024

A weekly digest of client library updates from across the Cloud SDK.

Node.js

Changes for @google-cloud/bigquery

7.5.2 (2024-03-27)

Bug Fixes

• Set projectId in jobs emitted by load streams (#1346) (bd8254c)

Go

Changes for bigquery/storage/apiv1beta1

1.60.0 (2024-03-27)

Features

• bigquery/analyticshub: Support selective sharing on data clean room Listings (a3bb7c0)
• bigquery/datatransfer: Add UnenrollDataSources API which gives users a programmatic way to unenroll data sources (a86aa8e)
• bigquery/storage: Add the RANGE type to the google.cloud.bigquery.storage.v1.TableFieldSchema (0195fe9)

Bug Fixes

• bigquery/storage/managedwriter: Fix flowcontrol refund on error (#9649) (a07bf1d)
• bigquery/storage/managedwriter: Retry improvements (#9642) (48a9258)
• bigquery: Update protobuf dep to v1.33.0 (30b038d)

Documentation

• bigquery/datatransfer: Update unenrollDataSources API documentation (#9449) (da644cc)
• bigquery/storage: Mark BigQueryWrite v1beta2 as deprecated (d130d86)

Python

Changes for google-cloud-bigquery

3.20.0 (2024-03-27) - YANKED

Reason this release was yanked:

undeclared dependency on pyarrow

Features

• Add fields parameter to set_iam_policy for consistency with update methods (#1872) (08b1e6f)

Bug Fixes

• Correct type checking (#1848) (2660dbd)
• Update error logging when converting to pyarrow column fails (#1836) (0ac6e9b)
• Updates a number of optional dependencies (#1864) (c2496a1)
• Use an allowlist instead of denylist to determine when query_and_wait uses jobs.query API (#1869) (e265db6)

You can now enable, disable, and analyze history-based optimizations for queries. This feature is in preview.

BigQuery Studio is now available in the South Carolina (us-east1) region to manage versions of code assets such as notebooks and saved queries.

The BigQuery Data Transfer Service for Search Ads 360 now supports the new Search Ads 360 Reporting API. You can use the Search Ads 360 connector to specify custom Floodlight variables and custom columns when transferring Search Ads 360 data to BigQuery. This feature is now generally available (GA).

March 28, 2024

You can now query data in SAP Datasphere using a federated query. This feature is in preview.

Query optimization using search indexes is now applied to comparisons of string literals and indexed data, including the equal (=), IN, and LIKE operators and the STARTS_WITH function. This feature is generally available (GA).

March 27, 2024

An updated version of JDBC driver for BigQuery is now available.

March 26, 2024

The Help me code tool lets you use natural language to generate a SQL query that can then be run in BigQuery. This feature is now in preview.

The following Generative AI features are now in preview:

• Creating a remote model based on a Vertex AI gemini-pro-vision large vision model (VLM).
• Using the ML.GENERATE_TEXT function with this remote model to perform Vision Generative AI tasks, such as image or video captioning and visual Q&A, for visual content stored in BigQuery object tables.

Try these features with the Generate text that describes visual content how-to topic.

Duet AI in BigQuery is now Gemini for BigQuery. See our blog post for more information.

March 25, 2024

A weekly digest of client library updates from across the Cloud SDK.

Java

Changes for google-cloud-bigquery

2.38.2 (2024-03-21)

Dependencies

• Update actions/checkout action (#3190) (940e4f6)
• Update arrow.version to v15.0.1 (#3189) (fb6284e)
• Update dependency com.google.api.grpc:proto-google-cloud-bigqueryconnection-v1 to v2.39.0 (#3186) (9e705a1)
• Update dependency com.google.apis:google-api-services-bigquery to v2-rev20240229-2.0.0 (#3188) (a018424)
• Update dependency com.google.cloud:google-cloud-datacatalog-bom to v1.43.0 (#3187) (497ff29)
• Update dependency com.google.cloud:sdk-platform-java-config to v3.28.1 (#3196) (61f23a3)
• Update github/codeql-action action to v2.24.6 (#3178) (8843cae)
• Update github/codeql-action action to v2.24.7 (#3194) (2e2d730)
• Update github/codeql-action action to v2.24.8 (#3198) (bd81a56)

March 22, 2024

The March 20, 2024 release notes announced the preview for user-defined aggregate functions, but user-defined aggregate functions are not yet supported.

March 21, 2024

You can now add Salesforce Data Cloud data to BigQuery. This feature is generally available (GA).

Incremental materialized views now support LEFT OUTER JOIN and UNION ALL. This feature is in preview.

March 20, 2024

The maximum notebook size has been increased from 10 MB to 20 MB. Notebooks are available in preview.

You can now view lists of all saved queries and all notebooks in your project. These features are available in preview.

March 19, 2024

You can now create and run Spark stored procedures that are written in Python, Java, and Scala. You can also use the PySpark editor in BigQuery to create stored Python procedures for Apache Spark. This feature is now generally available (GA).

The minimum duration between scheduled queries has been reduced from 15 minutes to 5 minutes. This feature is generally available.

March 18, 2024

A weekly digest of client library updates from across the Cloud SDK.

Node.js

Changes for @google-cloud/bigquery

7.5.1 (2024-03-14)

Bug Fixes

• Add better documentation around usage of BigQueryTimestamp class and .timestamp method. (2b2c3e0)
• BigQueryTimestamp should keep accepting floats #1339 (2b2c3e0)
• Restores BigQueryTimestamp behavior to accept a numeric value in the constructor representing epoch-seconds. The affected 7.5.0 version would parse a numeric value as epoch-microseconds. (2b2c3e0)

Python

Changes for google-cloud-bigquery

3.19.0 (2024-03-11)

Features

• Support RANGE query parameters (#1827) (b359a9a)
• Support range sql (#1807) (86a45c9)

Bug Fixes

• Add google-auth as a direct dependency (713ce2c)
• Augment universe_domain handling (#1837) (53c2cbf)
• deps: Require google-api-core>=1.34.1, >=2.11.0 (713ce2c)
• Supplementary fix to env-based universe resolution (#1844) (b818992)
• Supplementary fix to env-based universe resolution (#1847) (6dff50f)

You can now undelete a dataset that is within your time travel window to recover it to the state that it was in when it was deleted. This feature is in preview.

These BigQuery features are now generally available (GA):

Text analysis configuration options for the following:

• CREATE SEARCH INDEX DDL
• Existing LOG_ANALYZER and new PATTERN_ANALYZER analyzers, which are used in various functions, including SEARCH
• The TEXT_ANALYZE function

The following advanced processing functions:

• ML.BAG_OF_WORDS
• ML.TF_IDF
• BAG_OF_WORDS
• TF_IDF
• COSINE_DISTANCE
• EUCLIDEAN_DISTANCE
• EDIT_DISTANCE

You can now perform hierarchical forecasts in BigQuery ML time series models, which let you aggregate and roll up values for all time series in the model. This feature is generally available (GA).

March 11, 2024

A weekly digest of client library updates from across the Cloud SDK.

Java

Changes for google-cloud-bigquery

2.38.1 (2024-03-07)

Dependencies

• Update dependency com.google.api.grpc:proto-google-cloud-bigqueryconnection-v1 to v2.38.0 (#3159) (d6c65ab)
• Update dependency com.google.cloud:google-cloud-datacatalog-bom to v1.42.0 (#3160) (e31b5b7)
• Update dependency com.google.cloud:sdk-platform-java-config to v3.27.0 (#3176) (b93e62e)
• Update dependency org.graalvm.buildtools:junit-platform-native to v0.10.1 (#3153) (436f58c)
• Update dependency org.graalvm.buildtools:native-maven-plugin to v0.10.1 (#3154) (b68ab42)
• Update github/codeql-action action to v2.24.5 (#3165) (8ac7722)

Python

Changes for google-cloud-bigquery

3.18.0 (2024-02-29)

Features

• Support nullable boolean and Int64 dtypes in insert_rows_from_dataframe (#1816) (ab0cf4c)
• Support slot_ms in QueryPlanEntry (#1831) (d62cabb)

Bug Fixes

• Keyword rendering and docstring improvements (#1829) (4dfb920)

Documentation

• samples: Updates to urllib3 constraint for Python 3.7 (#1834) (b099c32)
• Update client_query_w_named_params.py to use query_and_wait API (#1782) (89dfcb6)

March 06, 2024

The INFORMATION_SCHEMA.WRITE_API_TIMELINE* views, containing per minute aggregated BigQuery Storage Write API ingestion statistics, are now generally available (GA).

Duet AI in BigQuery can now assist with Python code generation and code completion. This feature is in preview.

March 04, 2024

A weekly digest of client library updates from across the Cloud SDK.

Node.js

Changes for @google-cloud/bigquery

7.5.0 (2024-02-21)

Features

• Add support for TPC Universes (#1333) (f67a841)
• Switch timestamp representation to int64 usec (#1332) (fb10f03)
• Update types and generation script (#1336) (496f52c)

You can now selectively refresh the metadata cache for BigLake tables by using the BQ.REFRESH_EXTERNAL_METADATA_CACHE system procedure. This feature is generally available (GA).

February 29, 2024

The following BigQuery cross-cloud features are now generally available (GA):

• You can take advantage of the benefits of materialized views over Amazon S3 metadata cache-enabled BigLake tables.
• You can create materialized view replicas of materialized views over Amazon S3 metadata cache-enabled Biglake tables. Materialized view replicas let you use the materialized view data in queries while avoiding data egress costs and improving query performance.
• You can get information about materialized view replicas by using SQL, the bq command-line tool, or the BigQuery API.
• You can use cross-cloud joins to run queries that span both Google Cloud and BigQuery Omni regions.

The SQL translation API combines the interactive and batch translator into a single workflow, improving the efficiency and stability of your translation jobs created using the API. This feature is available in preview.

February 28, 2024

Materialized views can now reference logical views. This feature is in preview.

The ability to perform anomaly detection with BigQuery ML multivariate time series (ARIMA_PLUS_XREG) models is now in preview. This feature enables you to detect anomalies in historical time series data or in new data with multiple feature columns. Try this new feature by using the Perform anomaly detection with a multivariate time-series forecasting model tutorial.

The following statements are now generally available (GA) with billing enabled:

• CREATE TABLE AS SELECT
• CREATE TABLE IF NOT EXISTS AS SELECT
• CREATE OR REPLACE TABLE AS SELECT
• INSERT INTO SELECT

These statements let you filter data from files in Amazon S3 and Azure Blob Storage before transferring results into BigQuery tables.

February 27, 2024

You can now use time series and range functions to support time series analysis. This feature is in preview.

You can now use data manipulation language (DML) statements to efficiently delete entire partitions. If a DELETE statement targets all rows in a partition, then the entire partition is deleted without scanning bytes or consuming slots. This feature is now generally available (GA).

February 26, 2024

A weekly digest of client library updates from across the Cloud SDK.

Java

Changes for google-cloud-bigquery

2.38.0 (2024-02-22)

Features

• Add MetadataCacheStatistics to Job QueryStatistics (#3133) (f3f387b)

Dependencies

• Update dependency com.google.apis:google-api-services-bigquery to v2-rev20240211-2.0.0 (#3152) (e5d6888)
• Update github/codeql-action action to v2.24.3 (#3148) (a0a7b01)
• Update github/codeql-action action to v2.24.3 (#3150) (042fcf0)
• Update github/codeql-action action to v2.24.4 (#3161) (531b1a0)

The BigQuery Data Transfer Service can now transfer data from the following data sources:

• Facebook Ads
• Oracle
• Salesforce
• Salesforce Marketing Cloud
• ServiceNow

Transfers from these data sources are supported in preview.

The following SQL features are now generally available (GA):

• GROUP BY GROUPING SETS clause: Produces aggregated data for one or more grouping sets.
• GROUP BY CUBE clause: Produces aggregated data for all grouping set permutations.
• GROUPING function: Checks if a groupable value in the GROUP BY clause is aggregated.

The GROUP BY ALL clause, which groups rows by inferring grouping keys from the SELECT items, is now in preview.

February 22, 2024

The following BigQuery text embedding features are now generally available (GA):

• Creating a BigQuery ML remote model that references a Vertex AI textembedding-gecko* text embedding model.
• Using the ML.GENERATE_EMBEDDING function with the remote model to embed text stored in BigQuery.
• Generating text embeddings with the NNLM, SWIVEL, and BERT TensorFlow models.

February 19, 2024

A weekly digest of client library updates from across the Cloud SDK.

Go

Changes for bigquery/storage/apiv1beta1

1.59.1 (2024-02-12)

Bug Fixes

• bigquery: Align return time.Time values to UTC (#9411) (4ac005d)

Java

Changes for google-cloud-bigquery

2.37.2 (2024-02-14)

Dependencies

• Update dependency com.google.api.grpc:proto-google-cloud-bigqueryconnection-v1 to v2.37.0 (#3132) (3a1efc2)
• Update dependency com.google.apis:google-api-services-bigquery to v2-rev20240203-2.0.0 (#3126) (5e28419)
• Update dependency com.google.cloud:google-cloud-datacatalog-bom to v1.41.0 (#3135) (9ab79ec)
• Update dependency com.google.cloud:google-cloud-shared-dependencies to v3.25.0 (#3140) (e61a7bc)
• Update github/codeql-action action to v2.24.1 (#3139) (4b3a429)

February 15, 2024

The following Generative AI features are now generally available (GA):

• Creating a remote model based on the gemini-pro Vertex AI large language model (LLM).
• Using the ML.GENERATE_TEXT function with a remote model based upon gemini-pro to perform generative natural language tasks on text stored in BigQuery tables.
• Use the BigQuery DataFrames GeminiTextGenerator class in the bigframes.ml.llm module to create estimator-like Gemini text generator models.

After you run a query in the query editor, in the Chart tab, you can now see a visualization of your query results. This feature is generally available (GA).

February 12, 2024

A weekly digest of client library updates from across the Cloud SDK.

Node.js

Changes for @google-cloud/bigquery

7.4.0 (2024-02-06)

Features

• Implementing mocha retries (#1295) (6cda9e1)
• Support JSON query param (#1329) (e9f133b)

Bug Fixes

• Prefer usage of projectId from the Dataset (#1326) (9e85219)

Go

Changes for bigquery/storage/apiv1beta1

1.59.0 (2024-02-06)

Features

• bigquery: Add ExportDataStatstics to QueryStatistics (#9371) (261c8d9)
• bigquery: Switch all timestamp representations to int64 usec (#9368) (8c1fb7d)

Bug Fixes

• bigquery/storage/managedwriter: Resolve data races (#9360) (fa31ec0)
• bigquery: Enable universe domain resolution options (fd1d569)
• bigquery: Support more timestamp formats for query param (#9236) (cc98509), refs #9221

Java

Changes for google-cloud-bigquery

2.37.1 (2024-02-06)

Features

• Add queryId to TableResult (#3106) (2156f02)
• Update universe domain exception error code/message (#3113) (5a82c85)

Dependencies

• Update actions/upload-artifact action to v4.3.1 (#3121) (3abdc70)
• Update dependency com.google.apis:google-api-services-bigquery to v2-rev20240124-2.0.0 (#3104) (6eff68e)
• Update dependency com.google.cloud:google-cloud-shared-dependencies to v3.24.0 (#3109) (5ad778c)
• Update dependency org.graalvm.buildtools:junit-platform-native to v0.10.0 (#3110) (3f8e8d1)
• Update dependency org.graalvm.buildtools:native-maven-plugin to v0.10.0 (#3111) (2858e96)
• Update dependency org.junit.vintage:junit-vintage-engine to v5.10.2 (#3119) (4b4fdd8)
• Update github/codeql-action action to v2.23.2 (#3102) (2cc545e)
• Update github/codeql-action action to v2.24.0 (#3114) (01f0405)

Python

Changes for google-cloud-bigquery

3.17.2 (2024-01-30)

Bug Fixes

• Change load_table_from_json autodetect logic (#1804) (6249032)

Documentation

• Update to use API (#1781) (81563b0)
• Update client_query_destination_table.py sample to use query_and_wait (#1783) (68ebbe1)
• Update query_external_sheets_permanent_table.py to use query_and_wait API (#1778) (a7be88a)
• Update sample for query_to_arrow to use query_and_wait API (#1776) (dbf10de)
• Update the query destination table legacy file to use query_and_wait API (#1775) (ef89f9e)
• Update to use query_and_wait in client_query_w_positional_params.py (#1786) (410f71e)
• Update to use query_and_wait in samples/client_query_w_timestamp_params.py (#1785) (ba36948)
• Update to_geodataframe to use query_and_wait functionality (#1800) (1298594)

February 08, 2024

Custom data masking is now generally available (GA). You can define custom masking routines for custom masking capabilities such as salt based hash. The feature is available on the Enterprise Plus edition.

BigQuery now offers entity resolution. This feature lets users match records across datasets even when a common identifier is missing. It utilizes an identity provider for this process; BigQuery supports LiveRamp and provides a framework for other identity providers to offer similar services. This feature is generally available (GA).

February 07, 2024

You can now view query plans to see details of SQL pushdowns in federated queries. This feature is now generally available.

February 06, 2024

Billing for Spark stored procedures begins on March 12, 2024. Until that date, Spark stored procedures are offered at no extra cost.

January 31, 2024

The following information schema views display the history of configuration changes to the options of your organization and projects:

• ORGANIZATION_OPTIONS_CHANGES view displays the configuration changes to an organization, including all organization and project-level changes.
• PROJECT_OPTIONS_CHANGES view displays the configuration changes to a project.

This feature is now in preview.

BigQuery now supports vector search and vector indexes. These features are in preview.

You can use the VECTOR_SEARCH function to search embeddings in order to identify semantically similar entities.

You can use vector indexes to make VECTOR_SEARCH more efficient, with the trade-off of returning more approximate results.

Try the new vector search and vector index capabilities with the Search embeddings with vector search tutorial.

January 29, 2024

A weekly digest of client library updates from across the Cloud SDK.

Java

Changes for google-cloud-bigquery

2.37.0 (2024-01-25)

Features

• Add support for Table resource tags (#3046) (7d61111)
• Add universe domain (#3090) (b2814a2)

Dependencies

• Update actions/upload-artifact action to v4.1.0 (#3071) (3fbb2bb)
• Update actions/upload-artifact action to v4.2.0 (#3081) (af81354)
• Update actions/upload-artifact action to v4.3.0 (#3091) (f4411b0)
• Update arrow.version to v15 (#3084) (4d4cbae)
• Update dependency com.google.api.grpc:proto-google-cloud-bigqueryconnection-v1 to v2.35.0 (#3066) (48cdaa8)
• Update dependency com.google.api.grpc:proto-google-cloud-bigqueryconnection-v1 to v2.36.0 (#3093) (24456a3)
• Update dependency com.google.apis:google-api-services-bigquery to v2-rev20240105-2.0.0 (#3073) (f371d67)
• Update dependency com.google.cloud:google-cloud-datacatalog-bom to v1.39.0 (#3067) (6ff4f04)
• Update dependency com.google.cloud:google-cloud-datacatalog-bom to v1.40.0 (#3094) (110bcc5)
• Update dependency com.google.cloud:google-cloud-shared-dependencies to v3.22.0 (#3080) (a5b119c)
• Update dependency com.google.cloud:google-cloud-shared-dependencies to v3.23.0 (#3096) (0933b34)
• Update dependency com.google.oauth-client:google-oauth-client-java6 to v1.35.0 (#3078) (2614df2)
• Update dependency com.google.oauth-client:google-oauth-client-jetty to v1.35.0 (#3079) (f03c4fc)
• Update github/codeql-action action to v2.23.0 (#3061) (0fbdfba)
• Update github/codeql-action action to v2.23.1 (#3077) (e3f417c)

Python

Changes for google-cloud-bigquery

3.17.1 (2024-01-24)

Bug Fixes

• Add pyarrow.large_strign to the _ARROW_SCALAR_IDS_TO_BQ map (#1796) (b402a6d)
• Retry 'job exceeded rate limits' for DDL queries (#1794) (39f33b2)

3.17.0 (2024-01-24)

Features

• Support universe resolution (#1774) (0b5c1d5)

Bug Fixes

• query_and_wait now retains unknown query configuration _properties (#1793) (4ba4342)
• Raise ValueError in query_and_wait with wrong job_config type (4ba4342)

Documentation

• Remove unused query code sample (#1769) (1f96439)
• Update snippets.py to use query_and_wait (#1773) (d90602d)
• Update multiple samples to change query to query_and_wait (#1784) (d1161dd)
• Update the query with no cache sample to use query_and_wait API (#1770) (955a4cd)
• Updates query to query and wait in samples/desktopapp/user_credentials.py (#1787) (89f1299)

You can now use tags on BigQuery tables to conditionally grant or deny access with Identity and Access Management (IAM) policies. This feature is in preview.

Cloud console updates: You can now sort query results by column. Click Open sort menu next to the column name and select a sort order. This feature is generally available (GA).

January 24, 2024

BigQuery now natively supports the Delta Lake format for Amazon S3 and Azure tables. This feature is now in preview.

BigQuery ML has added a new residual column to the output of the ML.EXPLAIN_FORECAST function for ARIMA_PLUS and ARIMA_PLUS_XREG models. The residual column contains the difference between the actual time series and the fitted time series for the historical data. This lets you compare the modeled historical data that is returned in the other output columns of ML.EXPLAIN_FORECAST with the actual historical data.

To improve BigQuery ML training performance, the APPROX_GLOBAL_FEATURE_CONTRIB argument now defaults to TRUE when you set the ENABLE_GLOBAL_EXPLAIN argument to TRUE, and you set the NUM_PARALLEL_TREE argument to greater than 10 for boosted tree models or greater than 50 for random forest models.

BigQuery now supports the ST_LINEINTERPOLATEPOINT geography function, which gets a point at a specific fraction in a linestring. This feature is generally available (GA).

January 22, 2024

A weekly digest of client library updates from across the Cloud SDK.

Go

Changes for bigquery/storage/apiv1beta1

1.58.0 (2024-01-18)

Features

• bigquery/storage/managedwriter: Improve schema comparison stability (#9241) (faccb68)
• bigquery/storage: Add ability to request compressed ReadRowsResponse rows (c3f1174)
• bigquery: Add DataGovernanceType to routines (#8990) (57491ae)
• bigquery: Add schema support for RANGE type (#9050) (477ccee)
• bigquery: Add support for AllowNonIncrementalDefinition and Staleness on MaterializedView (#8673) (6ec2bb2)
• bigquery: Add table resource tags support (#9084) (3569cc2)
• bigquery: Expose query id on row iterator if available (#9224) (bbff8ac)

Documentation

• bigquery/storage/managedwriter: Add documentation for ingesting JSON to a ManagedStream (#9137) (cae54ed)
• bigquery: Update package docs to show query parameterization (#8965) (cc76ab3)

BigQuery is now available in the Berlin (europe-west10) region.

January 16, 2024

You can now use cross-cloud joins to run queries that span both Google Cloud and BigQuery Omni regions. This feature is in preview.

January 15, 2024

A weekly digest of client library updates from across the Cloud SDK.

Java

Changes for google-cloud-bigquery

2.36.0 (2024-01-10)

Features

• Support RANGE schema (#3043) (febfc1f)
• Use location in BigQueryOption as the default for query (#3047) (270f866)

Bug Fixes

• bigquery.create NullPointerException when job already exists (#3035) (38191b1)

Dependencies

• Update actions/upload-artifact action to v4 (#3055) (7d76100)
• Update arrow.version to v14.0.2 (#3050) (b0dc33a)
• Update dependency com.google.api.grpc:proto-google-cloud-bigqueryconnection-v1 to v2.34.0 (#3033) (a710632)
• Update dependency com.google.cloud:google-cloud-shared-dependencies to v3.21.0 (#3060) (78995c4)
• Update github/codeql-action action to v2.22.11 (#3002) (52d5e97)

Python

Changes for google-cloud-bigquery

3.16.0 (2024-01-12)

Features

• Add table_constraints field to Table model (#1755) (a167f9a)
• Support jsonExtension in LoadJobConfig (#1751) (0fd7347)

Bug Fixes

• Add detailed message in job error (#1762) (08483fb)

3.15.0 (2024-01-09)

Features

• Support JSON type in insert_rows and as a scalar query parameter (#1757) (02a7d12)
• Support RANGE in schema (#1746) (8585747)

Bug Fixes

• Deserializing JSON subfields within structs fails (#1742) (0d93073)
• Due to upstream change in dataset, updates expected results (#1761) (132c14b)
• Load_table_from_dataframe for higher scale decimal (#1703) (b9c8be0)
• Updates types-protobuf version for mypy-samples nox session (#1764) (c0de695)

Performance Improvements

• DB-API uses more efficient query_and_wait when no job ID is provided (#1747) (d225a94)

January 02, 2024

Analytics Hub listings can now include data encrypted with customer-managed encryption keys (CMEK). This feature is generally available (GA).

December 18, 2023

A weekly digest of client library updates from across the Cloud SDK.

Python

Changes for google-cloud-bigquery

3.14.1 (2023-12-13)

Bug Fixes

• Add missing handler for deserializing json value (#1587) (09017a9)

3.14.0 (2023-12-08)

Features

• Add Client.query_and_wait which directly returns a RowIterator of results (#1722) (89a647e)
• Add job_id, location, project, and query_id properties on RowIterator (#1733) (494f275)
• Add job_timeout_ms to job configuration classes (#1675) (84d64cd)
• Add support dataset.max_time_travel_hours (#1683) (f22eff2)
• Add support for Dataset.isCaseInsensitive (#1671) (386fa86)
• Add support for Python 3.12 (#1736) (3c0976a)
• Removed pkg_resources from all test files and moved importlib into pandas extra (#1726) (1f4ebb1)
• Support data_governance_type (#1708) (eff365d)

Bug Fixes

• load_table_from_dataframe now assumes there may be local null values (#1735) (f05dc69)
• Ensure query job retry has longer deadline than API request deadline (#1734) (5573579)
• Keep RowIterator.total_rows populated after iteration (#1748) (8482f47)
• Move grpc, proto-plus and protobuf packages to extras (#1721) (5ce4d13)

Performance Improvements

• Use the first page a results when query(api_method="QUERY") (#1723) (6290517)

Operational Health administrative resource charts are now in preview. You can use charts to view slot and shuffle usage, job concurrency, errors, and other metrics.

December 14, 2023

The BigQuery Data Transfer Service now supports federated workforce identities when creating a data transfer from most data sources. This feature is generally available (GA).

The Apache Hive connector is now generally available (GA) for data analytics pipeline migration.

December 12, 2023

The following BigQuery cross-cloud features are now in preview:

• You can now take advantage of the benefits of materialized views over Amazon S3 metadata cache-enabled BigLake tables.
• You can create materialized view replicas of materialized views over Amazon S3 metadata cache-enabled BigLake tables. Materialized view replicas let you use the materialized view data in queries while avoiding data egress costs and improving query performance.

December 11, 2023

A weekly digest of client library updates from across the Cloud SDK.

Java

Changes for google-cloud-bigquery

2.35.0 (2023-12-01)

Features

• Add InputBytes to extract job statistics (#2998) (19b7c3a)
• Add Routine DataGovernanceType (#3006) (ecb567b)

Bug Fixes

• Update TableInsertRows.java (#2999) (ff4a086)

Dependencies

• Update actions/github-script action to v7 (#3001) (d1bdeab)
• Update actions/setup-java action to v4 (#3018) (14ed571)
• Update arrow.version to v14 (#3023) (759fd64)
• Update dependency com.google.apis:google-api-services-bigquery to v2-rev20231111-2.0.0 (#3020) (ef48002)
• Update dependency com.google.cloud:google-cloud-shared-dependencies to v3.20.0 (#3019) (0293edb)

December 07, 2023

The following BigQuery ML data preprocessing features are now in preview:

• The ML.TRANSFORM function, which you can use to preprocess feature data. This function processes input data by applying the data transformations captured in the TRANSFORM clause of an existing model.
• Transform-only models which you can use to apply preprocessing functions to input data and return the preprocessed data. Transform-only models decouple data preprocessing from model training, making it easier for you to capture and reuse a set of data preprocessing rules.

December 04, 2023

A weekly digest of client library updates from across the Cloud SDK.

Python

Changes for google-cloud-bigquery

3.14.0rc0 (2023-11-30)

Features

• Add job_id, location, project, and query_id properties on RowIterator (#1733) (494f275)
• Add job_timeout_ms to job configuration classes (#1675) (84d64cd)
• Add support dataset.max_time_travel_hours (#1683) (f22eff2)
• Add support for Dataset.isCaseInsensitive (#1671) (386fa86)
• Removed pkg_resources from all test files and moved importlib into pandas extra (#1726) (1f4ebb1)
• Support data_governance_type (#1708) (eff365d)

Bug Fixes

• load_table_from_dataframe now assumes there may be local null values (#1735) (f05dc69)
• Ensure query job retry has longer deadline than API request deadline (#1734) (5573579)
• Move grpc, proto-plus and protobuf packages to extras (#1721) (5ce4d13)

Performance Improvements

• Use the first page a results when query(api_method="QUERY") (#1723) (6290517)

November 30, 2023

You can use configuration YAML files to transform SQL code when you translate SQL queries from your source database. Configuration YAML files can be used with the batch SQL translator, the interactive SQL translator, and the batch translation Python client. This feature is now generally available (GA).

The slot estimator now supports project level cost-optimal commitment and autoscale recommendations for on-demand workloads. This feature is now in preview.

November 16, 2023

The following BigQuery ML features for Vertex AI large language models (LLMs) are now generally available (GA):

• The SQL syntax for remote models has been updated to provide access to all text generation and text embedding LLMs (for example, text-bison-32k and textembedding-gecko-multilingual) and also to provide support for different LLM versions.

• Region support for text-bison* LLM models has been expanded to include the following locations in addition to us and us-central1:

  • asia-northeast3
  • asia-southeast1
  • eu
  • europe-west1
  • europe-west2
  • europe-west3
  • europe-west4
  • europe-west9
  • us-west4

November 14, 2023

You can now see query performance insights about partition skew. This feature is in preview.

November 13, 2023

A weekly digest of client library updates from across the Cloud SDK.

Java

Changes for google-cloud-bigquery

2.34.2 (2023-11-07)

Dependencies

• Update dependency com.google.api.grpc:proto-google-cloud-bigqueryconnection-v1 to v2.32.0 (#2989) (47a61a7)
• Update dependency com.google.cloud:google-cloud-datacatalog-bom to v1.36.0 (#2990) (81c0727)

2.34.1 (2023-11-06)

Dependencies

• Update dependency com.google.cloud:google-cloud-shared-dependencies to v3.19.0 (#2986) (0d400da)
• Update dependency org.checkerframework:checker-compat-qual to v2.5.6 (#2982) (c137f1f)
• Update dependency org.junit.vintage:junit-vintage-engine to v5.10.1 (#2984) (a64b91c)
• Update github/codeql-action action to v2.22.5 (#2975) (0b88846)

The following BigQuery ML point-in-time lookup functions are now generally available (GA). These functions let you specify a point-in-time cutoff when retrieving features for training a model or running inference, in order to avoid data leakage.

• Use the ML.FEATURES_AT_TIME function to use the same point-in-time cutoff for all entities when retrieving features.
• Use the ML.ENTITY_FEATURES_AT_TIME function to retrieve features from multiple points in time for multiple entities.

The following AI features in BigQuery are now in preview:

• The ability to process documents from BigQuery object tables by doing the following:

  • Creating a remote model based on the Document AI API, including specifying a document processor to use.
  • Using the ML.PROCESS_DOCUMENT function with a Document AI-based remote model to process the documents.
    
    Try this feature with the Process documents with the ML.PROCESS_DOCUMENT function how-to.
    
    

• The ability to transcribe audio files from BigQuery object tables by doing the following:

  • Creating a remote model based on the Speech-to-Text API, including specifying a speech recognizer to use.
  • Using the ML.TRANSCRIBE function with a Speech-to-Text-based remote model to transcribe the audio files.
    
    Try this feature with the Transcribe audio files with the ML.TRANSCRIBE function how-to.

November 07, 2023

The batch SQL translator has added enhancements when viewing SQL translation reports. You can now see a log summary of all issues during a translation job, as well as a code tab that displays a side-by-side comparison of your input and output files from a translation. This feature is in preview.

November 06, 2023

A weekly digest of client library updates from across the Cloud SDK.

Go

Changes for bigquery/storage/apiv1beta1

1.57.1 (2023-11-01)

Bug Fixes

• bigquery: Correct deps issue (#8958) (abd6ca7)

1.57.0 (2023-10-30)

Features

• bigquery/biglake: Promote to GA (e864fbc)
• bigquery/storage/managedwriter: Support default value controls (#8686) (dfa8e22)
• bigquery: Expose Apache Arrow data through ArrowIterator (#8506) (c8e7692), refs #8100
• bigquery: Introduce query preview features (#8653) (f29683b)

Bug Fixes

• bigquery: Handle storage read api Recv call errors (#8666) (c73963f)
• bigquery: Update golang.org/x/net to v0.17.0 (174da47)
• bigquery: Update grpc-go to v1.56.3 (343cea8)
• bigquery: Update grpc-go to v1.59.0 (81a97b0)

Python

Changes for google-cloud-bigquery

3.13.0 (2023-10-30)

Features

• Add Model.transform_columns property (#1661) (5ceed05)
• Add support for dataset.default_rounding_mode (#1688) (83bc768)

Bug Fixes

• AccessEntry API representation parsing (#1682) (a40d7ae)

Documentation

• Remove redundant bigquery_update_table_expiration code sample (#1673) (2dded33)
• Revised create_partitioned_table sample (#1447) (40ba859)
• Revised relax column mode sample (#1467) (b8c9276)

The BigQuery Data Transfer Service can now transfer campaign reporting and configuration data from Display & Video 360 into BigQuery. This feature is in preview.

The following BigQuery ML features for time series forecasting are now generally available (GA):

• Ensure forecasted values fall within specified limits. The FORECAST_LIMIT_LOWER_BOUND and FORECAST_LIMIT_UPPER_BOUND options of the CREATE MODEL statement let you set the lower and upper bounds of the forecasted values returned by the model.

  Try this feature with the Limit forecasted values for a time series model tutorial.

• Custom holiday modeling:

  • CREATE MODEL syntax lets you specify custom holiday modeling for time series models.
  • The ML.HOLIDAY_INFO function returns the list of holidays being modeled by an ARIMA_PLUS or ARIMA_PLUS_XREG time series forecasting model.
  • The updated ML.EXPLAIN_FORECAST function includes an explanation of the holiday effect for each holiday included in the model.

  Try this feature with the Use custom holidays in a time-series forecasting model tutorial.

November 02, 2023

BigQuery now supports text analysis configuration options for the following:

• CREATE SEARCH INDEX DDL
• Existing LOG_ANALYZER and new PATTERN_ANALYZER analyzers, which are used in various functions, including SEARCH
• New TEXT_ANALYZE function

BigQuery now also provides support for the following advanced processing functions:

• ML.BAG_OF_WORDS
• ML.TF_IDF
• BAG_OF_WORDS
• TF_IDF
• COSINE_DISTANCE
• EUCLIDEAN_DISTANCE
• EDIT_DISTANCE

These features are now in preview.

November 01, 2023

The following INFORMATION_SCHEMA views that show metadata for table storage usage are now in preview:

• Use the TABLE_STORAGE_USAGE_TIMELINE view to get total billable bytes per table per day at the project level.
• Use the TABLE_STORAGE_USAGE_TIMELINE_BY_ORGANIZATION view to get total billable bytes per table per day at the organization level.

You can now use cached results from the same query issued by other users in the same project when you use Enterprise or Enterprise Plus edition. This feature is generally available (GA).

October 31, 2023

BigQuery support for change data capture (CDC) by processing and applying streamed changes in real-time to existing data using the BigQuery Storage Write API is now generally available (GA).

You can now use data manipulation language (DML) to modify rows that have been recently written by the Storage Write API. This feature is in preview.

October 30, 2023

The BigQuery Data Transfer Service can now transfer data from Azure Blob Storage into BigQuery. This feature is now generally available (GA).

A weekly digest of client library updates from across the Cloud SDK.

Java

Changes for google-cloud-bigquery

2.34.0 (2023-10-26)

Features

• Add BigLakeConfiguration Property in StandardTableDefinition.java (#2916) (1d660fa)
• Add support for Dataset property storageBillingModel (#2913) (f452cf4)
• Add support for preview features (#2923) (113b8f2)

Dependencies

• Update actions/checkout action to v4.1.1 (#2950) (c556c18)
• Update dependency com.google.api.grpc:proto-google-cloud-bigqueryconnection-v1 to v2.30.0 (#2942) (e760fca)
• Update dependency com.google.api.grpc:proto-google-cloud-bigqueryconnection-v1 to v2.31.0 (#2967) (7ed55b5)
• Update dependency com.google.apis:google-api-services-bigquery to v2-rev20231008-2.0.0 (#2946) (3d0da5b)
• Update dependency com.google.cloud:google-cloud-datacatalog-bom to v1.34.0 (#2943) (18162c3)
• Update dependency com.google.cloud:google-cloud-datacatalog-bom to v1.35.0 (#2968) (219db2c)
• Update dependency com.google.cloud:google-cloud-shared-dependencies to v3.18.0 (#2955) (1ee18eb)
• Update dependency org.graalvm.buildtools:junit-platform-native to v0.9.28 (#2956) (b03effd)
• Update dependency org.graalvm.buildtools:native-maven-plugin to v0.9.28 (#2957) (6465e41)
• Update github/codeql-action action to v2.22.2 (#2944) (f584e59)
• Update github/codeql-action action to v2.22.3 (#2954) (1b2bc18)
• Update github/codeql-action action to v2.22.4 (#2958) (de9bcee)
• Update ossf/scorecard-action action to v2.3.1 (#2960) (855e698)

The administrative resource charts now supports the following features in preview:

• View your resource utilization chart at the project level.
• Filter your resource utilization data based on different billing models.

October 23, 2023

Custom data masking now supports an expanded list of functions, including SHA hash functions with salt. This feature is in preview.

October 19, 2023

Stored procedures for Apache Spark are now available without enrollment. This feature is in preview.

October 18, 2023

The BigQuery migration assessment is now available for Apache Hive in preview. You can use this feature to assess the complexity of migrating data from your Apache Hive data warehouse to BigQuery.

October 16, 2023

A weekly digest of client library updates from across the Cloud SDK.

Java

Changes for google-cloud-bigquery

2.33.2 (2023-10-11)

Bug Fixes

• GetDouble in read API path (#2919) (436ee8e)

Dependencies

• Update dependency com.google.api.grpc:proto-google-cloud-bigqueryconnection-v1 to v2.29.0 (#2911) (052f5c2)
• Update dependency com.google.apis:google-api-services-bigquery to v2-rev20230925-2.0.0 (#2921) (f0fb64f)
• Update dependency com.google.cloud:google-cloud-datacatalog-bom to v1.33.0 (#2912) (e053494)
• Update dependency com.google.cloud:google-cloud-shared-dependencies to v3.17.0 (#2931) (25a94f1)
• Update github/codeql-action action to v2.22.0 (#2926) (33ce4ae)
• Update github/codeql-action action to v2.22.1 (#2934) (7ae7b99)
• Update ossf/scorecard-action action to v2.3.0 (#2927) (93bfd8e)

You can now use DLP functions to support encryption and decryption between BigQuery and DLP, using AES-SIV. This feature is in preview.

October 12, 2023

The following geography functions are now generally available (GA):

• ST_LINESUBSTRING: Gets a segment of a single linestring at a specific starting and ending fraction.
• ST_HAUSDORFFDISTANCE: Gets the discrete Hausdorff distance between two geometries.

October 09, 2023

A weekly digest of client library updates from across the Cloud SDK.

Go

Changes for bigquery/storage/apiv1beta1

1.56.0 (2023-10-05)

Features

• bigquery/analyticshub: Add Subscription resource and RPCs (#8612) (9992249)
• bigquery: Add external dataset reference (#8545) (1001acf)
• bigquery: Add media options to LoadConfig (#8640) (62baf56)

Bug Fixes

• bigquery/storage/managedwriter: Automatic retry for multiplex test (#8601) (6ef1945)
• bigquery: Dependency detection on proto conversion (#8566) (763ab5d)

Documentation

• bigquery/datatransfer: Update transferConfig.name description to indicate that it supports both formats (0449518)

Python

Changes for google-cloud-bigquery

3.12.0 (2023-10-02)

Features

• Add Dataset.storage_billing_model setter, use client.update_dataset(ds, fields=["storage_billing_model"]) to update (#1643) (5deba50)
• Search statistics (#1616) (b930e46)
• Widen retry predicate to include ServiceUnavailable (#1641) (3e021a4)

Bug Fixes

• Allow storage_billing_model to be explicitly set to None to use project default value (#1665) (514d3e1)
• Relax timeout expectations (#1645) (1760e94)
• Use isinstance() per E721, unpin flake8 (#1659) (54a7769)

Documentation

• Revise update_table_expiration sample (#1457) (03194e0)

Queries now support additional ways to work with grouping sets, which include:

• GROUP BY GROUPING SETS clause (new): Produce aggregated data for one or more grouping sets.
• GROUP BY CUBE clause (new): Produce aggregated data for all grouping set permutations.
• GROUP BY ROLLUP clause (update): You can now include groupable items sets in this clause.
• GROUPING function (new): Check if a groupable value in the GROUP BY clause is aggregated.

This feature is in preview.

Adding descriptions to the columns of a view is now generally available (GA). Use the CREATE VIEW or ALTER COLUMN DDL statements to add descriptions.

BigQuery is now available in the Dammam (me-central2) region.

BigQuery ML is now available in the Dammam (me-central2) region.

BigQuery Data Transfer Service is now available in the Dammam (me-central2) region.

October 05, 2023

The BigQuery migration assessment is now available for Snowflake in preview. You can use this feature to assess the complexity of migrating data from your Snowflake data warehouse to BigQuery.

October 04, 2023

You can now copy tables across regions. This feature is now in preview.

October 03, 2023

The following Google Cloud Blockchain Analytics datasets are now available in Preview and available through the Public Datasets Program and Analytics Hub:

• Google Cloud's Tron Mainnet data

• Google Cloud's Optimism Mainnet data

• Google Cloud's Avalanche Contract Chain data

• Google Cloud's Fantom Opera data

• Google Cloud's Ethereum Mainnet data

• Google Cloud's Arbitrum One Chain data

• Google Cloud's Cronos Mainnet Chain data

October 02, 2023

A weekly digest of client library updates from across the Cloud SDK.

Node.js

Changes for @google-cloud/bigquery

7.3.0 (2023-09-28)

Features

• Re-exporting types for google-common (#1271) (cdf960a)

Bug Fixes

• Avoid TypeError if resp is undefined (#1273) (ff51c1d)
• Updated types from API discovery doc (#1284) (1d8a2b7)
• Updating type for test blocking dependency updates (#1282) (1dbe0fe)

Java

Changes for google-cloud-bigquery

2.33.1 (2023-09-28)

Bug Fixes

• Dry run NPE when there is no query parameters (#2899) (8f85a4d)

2.33.0 (2023-09-27)

Features

• Add support for FileSetSpec (#2888) (3895bd9)

Bug Fixes

• Update samples snippet to write to BYTES instead of ARRAY<BYTES> (#2876) (7e040e9)

Dependencies

• Update actions/checkout action (#2893) (e3655af)
• Update dependency com.google.cloud:google-cloud-shared-dependencies to v3.16.1 (#2892) (e1d9871)
• Update dependency org.graalvm.buildtools:junit-platform-native to v0.9.27 (#2885) (2237ca2)
• Update dependency org.graalvm.buildtools:native-maven-plugin to v0.9.27 (#2886) (539b4e6)
• Update github/codeql-action action to v2.21.4 (#2829) (599e3b3)
• Update github/codeql-action action to v2.21.8 - abandoned (#2897) (ab4e1d0)
• Update github/codeql-action action to v2.21.8 (#2889) (b568026)
• Update github/codeql-action action to v2.21.9 (#2901) (33a729f)

BigQuery native integration in Looker Studio enables monitoring features for Looker Studio queries, improves query performance, and supports many BigQuery features. This feature is in preview.

September 29, 2023

As a BigQuery administrator, to monitor your organization's slots utilization and BigQuery jobs' performance over time, use can now use administrative query inspector. This feature is now generally available.

September 28, 2023

The following BigQuery ML point-in-time lookup functions are now in preview. These functions let you specify a point-in-time cutoff when retrieving features for training a model or running inference, in order to avoid data leakage.

• Use the ML.FEATURES_AT_TIME function to use the same point-in-time cutoff for all entities when retrieving features.
• Use the ML.ENTITY_FEATURES_AT_TIME function to retrieve features from multiple points in time for multiple entities.

You can now use IAM conditions to control access to BigQuery resources. This feature is in preview.

September 25, 2023

Materialized views over BigLake metadata cache-enabled tables can reference structured data stored in Cloud Storage. These materialized views function like materialized views over BigQuery-managed storage tables, including the benefits of automatic refresh and smart tuning. This feature is now generally available (GA).

Authorized stored procedures are now generally available (GA). This feature lets you share stored procedures with users or groups without giving them direct access to the underlying tables.

Support for Google AdWords is now deprecated by the BigQuery Data Transfer Service. For information about transfers from Google Ads, see Google Ads transfers.

September 20, 2023

The maximum number of rows for results returned in Connected Sheets has increased as follows:

• Pivot tables increased from 30,000 to 50,000 rows
• Data extracts increased from 25,000 to 50,000 rows

September 18, 2023

A weekly digest of client library updates from across the Cloud SDK.

Go

Changes for bigquery/storage/apiv1beta1

1.55.0 (2023-09-11)

Features

• bigquery/datapolicies: Support using custom UDF in the data policy (20725c8)
• bigquery: Set PreferredMinStreamCount when creating read session (#8476) (22e095a), refs #8432

Bug Fixes

• bigquery/storage/managedwriter: Address possible deadlocks (#8507) (48b08bf)
• bigquery: Field descriptor proto name should not be lowercase (#8495) (4287e4b)
• bigquery: Value for datasetID on foreign keys (#8447) (fa6e827), refs #8442

Java

Changes for google-cloud-bigquery

2.32.0 (2023-09-14)

Features

• Add support for converting interval fields to threeten PeriodDuration (#2838) (2294c2f)
• Add support for ExternalDatasetReference (#2871) (bbb86fd)

Dependencies

• Update actions/checkout action to v4 (#2862) (902e9b9)
• Update actions/upload-artifact action to v3.1.3 (#2867) (cbbf0fb)
• Update dependency com.google.api.grpc:proto-google-cloud-bigqueryconnection-v1 to v2.26.0 (#2873) (6196625)
• Update dependency com.google.cloud:google-cloud-datacatalog-bom to v1.30.0 (#2874) (6cafedf)
• Update dependency com.google.cloud:google-cloud-shared-dependencies to v3.15.0 (#2870) (f24439b)
• Update dependency org.graalvm.buildtools:junit-platform-native to v0.9.26 (#2868) (d01031c)
• Update dependency org.graalvm.buildtools:native-maven-plugin to v0.9.26 (#2869) (edd7141)

The BigQuery Data Transfer Service now supports transfers from Search Ads 360 using the new Search Ads 360 reporting API. This feature is in preview. Customers with existing Search Ads 360 transfers should migrate their workflows to be compatible with the new Search Ads 360. The BigQuery Data Transfer Service will stop its support for the old Search Ads 360 reporting API on May 31st, 2024.

September 13, 2023

You can now create a federated dataset in BigQuery that federates to an existing database in AWS Glue. This feature is in preview.

September 12, 2023

The array subscript operator now returns a value in an array directly by index. Previously, only offset and ordinal were available. This feature is generally available (GA).

The struct subscript operator has been added. With this operator, you can access a STRUCT field by index, offset, or ordinal. This feature is generally available (GA).

September 11, 2023

A weekly digest of client library updates from across the Cloud SDK.

Java

Changes for google-cloud-bigquery

2.31.2 (2023-09-05)

Bug Fixes

• Hide TableReference data struct (#2855) (2cbded6)
• SearchStats IndexUnusedReasons null bug (#2825) (309ea60)

August 30, 2023

You can now use EXPORT DATA statements to directly export BigQuery data to Bigtable. This feature is in preview.

August 29, 2023

Data clean rooms is now in preview. Data clean rooms provide a secure environment in which multiple parties can share, join, and analyze their data assets without moving or revealing the underlying data. To learn more, see the following topics:

• Use data clean rooms
• Aggregation threshold for queries and views
• Aggregation threshold clause

Duet AI in BigQuery, an AI-powered collaborator in Google Cloud, can help you complete, generate, and explain SQL queries. This feature is in preview.

BigQuery Studio is now in preview. BigQuery Studio offers features to make it easier for you to discover, explore, analyze, and run inference on data in BigQuery, including:

• Python notebooks, powered by Colab Enterprise. Notebooks provide one-click Python development runtimes, and built-in support for BigQuery DataFrames.
• Asset management and version history for notebooks and saved queries, powered by Dataform.

BigQuery DataFrames is now in preview. BigQuery DataFrames is a Python API that you can use to analyze data and perform machine learning tasks in BigQuery. BigQuery DataFrames consists of the following parts:

• bigframes.pandas implements a DataFrame API (with partial Pandas compatibility) on top of BigQuery.
• bigframes.ml implements a Python API for BigQuery ML (with partial scikit-learn compatibility).

Get started with BigQuery DataFrames by using the BigQuery DataFrames quickstart.

The following Generative AI features are now generally available (GA) in BigQuery ML:

• Creating a remote model based on the Vertex AI large language model (LLM) text-bison.
• Using the ML.GENERATE_TEXT function with an LLM-based remote model to perform generative natural language tasks on text stored in BigQuery tables.

Try these features with the Generate text by using a remote model and the ML.GENERATE_TEXT function tutorial.

August 24, 2023

The following text embedding features are now available in preview:

• Creating a BigQuery ML remote model that references the Vertex AI PaLM APIs for embeddings (textembedding-gecko).
• Using the ML.GENERATE_TEXT_EMBEDDING function with the remote embedding model to embed text stored in BigQuery.
• Using the ARRAY<NUMERIC> type as an input feature type to other models.
• Generating text embeddings with the NNLM, SWIVEL, and BERT TensorFlow models.

For more information, see the tutorial for performing basic semantic search with text embeddings.

The following BigQuery ML inference features are now generally available (GA):

• Importing ONNX, XGBoost, and TensorFlow Lite models so that you can run them within the BigQuery ML inference engine.
• Hosting models remotely on Vertex AI Prediction so you can do inference with BigQuery ML, removing the need to build data pipelines manually.
• Using BigQuery ML functions to perform inference on Vertex AI pretrained models so that you can accomplish natural language processing, translation, and computer vision tasks in BigQuery. These functions work with the Cloud Vision, Cloud Natural Language, and Cloud Translation APIs.

August 22, 2023

BigQuery now allows you to create your own masking routines for your data. You can use the REGEX_REPLACE scalar function to create custom masking rules to obfuscate your sensitive data. This feature is currently in preview.

August 21, 2023

You can now scan tables to create data profiles and monitor data quality. These features help you better understand your data and ensure it is accurate and reliable. These features are generally available (GA).

A weekly digest of client library updates from across the Cloud SDK.

Node.js

Changes for @google-cloud/bigquery

7.2.0 (2023-08-17)

Features

• Partial metadata projection (#1258) (b70d3a4)

Bug Fixes

• deps: Update dependency @google-cloud/common to v5 (#1253) (0cbfdcd)
• deps: Update dependency @google-cloud/paginator to v5 (#1254) (658ff1a)
• deps: Update dependency @google-cloud/precise-date to v4 (#1261) (7f12f7d)
• deps: Update dependency @google-cloud/promisify to v4 (#1252) (6f04f39)
• Manual update for renovate-bot google cloud storage v7 (#1262) (b58fc29)

Go

Changes for bigquery/storage/apiv1beta1

1.54.0 (2023-08-17)

Features

• bigquery/biglake: Start generating apiv1 and apiv1alpha1 (#8433) (916f779)
• bigquery/datatransfer: Add EncryptionConfiguration to TransferConfig (e3f8c89)
• bigquery/storage/managedwriter: Refine connection metrics (#8324) (54fcf36)
• bigquery/storage: Add beta2 deprecation message (b3dbdde)
• bigquery/storage: Add default_missing_value_interpretation field; indicate KMS_SERVICE_ERROR is retryable (b3dbdde)
• bigquery: Add ProjectID to JobIDConfig (#8405) (7fafd80)

Bug Fixes

• bigquery/datapolicies: Sync the new PredefinedEpxression types to the client library (fcb41cc)
• bigquery: Use JobFromProject for storage iterator (#8372) (bfbf18c)

Documentation

• bigquery/storage/managedwriter: Augment context usage in package docs (#8334) (b30a7f4)
• bigquery: Improve RowIterator docs and out of process pagination with Storage API (#8419) (40f8f80)

Analytics Hub now lets you manage subscriptions. The subscription resource stores relevant information about the subscriber and represents the connection between the shared resource and linked dataset. This feature is now in preview.

August 17, 2023

You can now replicate a dataset from the source region to one or more other regions with cross-region dataset replication. This feature is now in preview.

August 14, 2023

A weekly digest of client library updates from across the Cloud SDK.

Java

Changes for google-cloud-bigquery

2.31.1 (2023-08-09)

Dependencies

• Update dependency com.google.api.grpc:proto-google-cloud-bigqueryconnection-v1 to v2.25.0 (#2845) (d940f8d)
• Update dependency com.google.cloud:google-cloud-datacatalog-bom to v1.29.0 (#2846) (87a0a10)
• Update dependency com.google.cloud:google-cloud-shared-dependencies to v3.14.0 (#2834) (79fe14c)
• Update dependency org.graalvm.buildtools:junit-platform-native to v0.9.24 (#2839) (ae752bc)
• Update dependency org.graalvm.buildtools:native-maven-plugin to v0.9.24 (#2840) (1ae6cb9)
• Update github/codeql-action action to v2.21.1 (#2824) (9978971)
• Update jmh.version to v1.37 (#2836) (4b3a3c2)

Starting September 15, 2023, prices will apply for network egress from a BigQuery Google Cloud region to another Google Cloud region on the same continent and between different continents. For more information, see BigQuery Network Egress Traffic Pricing Charges Announcement.

August 10, 2023

You can now see query performance insights about high cardinality joins. This feature is generally available (GA).

The September 14, 2022 release notes announced that you could configure the connector to authenticate the connection using an external account with workload identity federation for ODBC driver update release 2.5.0 1001, but workload identity federation is not supported. Workforce identity federation is still supported.

You can now use user-defined functions to export BigQuery data as Protocol Buffer (Protobuf) columns. This feature is generally-available.

August 08, 2023

The following features are now generally available (GA) in queries and materialized views:

• HAVING MAX and HAVING MIN clauses for the ANY_VALUE function.

• MAX_BY function, which is a synonym for ANY_VALUE(x HAVING MAX y).

• MIN_BY function, which is a synonym for ANY_VALUE(x HAVING MIN y).

August 07, 2023

The quantitive LIKE operator is now in preview. With this operator, you can check a search value for matches against several patterns, using one of these conditions:

• LIKE ANY: Checks if at least one pattern matches.
• LIKE SOME: Synonym for LIKE ANY.
• LIKE ALL: Checks if every pattern matches.

The following JSON functions are now generally available (GA).

• JSON_ARRAY: Creates a JSON array.
• JSON_ARRAY_APPEND: Appends JSON data to the end of a JSON array.
• JSON_ARRAY_INSERT: Inserts JSON data into a JSON array.
• JSON_OBJECT: Creates a JSON object.
• JSON_REMOVE: Produces JSON with the specified JSON data removed.
• JSON_SET: Inserts or replaces JSON data.
• JSON_STRIP_NULLS: Removes JSON nulls.
• LAX_BOOL: Attempts to convert a JSON value to a SQL BOOL value.
• LAX_FLOAT64: Attempts to convert a JSON value to a SQL FLOAT64 value.
• LAX_INT64: Attempts to convert a JSON value to a SQL INT64 value.
• LAX_STRING: Attempts to convert a JSON value to a SQL STRING value.

BigQuery now supports the ability to deny access to principals via deny policies for the following IAM permissions :

• Managing reservations and capacity commitments: bigquery.googleapis.com/capacityCommitments.*, bigquery.googleapis.com/bireservations.*, bigquery.googleapis.com/reservationAssignments.*, bigquery.googleapis.com/reservations.*
• Resource Deletion: bigquery.googleapis.com/[datasets, tables, models, routines, jobs, connections].delete
• Dataset tag bindings: bigquery.googleapis.com/datasets.[createTagBinding, listTagBinding]
• Row Access Policies: bigquery.rowAccessPolicies.[create, delete, update, setIamPolicy]

A weekly digest of client library updates from across the Cloud SDK.

Node.js

Changes for @google-cloud/bigquery

7.1.1 (2023-08-03)

Bug Fixes

• Node release schedule link (#1241) (3bcb08a)
• Update @types/big.js (8d16fae)

7.1.0 (2023-08-01)

Features

• Support and optionally parse JSON field (#1229) (cd11447)
• Support create/list datasets on a different project (#1230) (86c63fb)

7.0.0 (2023-07-31)

⚠ BREAKING CHANGES

• migrate to node 14 (#1236)

Bug Fixes

• deps: Update dependency google-auth-library to v9 (#1228) (22cf786)

Miscellaneous Chores

• Migrate to node 14 (#1236) (73bf31f)

Analytics Hub now supports the use of routines in linked datasets. This feature is now in preview.

August 04, 2023

BigQuery now supports using manifest files for external tables. This feature is now generally available (GA).

August 03, 2023

With Analytics Hub, you can now track usage metrics of your shared datasets. This feature is generally available (GA). The usage metrics include the following:

• Jobs that run against your shared dataset.
• The consumption details of your shared dataset by subscribers' projects and organizations.
• The number of rows and bytes processed by the job.

Cloud console updates: The following features are now available in preview:

• On the Welcome page, in the Recently accessed section, you can view your 10 most recently accessed resources.
• After you run a query in the query editor, in the Chart tab, you can see the visualization of your query results.

You can now GRANT or REVOKE access to materialized views with a SQL statement. This feature is generally available (GA).

July 31, 2023

A weekly digest of client library updates from across the Cloud SDK.

Go

Changes for bigquery/storage/apiv1beta1

1.53.0 (2023-07-24)

Features

• bigquery/analyticshub: Promote to GA (130c571)
• bigquery/connection: Add support for Salesforce connections, which are usable only by allowlisted partners (bac978a)
• bigquery/datapolicies: Promote to GA (130c571)
• bigquery/storage: Add ResourceExhausted to retryable error for Write API unary calls (#8214) (8ff13bf)

Bug Fixes

• bigquery/storage/managedwriter: Context refactoring (#8275) (c4104ea)

Java

Changes for google-cloud-bigquery

2.31.0 (2023-07-25)

Features

• Adds Exception handling to handle ALREADY EXISTS error (#2788) (67a07ea)

Dependencies

• Update arrow.version to v12.0.1 (#2750) (f92bee5)
• Update dependency com.google.api.grpc:proto-google-cloud-bigqueryconnection-v1 to v2.24.0 (#2811) (b660063)
• Update dependency com.google.cloud:google-cloud-datacatalog-bom to v1.28.0 (#2812) (921716d)
• Update dependency com.google.cloud:google-cloud-shared-dependencies to v3.13.1 (#2806) (df3cd76)
• Update dependency org.junit.vintage:junit-vintage-engine to v5.10.0 (#2808) (694f711)
• Update github/codeql-action action to v2.21.0 (#2803) (c6c536c)

BigQuery Omni is now available in the AWS - US West (Oregon) (aws-us-west-2) and the AWS - Europe (Ireland) (aws-eu-west-1) regions.

BigQuery Storage Write API multiplexing is now generally available (GA). You can use multiplexing in the default stream to write to multiple destination tables with shared connections.

July 28, 2023

Query queues are now generally available (GA). With query queues, BigQuery automatically determines your query concurrency based on available slots rather than a fixed limit. Once the maximum concurrency is reached, additional queries are queued until processing resources are available. Query queues are enabled by default and have been rolled out over the last several weeks; no user action is required and you shouldn't see any degradation in your query performance. You can optionally set the maximum concurrency target for a reservation. You can also control the interactive and batch query queue timeout by using default configurations.

July 24, 2023

A weekly digest of client library updates from across the Cloud SDK.

Java

Changes for google-cloud-bigquery

2.30.1 (2023-07-18)

Dependencies

• Update dependency com.google.api.grpc:proto-google-cloud-bigqueryconnection-v1 to v2.23.0 (#2791) (940301b)
• Update dependency com.google.cloud:google-cloud-datacatalog-bom to v1.27.0 (#2792) (c791066)

2.30.0 (2023-07-17)

Features

• Add missing storage related fields to Table, TableInfo and StandardTableDefinition (#2673) (e3003f4)
• Add support for Search statistics (#2787) (344f695)

Dependencies

• Update dependency com.google.api.grpc:proto-google-cloud-bigqueryconnection-v1 to v2.22.0 (#2777) (078f244)
• Update dependency com.google.cloud:google-cloud-datacatalog-bom to v1.26.0 (#2778) (2ee52c9)
• Update dependency com.google.cloud:google-cloud-shared-dependencies to v3.13.0 (#2786) (dd14eee)
• Update github/codeql-action action to v2.20.1 (#2766) (2014613)
• Update github/codeql-action action to v2.20.4 (#2784) (e886f5f)
• Update ossf/scorecard-action action to v2.2.0 (#2775) (688b2a0)

Python

Changes for google-cloud-bigquery

3.11.4 (2023-07-19)

Bug Fixes

• Updates typing in function definitions (#1613) (db755ce)

July 20, 2023

Multivariate time series forecasting with the ARIMA_PLUS_XREG model in BigQuery ML is now generally available (GA). This feature lets you perform time series forecasting with extra feature columns. For more information, see the ARIMA_PLUS_XREG information in the end-to-end user journey topic, and try the multivariate time-series forecasting from Seattle air quality data tutorial.

BigQuery ML has introduced new Explainable AI capabilities for better model explainability:

• You can now use the ML.EXPLAIN_FORECAST function with ARIMA_PLUS_XREG models.
• You can use the updated ML.EXPLAIN_FORECAST function to get explanations of the holiday effect for holidays in time series forecasting models (both ARIMA_PLUS and ARIMA_PLUS_XREG).
• You can now use the ML.GLOBAL_EXPLAIN function with AutoML Tables models for global model explainability.
• For Boosted Tree and Random Forest models, you can now use the approx_global_feature_contrib training option to use fast approximation for global feature contribution computation in model training, and the approx_feature_contrib option in the ML.EXPLAIN_PREDICT function to use the fast approximation for local feature contribution computation in model inference.

Now you can also use Vertex Explainable AI on BigQuery ML models that you've registered to the Vertex AI Model Registry. To learn more, see Explainable AI for BigQuery ML models.

July 19, 2023

BigQuery can now use search indexes to optimize some queries that contain the equal operator (=), IN operator, LIKE operator, or STARTS_WITH function to compare string literals with indexed data. This feature is in preview.

July 17, 2023

Primary and foreign key table constraints are now generally available. In addition to the features available in preview, you can now also manage constraints through the BigQuery API and view constraints in the BigQuery console.

A weekly digest of client library updates from across the Cloud SDK.

Node.js

Changes for @google-cloud/bigquery

6.2.1 (2023-07-10)

Bug Fixes

• Change view name to match sample comments (#1207) (1002dee)
• Encode values with null prototype (#1219) (1b7a52e)

The google.cloud.bigquery.storage.v1beta2 API package for BigQueryWrite operations is deprecated and will be removed on July 17, 2024. After that date, requests to that package version for use with the BigQuery Storage Write API will fail. Data written to BigQuery using the BigQuery Storage Write API is accessible by using the google.cloud.bigquery.storage.v1 package.

Next steps: If you call the API directly, switch to google.cloud.bigquery.storage.v1, the generally available (GA) version of the API, to prevent any impact on your workflow.

July 12, 2023

The following BigQuery ML feature preprocessing functionality is now generally available (GA).

You can export models that use the TRANSFORM clause for feature preprocessing to the TensorFlow SavedModel format. There are 13 data types supported for TRANSFORM clause input, and 127 SQL functions supported for use within the TRANSFORM clause.

You can also now deploy a model trained with the TRANSFORM clause to Vertex AI and locally.

Use the following functions to perform feature preprocessing:

• ML.IMPUTER
• ML.LABEL_ENCODER
• ML.MAX_ABS_SCALER
• ML.MULTI_HOT_ENCODER
• ML.NORMALIZER
• ML.ONE_HOT_ENCODER
• ML.ROBUST_SCALER

Custom holiday modeling for time series forecasting is now in preview. This release offers the following features to improve the transparency, flexibility, and explainability of time series forecasting in BigQuery ML:

• New CREATE MODEL syntax to specify custom holiday modeling for time series models.
• The new ML.HOLIDAY_INFO function, which returns the list of holidays being modeled by an ARIMA_PLUS or ARIMA_PLUS_XREG time series forecasting model.
• An updated ML.EXPLAIN_FORECAST function, which includes an explanation of the holiday effect for each holiday included in the model.

A new public table, bigquery-public-data.ml_datasets.holidays_and_events_for_forecasting, has also been added to provide easy look-up of the built-in holidays used in time series forecasting models.

Try these features with the Use custom holidays in a time-series forecasting model tutorial.

July 06, 2023

Spanner Data Boost lets you execute analytics queries and data exports with near-zero impact to existing workloads on your provisioned Spanner instance. This feature is now generally available (GA) in the following regions:

• asia-northeast1 (Tokyo)
• us-central1 (Iowa)
• southamerica-east1 (São Paulo)
• europe-west1 (Belgium)
• europe-west2 (London)
• europe-west3 (Frankfurt)

July 05, 2023

BigQuery is now available in the Turin (europe-west12) and Doha (me-central1) regions.

You can use the LOAD DATA SQL statement to load data from Avro, CSV, newline delimited JSON, JSON, ORC, or Parquet files into a table. This feature is generally available (GA).

The slot estimator now provides cost-optimal commitment and autoscale recommendations based on editions pricing and historical performance metrics. This feature is in preview.

The fail-safe period is now generally available (GA). The fail-safe period offers an additional seven days of data storage after the time travel window, so that the data is available for emergency recovery. Billed costs won't include the bytes used for fail-safe storage until July 17th, 2023.

The ability to use physical bytes for storage billing is now generally available (GA). When you set your dataset's storage billing model to use physical bytes, the total active storage costs you are billed for includes the bytes used for time travel and fail-safe storage. For more information, see Dataset storage billing models.

The ability to configure the time travel window is now generally available (GA). You can specify the duration of the time travel window from a minimum of two days to a maximum of seven days.

BigQuery capacity commitments have changed as follows:

• Annual commitments are now only available in Enterprise or Enterprise Plus edition. Flat-rate annual commitments are no longer available. For more information about pricing, see Capacity compute (analysis) pricing.
• Monthly and flex commitments are no longer available. For more information about commitment options, see Capacity commitment plans.

You can now restrict data egress on Analytics Hub listings. This feature is now in preview.

July 03, 2023

A weekly digest of client library updates from across the Cloud SDK.

Go

Changes for bigquery/storage/apiv1beta1

1.52.0 (2023-06-23)

Features

• bigquery/storage: Add estimated physical file sizes to ReadAPI v1 (94ea341)
• bigquery/storage: Add table sampling to ReadAPI v1 (ca94e27)
• bigquery: Support for tables primary and foreign keys (#8055) (93d6a1a)
• bigquery: Update all direct dependencies (b340d03)

Bug Fixes

• bigquery/storage/managedwriter: Correct reconnection logic (#8164) (a67d53d)
• bigquery: REST query UpdateMask bug (df52820)
• bigquery: RowIterator.Schema not filled when using Storage Read API (#7671) (31040e8)

Python

Changes for google-cloud-bigquery

3.11.3 (2023-06-27)

Bug Fixes

• Type annotations include Optional when None is accepted (#1554) (6c1ab80)

June 30, 2023

Metadata caching is now available for BigLake tables that reference Amazon S3 data. This feature is in preview. Using cached metadata might improve query performance for BigLake tables.

June 29, 2023

Support for the following compliance programs is now generally available (GA):

• EU Regions and Support with Sovereignty Controls
• Sovereign Controls by Partners
• International Traffic in Arms Regulation (ITAR)

June 26, 2023

You can now create stored procedures for Apache Spark using Java or Scala. You can also use the Google Cloud console PySpark editor to add options for stored Python procedures for Apache Spark. This feature is in Preview.

A weekly digest of client library updates from across the Cloud SDK.

Java

Changes for google-cloud-bigquery

2.29.0 (2023-06-23)

Features

• Increase default Read API timeout to 60s (#2764) (f606d0b)

Dependencies

• Update dependency com.google.cloud:google-cloud-shared-dependencies to v3.12.0 (#2771) (7537e0f)
• Update dependency org.graalvm.buildtools:junit-platform-native to v0.9.23 (#2759) (27ba48a)
• Update dependency org.graalvm.buildtools:native-maven-plugin to v0.9.23 (#2760) (8cddf8f)

2.28.0 (2023-06-19)

Features

• Add primary key and foreign keys (#2744) (afb571c)
• Partial Projection of Table Metadata (#2756) (9207743)
• Return JobID with TableResult (#2689) (aa38428)

Python

Changes for google-cloud-bigquery

3.11.2 (2023-06-21)

Bug Fixes

• Updates tests based on revised hacker_news tables (#1591) (d73cf49)

June 21, 2023

TRUNCATE TABLE is now supported for multi-statement transactions. This feature is now generally available (GA).

June 20, 2023

Metadata caching is now generally available (GA). Using cached metadata might improve query performance for BigLake tables and object tables that reference large numbers of objects, by allowing the query to avoid listing objects from Cloud Storage.

This release includes support for the following new features:

• Protecting metadata cache data with customer-managed encryption keys.
• Statistics on metadata cache usage.
• Table statistics for better query plan performance.

Metadata cache usage is billed going forward. For more information, see Costs.

BigQuery now supports querying Apache Iceberg tables that are created by open source engines. This feature is now generally available (GA).

June 19, 2023

A weekly digest of client library updates from across the Cloud SDK.

Java

Changes for google-cloud-bigquery

2.27.1 (2023-06-13)

Dependencies

• Update actions/checkout action to v3.5.3 (#2746) (17f8438)
• Update dependency com.google.api.grpc:proto-google-cloud-bigqueryconnection-v1 to v2.21.0 (#2741) (d665e52)
• Update dependency com.google.cloud:google-cloud-datacatalog-bom to v1.25.0 (#2743) (5d38d23)
• Update dependency com.google.cloud:google-cloud-shared-dependencies to v3.11.0 (#2738) (3b56445)
• Update github/codeql-action action to v2.20.0 (#2751) (42ae181)
• Update github/codeql-action action to v2.3.6 (#2712) (f043ed6)

Python

Changes for google-cloud-bigquery

3.11.1 (2023-06-09)

Documentation

• Add/reformat return types for cloud RAD docs (#1582) (6efdce1)

June 15, 2023

The following Generative AI features are now in preview with allowlist:

• Creating a remote model based on the Vertex AI large language model (LLM) text-bison.
• Using the ML.GENERATE_TEXT function with an LLM-based remote model to perform generative natural language tasks on text stored in BigQuery tables.

Try these features with the Generate text by using a remote model and the ML.GENERATE_TEXT function tutorial.

June 14, 2023

BigQuery now provides information about the fail-safe period. The fail-safe period offers an additional seven days of data storage after the time travel window, so that the data is available for emergency recovery. This feature is in preview.

The INFORMATION_SCHEMA views that show table storage metadata are now generally available (GA):

• Use the TABLE_STORAGE view to get a snapshot of current storage usage for tables at the project level.
• Use the TABLE_STORAGE_BY_ORGANIZATION view to get a snapshot of current storage usage for tables at the organization level.

BigLake Metastore is now generally available (GA). You can use BigLake Metastore to access and manage Iceberg table metadata from multiple sources.

June 12, 2023

The query execution graph is now generally available (GA). You can use the query execution graph to diagnose query performance issues, and to receive query performance insights.

June 05, 2023

A weekly digest of client library updates from across the Cloud SDK.

Java

Changes for google-cloud-bigquery

2.27.0 (2023-05-30)

Features

• Add support for session id on TableDataWriteChannel (#2715) (42851d8)

Bug Fixes

• Add support for repeated record query parameters (#2698) (51aff50)

Dependencies

• Update dependency com.google.api.grpc:proto-google-cloud-bigqueryconnection-v1 to v2.20.0 (#2720) (4962cac)
• Update dependency com.google.apis:google-api-services-bigquery to v2-rev20230506-2.0.0 (#2707) (4d2ec07)
• Update dependency com.google.apis:google-api-services-bigquery to v2-rev20230520-2.0.0 (#2723) (5c64797)
• Update dependency com.google.cloud:google-cloud-bigquerystorage-bom to v2.37.2 (#2726) (052c47a)
• Update dependency com.google.cloud:google-cloud-datacatalog-bom to v1.24.0 (#2721) (7c357fb)
• Update dependency com.google.cloud:google-cloud-shared-dependencies to v3.10.1 (#2713) (744e83a)

Python

Changes for google-cloud-bigquery

3.11.0 (2023-06-01)

Features

• Add remote function options to routines (#1558) (84ad11d)

Bug Fixes

• Filter None values from OpenTelemetry attributes (#1567) (9ea2e21)
• Handle case when expirationMs is None (#1553) (fa6e13d)
• Raise most recent exception when not able to fetch query job after starting the job (#1362) (09cc1df)

May 25, 2023

The BigQuery partitioning and clustering recommender is now in preview. The recommender analyzes your BigQuery tables to identify partitioning or clustering opportunities for potential cost savings. You can view partition or cluster recommendations through the BigQuery UI or recommender API. You can also apply recommendations directly to your BigQuery tables.

May 23, 2023

DML statements no longer count toward the number of table or partitioned tables modifications per day. The limit of table and partitioned table modifications has not changed.

May 22, 2023

A weekly digest of client library updates from across the Cloud SDK.

Java

Changes for google-cloud-bigquery

2.26.1 (2023-05-16)

Bug Fixes

• Custom host for resumable uploads (#2696) (2b4eff1)

2.26.0 (2023-05-15)

Features

• Add field in HivePartitioningOptions (#2678) (4165e55)
• Allow passing autodetect_schema on table update (#2661) (4c01698)

Bug Fixes

• Move ratio calculation for whether to use read API to avoid NPE with setUseReadAPI(false) (#2509) (e1326c8)

Dependencies

• Update arrow.version to v12 (major) (#2675) (7700cf5)
• Update dependency com.google.api.grpc:proto-google-cloud-bigqueryconnection-v1 to v2.19.0 (#2691) (1939803)
• Update dependency com.google.cloud:google-cloud-datacatalog-bom to v1.23.0 (#2692) (f56e541)
• Update dependency com.google.cloud:google-cloud-shared-dependencies to v3.9.0 (#2685) (b74da29)
• Update dependency org.graalvm.buildtools:junit-platform-native to v0.9.22 (#2687) (cf5d758)
• Update dependency org.graalvm.buildtools:native-maven-plugin to v0.9.22 (#2688) (32ea8ab)
• Update github/codeql-action action to v2.3.3 (#2658) (487f207)

May 19, 2023

EXTERNAL_QUERY SQL pushdown optimizes data retrieval from external sources like Cloud SQL or Cloud Spanner databases. Transferring less data reduces execution time and cost. SQL pushdown encompasses both column pruning (SELECT clauses) and filter pushdowns (WHERE clauses). SQL pushdown applies to SELECT * FROM T queries, a significant percentage of all federated queries. Not all data types are supported for filter pushdowns. This feature is generally available (GA).

May 18, 2023

You can now sort your query results by using the sort menu next to a column name. This feature is in preview.

May 16, 2023

The VPC Service Controls perimeter that protects the BigQuery API now also protects the BigQuery Reservation API. Customers who have already configured VPC Service Controls for the BigQuery API or the BigQuery Reservation API should update their configurations to reflect this change. For more information, see BigQuery Reservation API.

May 15, 2023

A weekly digest of client library updates from across the Cloud SDK.

Go

Changes for bigquery/storage/apiv1beta1

1.51.2 (2023-05-08)

Bug Fixes

• bigquery: Update grpc to v1.55.0 (1147ce0)

BigQuery Omni is now available in the AWS - Asia Pacific (Seoul) (aws-ap-northeast-2) region.

May 11, 2023

Object tables are now generally available (GA).

Object tables are read-only tables containing metadata for unstructured data stored in Cloud Storage. They enable you to analyze and perform inference on images, audio files, documents and other file types by using BigQuery ML and BigQuery remote functions. Object tables extend the data security and governance best practices currently applied to structured data to unstructured data as well.

The GA release includes the following new and updated functions:

• ML.DECODE_IMAGE: Decodes image data so that it can be interpreted by the ML.PREDICT function.
• ML.CONVERT_COLOR_SPACE: Converts images with an RGB color space to a different color space.
• ML.CONVERT_IMAGE_TYPE: Converts the data type of the pixel values in an image.
• ML.RESIZE_IMAGE: Resizes images.
• ML.DISTANCE: Computes the distance between two vectors.
• ML.LP_NORM: Computes the Lᵖ norm for a vector, where ᵖ is the degree.

May 10, 2023

BigQuery is now available in the Dallas (us-south1) region.

May 09, 2023

You can now view BI Engine Top Tables Cached Bytes, BI Engine Query Fallback Count, and Query Execution Count as dashboard metrics for BigQuery. This feature is now generally available (GA).

EXTERNAL_QUERY SQL pushdown optimizes data retrieval from external sources like Cloud SQL or Cloud Spanner databases. Transferring less data reduces execution time and cost. SQL pushdown encompasses both column pruning (SELECT clauses) and filter pushdowns (WHERE clauses). SQL pushdown applies to SELECT * FROM T queries, a significant percentage of all federated queries. Pushdowns have limitations, for example not all data types are supported for filter pushdowns. This feature is generally available (GA).

May 08, 2023

Differential privacy is now in preview and includes four differential privacy aggregate functions that can be used to anonymize data: AVG, COUNT, SUM, and PERCENTILE_CONT. To learn more, see the following topics:

• Use differential privacy
• Differential privacy clause
• Differentially private aggregate functions
• Extending differential privacy

A weekly digest of client library updates from across the Cloud SDK.

Go

Changes for bigquery/storage/apiv1beta1

1.51.1 (2023-05-04)

Bug Fixes

• bigquery/storage/managedwriter: Schema evolution improvements (#7838) (49a1621)
• bigquery: Upgrade arrow to v12 to fix build error (#7870) (402e365)

Documentation

• bigquery/storage/managedwriter: Improve tuning guidance (#7848) (ca2184c)

INFORMATION_SCHEMA.MATERIALIZED_VIEW view and enhanced job statistics now let you monitor materialized view usage and refresh jobs. This feature is in preview.

May 05, 2023

The INSERT INTO SELECT statement now lets you filter data from files in Amazon S3 and Azure Blob Storage and append it into BigQuery tables. This feature is in preview.

May 04, 2023

You can now use configuration YAML files to transform SQL code when you translate SQL queries from your source database. Configuration YAML files can be used with the batch SQL translator, the interactive SQL translator, and the batch translation Python client. This feature is now in preview.

May 03, 2023

The table clones feature of BigQuery is now generally available (GA).

May 01, 2023

A weekly digest of client library updates from across the Cloud SDK.

Java

Changes for google-cloud-bigquery

2.25.0 (2023-04-27)

Features

• Add ICEBERG format options (#2662) (55048ca)

Dependencies

• Update dependency com.google.api.grpc:proto-google-cloud-bigqueryconnection-v1 to v2.18.0 (#2648) (29bd415)
• Update dependency com.google.apis:google-api-services-bigquery to v2-rev20230408-2.0.0 (#2650) (b9c2f60)
• Update dependency com.google.cloud:google-cloud-bigquerystorage-bom to v2.36.1 (fea119b)
• Update dependency com.google.cloud:google-cloud-datacatalog-bom to v1.22.0 (#2649) (b6326f3)
• Update dependency com.google.cloud:google-cloud-shared-dependencies to v3.8.0 (#2659) (691a47a)
• Update dependency org.junit.vintage:junit-vintage-engine to v5.9.3 (#2660) (319f98e)

You can now add descriptions to the columns of a view. To do this, use the CREATE VIEW or ALTER COLUMN DDL statements. This feature is in preview.

If you use query queues, then you can set the interactive and batch queue timeouts in your default configuration. This feature is in preview.

April 27, 2023

JSON data type mapping is now available for Cloud Spanner federated queries. This feature is generally available (GA).

April 26, 2023

BigLake and non-BigLake external tables now support Cloud Storage custom dual-regions. This feature is generally available (GA).

April 25, 2023

Dynamic data masking has been updated to allow masking on RECORD columns that have been set to REPEATED mode. Previously, querying such columns when data masking had been applied would return internal errors. This feature is generally available (GA).

April 24, 2023

A weekly digest of client library updates from across the Cloud SDK.

Go

Changes for bigquery/storage/apiv1beta1

1.51.0 (2023-04-17)

Features

• bigquery/storage/managedwriter: Expose connection multiplexing as experimental (#7673) (3b8bfb4)
• bigquery: Add Iceberg as DataFormat option. (#7155) (7a9e211)
• bigquery: Expose dataset MaxTimeTravelHours (#7706) (22a666c)

Bug Fixes

• bigquery: Respect context during query execution (#7693) (56772f5)

Python

Changes for google-cloud-bigquery

3.10.0 (2023-04-18)

Features

• Add date, datetime, time, timestamp dtype to to_dataframe (#1547) (64e913d)

The BigQuery Data Transfer Service for Google Ads supports the new Google Ads API. The Google Ads connector supports PMax and Discovery campaigns, a limit of 8000 leaf accounts per transfer, the --table_filter flag, and backwards compatibility. This feature is now generally available (GA).

April 19, 2023

Updates to preferred tables for existing BI engine reservations now take up to ten seconds to propagate, down from five minutes. This feature is generally available (GA).

April 17, 2023

A weekly digest of client library updates from across the Cloud SDK.

Java

Changes for google-cloud-bigquery

2.24.5 (2023-04-14)

Dependencies

• Update actions/checkout action to v3.5.2 (#2630) (95e49fd)
• Update dependency com.google.api.grpc:proto-google-cloud-bigqueryconnection-v1 to v2.16.0 (#2625) (594a7b4)
• Update dependency com.google.apis:google-api-services-bigquery to v2-rev20230401-2.0.0 (#2631) (5d8d9a6)
• Update dependency com.google.cloud:google-cloud-bigquerystorage-bom to v2.35.0 (8439020)
• Update dependency com.google.cloud:google-cloud-datacatalog-bom to v1.20.0 (#2626) (f466b51)
• Update dependency com.google.cloud:google-cloud-shared-dependencies to v3.7.0 (#2637) (e8f07d7)
• Update dependency org.graalvm.buildtools:junit-platform-native to v0.9.21 (#2633) (3e376b1)
• Update dependency org.graalvm.buildtools:native-maven-plugin to v0.9.21 (#2634) (000f720)
• Update github/codeql-action action to v2.2.12 (#2635) (b2f97e9)
• Update ossf/scorecard-action action to v2.1.3 (#2618) (d166401)

Python

Changes for google-cloud-bigquery

1.28.3 (2022-12-14)

Bug Fixes

• Fix invalid version specification of pyarrow (#1403) (4812d82)

April 13, 2023

BigQuery supports setting the rounding mode to ROUND_HALF_EVEN or ROUND_HALF_AWAY_FROM_ZERO for parameterized NUMERIC or BIGNUMERIC columns at the column level. You can specify a default rounding mode at the table or dataset level that is automatically attached to any columns added within those entities. The ROUND() function also accepts the rounding mode as an optional argument. This feature is generally available GA.

April 10, 2023

The limit for maximum result size (20 GiB logical bytes) when querying Azure or Amazon Simple Storage service (S3) data is now generally available (GA). Querying Azure and Amazon S3 data are now subject to the following quotas and limitations:

• The maximum row size is 10 MiB. For more information, see Quotas for query jobs.

• If your query uses the ORDER BY clause and has a result size larger than 256 MB, then your query fails. Previously, this limit was 2 MB. For more information, see Limitations.

A weekly digest of client library updates from across the Cloud SDK.

Go

Changes for bigquery/storage/apiv1beta1

1.50.0 (2023-04-03)

Features

• bigquery/connection: Add spark connection properties type (#7570) (499b489)
• bigquery/migration: Add request_source field and update formatting (#7586) (c967961)
• bigquery/reservation: Add edition/autoscale related fields (#7608) (2b7bb66)
• bigquery/storage/managedwriter: Decouple connections and writers (#7314) (7d085b4)
• bigquery/storage/managedwriter: Introduce location routing header (#7663) (cf06802)

Bug Fixes

• bigquery/storage/managedwriter: Fix option propagation (#7669) (f684e16)

Documentation

• bigquery/reservation: Mention that some fields are deprecated (597ea0f)

The results for queries against table snapshots can now be returned from cache.

April 06, 2023

The add data demo guide walks you through the process of adding data to BigQuery through popular sources and is now in preview.

April 05, 2023

Non-incremental materialized views support most SQL queries, including OUTER JOIN, UNION, and HAVING clauses, as well as analytic functions. This feature is in preview.

April 04, 2023

BigQuery is now available in the Israel (me-west1) region.

April 03, 2023

A weekly digest of client library updates from across the Cloud SDK.

Java

Changes for google-cloud-bigquery

2.24.4 (2023-03-30)

Bug Fixes

• QueryWithStructsParameters sample mismatch (#2610) (71f9f55)

Dependencies

• Update dependency com.google.apis:google-api-services-bigquery to v2-rev20230318-2.0.0 (#2607) (a328eb2)
• Update dependency com.google.cloud:google-cloud-bigquerystorage-bom to v2.34.2 (#2619) (e4aa0fe)
• Update dependency com.google.cloud:google-cloud-shared-dependencies to v3.6.0 (#2612) (eac97ac)
• Update github/codeql-action action to v2.2.9 (#2608) (24aac14)

Python

Changes for google-cloud-bigquery

3.9.0 (2023-03-28)

Features

• Expose query job on dbapi cursor (#1520) (339eb0e)

Bug Fixes

• Keyerror when the load_table_from_dataframe accesses a unmapped dtype dataframe index (#1535) (a69348a)

3.8.0 (2023-03-24)

Features

• Add bool, int, float, string dtype to to_dataframe (#1529) (5e4465d)
• Add default LoadJobConfig to Client (#1526) (a2520ca)
• Expose configuration property on CopyJob, ExtractJob, LoadJob, QueryJob (#1521) (8270a10)

Bug Fixes

• Loosen ipywidgets restrictions further to address ipython compatibility issues (#1531) (50e5026)

March 30, 2023

BigQuery ML documentation is now integrated with BigQuery documentation to unify resources for data analysis and machine learning tasks such as inference. BigQuery ML documentation resources include:

• Get started with BigQuery ML
• End-to-end user journey for each model
• BigQuery ML SQL refence
• BigQuery ML pricing

BigQuery Partner Center, which can be used to discover and try validated partner applications, is now generally available (GA). In addition, the Google Cloud Ready - BigQuery initiative has added 14 new partners.

March 29, 2023

Compute (analysis) is now generally available (GA) in three new BigQuery editions: Standard, Enterprise, and Enterprise Plus. These editions support the slots autoscaling model to meet your organizations' needs and budgets.

Autoscaling slots are now generally available (GA). Autoscaling slot reservations and commitments created during the feature's preview have been set to BigQuery Enterprise edition.

March 28, 2023

You can now use the tf_version training option to specify the Tensorflow (TF) version during model training. By default, tf_version is set as '1.15'. If you want to use TF2 with Keras API, you can add tf_version = '2.8.0' when creating the model.

You can now use the xgboost_version training option to specify the XGBoost version during model training. By default, xgboost_version is set as '0.9'. You can choose XGBoost version 1.1 by specifying xgboost_version = '1.1'.

You can now use the instance_weight_col training option to identify the column containing weights for each data point in the training dataset. Currently the instance_weight_col option is only available for boosted tree and random forest models with non-array feature types.

You can now import model artifacts saved in ONNX, XGBoost, and TensorFlow Lite formats into BigQuery for inference, allowing you to leverage models built in popular frameworks directly within the BigQuery ML inference engine.

You can also host models remotely on Vertex AI Prediction and do inference with BigQuery ML, removing the need to build data pipelines manually.

You can do inference with Google Cloud's state of the art pretrained models using Cloud AI service table-valued functions (TVFs) to get insights from your data. The TVFs work with Cloud Vision API, Cloud Natural Language API and Cloud Translation API.

These features are in preview. To enroll to use this feature, complete the enrollment form.

March 27, 2023

A weekly digest of client library updates from across the Cloud SDK.

Node.js

Changes for @google-cloud/bigquery

6.2.0 (2023-03-22)

Features

• Add collation feature tests (#1188) (80d86ba)
• Add support for microseconds precision (#1192) (b5801a6)

Bug Fixes

• Add typings for named parameter structs (#1198) (c7c2ba1)
• Client endpoint sample error msg (#1193) (e3ea8cd)
• Pass Query.wrapIntegers to job.getQueryResults (#1191) (fb13510)

Java

Changes for google-cloud-bigquery

2.24.3 (2023-03-24)

Dependencies

• Update actions/checkout action to v3.5.0 (#2600) (f38d9f1)

2.24.2 (2023-03-22)

Dependencies

• Update github/codeql-action action to v2.2.8 (#2593) (d306ad8)

2.24.1 (2023-03-21)

Dependencies

• Update cloud client dependencies (7b07779)
• Update dependency com.google.cloud:google-cloud-bigquery to v2.23.2 (7b07779)
• Update dependency com.google.cloud:google-cloud-bigquery to v2.24.0 (7b07779)
• Update dependency com.google.cloud:google-cloud-bigquerystorage-bom to v2.34.1 (7b07779)
• Update dependency com.google.cloud:google-cloud-bigtable to v2.20.1 (7b07779)
• Update dependency com.google.cloud:libraries-bom to v26.10.0 (7b07779)

2.24.0 (2023-03-21)

Features

• Add support for clone (#2553) (2186c64)

Dependencies

• Update actions/checkout action to v3.4.0 (#2575) (6935a1e)
• Update actions/upload-artifact action to v3.1.2 (#2571) (aa0c70e)
• Update cloud client dependencies (#2583) (dcacc31)
• Update dependency com.google.api.grpc:proto-google-cloud-bigqueryconnection-v1 to v2.15.0 (#2577) (eaf09d6)
• Update dependency com.google.apis:google-api-services-bigquery to v2-rev20230311-2.0.0 (#2578) (aab037c)
• Update dependency com.google.cloud:google-cloud-shared-dependencies to v3.5.0 (#2580) (1764eeb)
• Update dependency com.google.cloud:google-cloud-storage to v2.20.0 (#2559) (8a854db)
• Update github/codeql-action action to v2.2.7 (#2572) (105f5ee)

BigQuery now supports change data capture (CDC) by processing and applying streamed changes in real-time to existing data using the BigQuery Storage Write API. This feature is in preview.

March 22, 2023

BigQuery now supports Unicode column naming using international character sets, alphanumeric and special characters. Existing columns can use these new capabilities using the RENAME command. This feature is now in preview.

March 20, 2023

A weekly digest of client library updates from across the Cloud SDK.

Go

Changes for bigquery/storage/apiv1beta1

1.49.0 (2023-03-16)

Features

• bigquery: Add support for storage billing model (#7510) (0132ca9), refs #6978
• bigquery: Update iam and longrunning deps (91a1f78)

Python

Changes for google-cloud-bigquery

3.7.0 (2023-03-06)

Features

• Add connection_properties and create_session to LoadJobConfig (#1509) (cd0aaa1)
• Add default_query_job_config property and property setter to BigQuery client (#1511) (a23092c)

Documentation

• Remove < 3.11 reference from README (#1502) (c7417f4)

The following AutoML Tables model features are now generally available:

• Availability in additional regions.
• CMEK support in available regions except multi-regions US and EU.
• OPTIMIZATION_OBJECTIVE now accepts two additional options:
  • MAXIMIZE_PRECISION_AT_RECALL
  • MAXIMIZE_RECALL_AT_PRECISION

March 14, 2023

The Lineage tab in the table properties page lets you track how your data moves and transforms through BigQuery. This feature is now generally available (GA).

March 13, 2023

A weekly digest of client library updates from across the Cloud SDK.

Java

Changes for google-cloud-bigquery

2.23.2 (2023-03-07)

Bug Fixes

• External table definition parquet format options (#2535) (eb45973)

Documentation

• Remove stale snippet comment (#2555) (a71b1b2)

You can now specify translation configurations in the BigQuery Interactive SQL Translator and use it to debug Batch SQL translator jobs. This feature is now in preview.

March 10, 2023

The CREATE TABLE AS SELECT statement now lets you filter data from files in Amazon S3 and Azure Blob Storage before transferring results into BigQuery tables This feature is in preview.

March 07, 2023

Case-insensitive collation support is now generally available (GA). In addition to features available in the preview, the GA release includes:

• MIN, MAX, COUNT with DISTINCT, and PERCENTILE_DISC windows functions
• ORDER BY and PARTITION BY in the WINDOWS clause
• LIKE operator with limitations
• Views
• Materialized views with limitations
• Table functions with limitations
• BigQuery BI engine

March 06, 2023

A weekly digest of client library updates from across the Cloud SDK.

Go

Changes for bigquery/storage/apiv1beta1

1.48.0 (2023-03-01)

Features

• bigquery/connection: Add cloud spanner connection properties - serverless analytics (#7487) (14771b1)
• bigquery/storage/managedwriter: Mark managedwriter as GA (#6804) (3d3eeda)

Java

Changes for google-cloud-bigquery

2.23.1 (2023-03-02)

Dependencies

• Update dependency com.google.api.grpc:proto-google-cloud-bigqueryconnection-v1 to v2.14.0 (#2545) (ad78ebb)
• Update dependency com.google.cloud:google-cloud-datacatalog-bom to v1.18.0 (#2546) (60e45e4)
• Update dependency com.google.cloud:google-cloud-shared-dependencies to v3.4.0 (#2547) (2588582)

March 02, 2023

The WITH RECURSIVE clause is now generally available (GA). This clause lets you include one or more recursive common table expressions (CTEs) in a query.

February 27, 2023

A weekly digest of client library updates from across the Cloud SDK.

Go

Changes for bigquery/storage/apiv1beta1

1.47.0 (2023-02-22)

Features

• bigquery/storage: Add default_value_expression to TableFieldSchema (#7400) (1244b3f)
• bigquery: Add support for session in load jobs (#7418) (f9ff2ca)

Bug Fixes

• bigquery: Avoid double-channel-close during context cancellation (#7467) (ca4b2ef)

Java

Changes for google-cloud-bigquery

2.23.0 (2023-02-22)

Features

• Add support for session_id in load jobs (#2519) (e431c17)

Dependencies

• Update cloud client dependencies (#2526) (4d88ccc)
• Update dependency com.google.api.grpc:proto-google-cloud-bigqueryconnection-v1 to v2.13.0 (#2533) (ed2cb74)
• Update dependency com.google.apis:google-api-services-bigquery to v2-rev20230210-2.0.0 (#2530) (62ff092)
• Update dependency com.google.cloud:google-cloud-shared-dependencies to v3.3.0 (#2534) (f1bcc33)
• Update dependency org.graalvm.buildtools:junit-platform-native to v0.9.20 (#2527) (5fe5e74)
• Update dependency org.graalvm.buildtools:native-maven-plugin to v0.9.20 (#2528) (554e75d)

Python

Changes for google-cloud-bigquery

3.6.0 (2023-02-22)

Features

• Adding preserveAsciiControlCharacter to CSVOptions (#1491) (f832e7a)

Bug Fixes

• Annotate optional integer parameters with optional type (#1487) (a190aaa)
• Loosen ipywidget dependency (#1504) (20d3276)
• Removes scope to avoid unnecessary duplication (#1503) (665d7ba)

Dependencies

• Update minimum google-cloud-core to 1.6.0 (a190aaa)

You can set default values on columns in your BigQuery tables. This feature is now generally available (GA).

The multivariate time-series forecasting model ARIMA_PLUS_XREG is now available to on-demand users.

February 23, 2023

Authorized stored procedures are now in preview. This feature lets you share stored procedures with users or groups without giving them direct access to the underlying tables.

February 22, 2023

Fixed linked datasets querying shared dataset that has data ingested through streaming inserts or the BigQuery Storage Write API.

Primary and foreign key table constraints are now available in preview. You can define table constraints using the CREATE TABLE statement, the ALTER TABLE ADD PRIMARY KEY statement, or the ALTER TABLE ADD CONSTRAINT statement.

February 21, 2023

The ALTER TABLE RENAME COLUMN statement and the ALTER TABLE DROP COLUMN statement are now generally available (GA).

February 15, 2023

You can now make a dataset and the tables in that dataset case-insensitive when you create a dataset or alter a dataset. This feature is generally available (GA).

You can now run bq commands using service account impersonation. This feature is generally available (GA).

In the Explorer pane, the resource corresponding to the focused tab is now selected. This feature is generally available (GA).

In the Explorer pane, you can now see all the resources in the searched resource's level by clicking Show more. This feature is generally available (GA).

February 13, 2023

A weekly digest of client library updates from across the Cloud SDK.

Go

Changes for bigquery/storage/apiv1beta1

1.46.0 (2023-02-06)

Features

• bigquery: Add dataset/table collation (#7235) (9f7bbeb)
• bigquery: Use storage api for query jobs (#6822) (26c04f4)

Bug Fixes

• bigquery: Create/update an isolated dataset for collation feature (#7256) (b371558)
• bigquery: Fetch dst table for jobs when readings with Storage API (#7325) (0bf80d7), refs #7322

Java

Changes for google-cloud-bigquery

2.22.0 (2023-02-08)

Features

• Add collation for Case sensitive string column (#2490) (3257737)

Dependencies

• Update arrow.version to v11 (major) (#2495) (94ed060)
• Update dependency com.google.api.grpc:proto-google-cloud-bigqueryconnection-v1 to v2.11.0 (#2482) (e6ffb9b)
• Update dependency com.google.api.grpc:proto-google-cloud-bigqueryconnection-v1 to v2.12.0 (#2512) (09f280d)
• Update dependency com.google.cloud:google-cloud-bigquerystorage-bom to v2.31.0 (#2499) (c0a393c)
• Update dependency com.google.cloud:google-cloud-datacatalog-bom to v1.15.0 (#2483) (5c2bf69)
• Update dependency com.google.cloud:google-cloud-shared-dependencies to v3.2.0 (#2513) (02832dd)

You can now create materialized views over BigLake metadata cache-enabled tables to reference structured data stored in Cloud Storage. This feature is in preview.

February 09, 2023

You can now apply four new types of dynamic data masking to table columns in BigQuery. These new data masking types include date year, email, first four characters, and last four characters masks. This feature is generally available (GA).

Autoscaling slot reservations are now available in preview. You can create autoscaling reservations and associated commitments using slots autoscaling.

Cloud console updates: In the Explorer pane, you can now refresh the contents of a resource (project or dataset). To refresh the contents of a resource, click View actions, and then click Refresh contents.

February 06, 2023

The HAVING MAX and HAVING MIN clause for the ANY_VALUE function is now in preview.

You can now view information related to query processing to monitor and optimize queries with the query_info column in INFORMATION_SCHEMA.JOBS, JOBS_BY_FOLDER and JOBS_BY_ORGANIZATION views. This feature is generally available (GA).

A weekly digest of client library updates from across the Cloud SDK.

Python

Changes for google-cloud-bigquery

3.5.0 (2023-01-31)

Features

• Add str method to DatasetReference (#1477) (f32df1f)
• Add preserveAsciiControlCharacter to LoadJobConfig (#1484) (bd1da9a)

Documentation

• Adds snippet for creating table with external data config (#1420) (f0ace2a)
• Revise delete label table code sample, add TODO to clean up sni… (#1466) (0dab7d2)
• samples: Table variable fix (#1287) (a71888a)

February 01, 2023

The BigQuery Data Transfer Service can now transfer data from Azure Blob Storage into BigQuery. This feature is now in preview.

January 31, 2023

Azure workload identity federation is now generally available (GA) for BigQuery Omni connections. You can now create a connection for federated identity using Google Cloud console.

Cloud console updates: When you create datasets, select locations to run specific queries, or create exchanges in Analytics Hub, you now see separate options for multi-region and specific regions. Based on your selection, you see a list with more options.

January 30, 2023

You can search for BigQuery partners in the BigQuery Partner Center. This feature is in Preview.

A weekly digest of client library updates from across the Cloud SDK.

Node.js

Changes for @google-cloud/bigquery

6.1.0 (2023-01-25)

Features

• Add set client endpoint sample (#1170) (df1dcd3)

Java

Changes for google-cloud-bigquery

2.21.0 (2023-01-23)

Features

• Migrate from google-http-client-jackson2 to google-http-client-gson (#2471) (09a8382)

Bug Fixes

• java: Skip fixing poms for special modules (#1744) (#2474) (4e8bbe0)

Dependencies

• Update com.google.cloud:google-cloud-bigquerystorage-bom to v2.28.3 (5a20c32)
• Update com.google.cloud:google-cloud-datacatalog-bom to v1.14.0 (606fc1e)
• Update com.google.cloud:google-cloud-storage to v2.17.1 (#2465) (606fc1e)
• Update com.google.cloud:google-cloud-storage to v2.17.2 (#2479) (5a20c32)
• Update dependency com.google.api.grpc:proto-google-cloud-bigqueryconnection-v1 to v2.10.0 (#2463) (5bfa7ae)
• Update dependency com.google.apis:google-api-services-bigquery to v2-rev20230114-2.0.0 (#2477) (b6409d5)
• Update dependency com.google.cloud:google-cloud-shared-dependencies to v3.1.2 (#2476) (688b6a6)

January 23, 2023

A weekly digest of client library updates from across the Cloud SDK.

Python

Changes for google-cloud-bigquery

3.4.2 (2023-01-13)

Bug Fixes

• Add support for python 3.11 (#1463) (730a1de)
• Require grpcio >= 1.49.1 for python 3.11 (72b25c5)

Dependencies

• Remove upper bound on packaging dependency (#1440) (6088129)

Documentation

• Create sample to write schema file from table (#1439) (093cc68)
• Created samples for load table and create table from schema file (#1436) (8ad2e5b)
• Revise create table cmek sample (#1452) (57740e4)
• Revise get table labels code sample, add TODO to clean up snipp… (#1464) (b5ccbfe)
• Revise label table code samples (#1451) (14ae1f2)
• Revise sample for nested schema (#1446) (a097631)

January 19, 2023

More than 20 BigQuery ML components for Vertex AI Managed Pipelines are now generally available. These components benefit AI/ML users for the following:

• Building pipelines using the KFP SDK and TFX SDK
• Linking and tracking metadata automatically
• Seamless integration with Vertex AI for online prediction

Major Google Cloud pipeline components available in Vertex AI are.

• BigqueryQueryJobOp
• BigqueryCreateModelJobOp
• BigqueryExportModelJobOp
• BigqueryPredictModelJobOp
• BigqueryEvaluateModelJobOp
• BigqueryDropModelJobOp
• BigqueryEvaluateModelJobOp
• BigqueryExplainForecastModelJobOp
• BigqueryExplainPredictModelJobOp
• BigqueryForecastModelJobOp

You can now attach Resource Manager tags to datasets, which let you conditionally apply Identity and Access Management (IAM) policies to your resources. This feature is generally available (GA).

You can now use the TRANSFORM clause to train models which you can then export in the Tensorflow SavedModel format. This feature is now available in preview.

The following functions have been added for BigQuery ML:

• ML.ROBUST_SCALER
• ML.NORMALIZER
• ML.ONE_HOT_ENCODER
• ML.IMPUTER
• ML.MAX_ABS_SCALER
• ML.LABEL_ENCODER

These features are now available in preview.

January 17, 2023

BigQuery ML support for multivariate time-series forecasting with the ARIMA_PLUS_XREG model is now available in preview. This feature lets you perform time-series forecasting with extra feature columns. For more information, see the ARIMA_PLUS_XREG sections in the end-to-end user journey and the multivariate time-series forecasting from Seattle air quality data tutorial.

Sparse input support in BigQuery ML model training is now generally available (GA). This feature improves model training for data whose values are mostly zero or empty. For additional examples, see the sparse features support in BigQuery blog.

January 16, 2023

A weekly digest of client library updates from across the Cloud SDK.

Java

Changes for google-cloud-bigquery

2.20.2 (2023-01-12)

Dependencies

• Update com.google.cloud:google-cloud-bigquerystorage-bom to 2.28.1 (b62391d)
• Update dependency com.google.apis:google-api-services-bigquery to v2-rev20221209-2.0.0 (#2449) (9d0a107)
• Update dependency com.google.apis:google-api-services-bigquery to v2-rev20221217-2.0.0 (#2459) (08e2927)
• Update dependency com.google.cloud:google-cloud-shared-dependencies to v3.1.1 (#2455) (412710c)
• Update dependency org.junit.vintage:junit-vintage-engine to v5.9.2 (#2456) (04d47ad)
• Update dependency org.threeten:threeten-extra to v1.7.2 (#2450) (485be56)

January 10, 2023

The ALTER CAPACITY SET OPTIONS statement and ALTER RESERVATION SET OPTIONS statement are now generally available (GA). Additionally, the CREATE CAPACITY, CREATE RESERVATION, and CREATE ASSIGNMENT statements now support the OPTIONS clause.

January 09, 2023

The following generally available (GA) features have been added for sessions:

• In a session, temporary functions are now maintained until the session ends.

• In a session, statements that include the TEMP keyword can also include the OR REPLACE and IF NOT EXISTS keywords.

A weekly digest of client library updates from across the Cloud SDK.

Go

Changes for bigquery/storage/apiv1beta1

1.45.0 (2023-01-05)

Features

• bigquery/datapolicies: Start generating apiv1 (#7204) (fd71cba)
• bigquery/datatransfer: Add location methods (06a54a1)
• bigquery: Add REST client (06a54a1)
• bigquery: Rewrite signatures and type in terms of new location (620e6d8)

Java

Changes for google-cloud-bigquery

2.20.1 (2023-01-04)

Dependencies

• Update dependency com.google.cloud:google-cloud-bigquerystorage-bom to v2.28.0 (#2451) (4b760e1)

January 04, 2023

BigQuery ML support for image analytics with vision models is available in preview. Customers can import vision models to perform inference modeling with images to detect objects, perform optical character recognition (OCR), and more. To request access to these features, complete the BigQuery ML interest sign up form.

This new capability uses BigQuery object tables to access image data stored in Cloud Storage and predict results from machine learning models. You can now generate insights from structured and unstructured data with the following steps:

1. Create an object table to access images stored in Cloud Storage.
2. Import vision models in TensorFlow vision models such as ImageNet or ResNet 50, or import your own models to detect objects from images, to annotate photos, and to perform OCR.
3. Unify image data with structured data such as user activities or sales orders to train machine learning models. You can then use prediction results to extract insights from your data.

January 03, 2023

BigQuery ML integration with Vertex AI Model Registry is now generally available. With this integration, you can now use the following capabilities:

• Register and monitor BigQuery ML models with Vertex AI Model Registry
• Deploy BigQuery ML models directly from Vertex AI Model Registry to Vertex Deployment endpoints
• Use Vertex AI to compare and track evaluation metrics.
• Explainable AI for BigQuery ML models, including built-in XAI, inside Vertex AI
• The seamless integration between BigQuery ML and Vertex AI lets you use Vertex AI for MLOps.

Key features include:

• Model versioning for models registered with Vertex AI Model Registry
• Revision alias for different model versions, and User specified model ID
• List the models by type (custom model, BigQuery ML, AutoML)
• BigQuery ML models can be registered with Vertex AI Model Registry to help you explore, manage, and govern your BigQuery ML models
• Ability to deploy BigQuery ML models to Vertex AI end points
• BigQuery ML models deployed on Vertex AI endpoints can use MLOps features such as model monitoring

Customers can use BigQuery ML to train and run models on BigLake in Cloud Storage. See Data Cloud Blog and End to end unstructured data use cases demo.

December 22, 2022

The Lineage tab in the table properties page lets you track how your data moves and transforms through BigQuery. This feature is now in preview.

BigQuery now blocks saving query results to Google Drive from projects inside a VPC Service Controls protected perimeter.

December 19, 2022

A weekly digest of client library updates from across the Cloud SDK.

Java

Changes for google-cloud-bigquery

2.20.0 (2022-12-13)

Features

• Add fast query path support when empty jobId object is passed (#2349) (42c083a)
• Next release from main branch is 2.20.0 (#2405) (9297a43)

Dependencies

• Update arrow.version to v10.0.1 (#2426) (aff9019)
• Update cloud client dependencies (#2444) (7255357)
• Update dependency com.google.api.grpc:proto-google-cloud-bigqueryconnection-v1 to v2.8.0 (#2418) (1ac1653)
• Update dependency com.google.api.grpc:proto-google-cloud-bigqueryconnection-v1 to v2.9.0 (#2441) (01cc3c3)
• Update dependency com.google.apis:google-api-services-bigquery to v2-rev20221127-2.0.0 (#2437) (eb52002)
• Update dependency com.google.cloud:google-cloud-datacatalog-bom to v1.12.0 (#2419) (c449031)
• Update dependency com.google.cloud:google-cloud-shared-dependencies to v3.1.0 (#2435) (c99b215)
• Update dependency com.google.cloud:google-cloud-storage to v2.15.1 (#2420) (baf337a)
• Update dependency org.graalvm.buildtools:junit-platform-native to v0.9.18 (#2424) (63b5196)
• Update dependency org.graalvm.buildtools:junit-platform-native to v0.9.19 (#2432) (396c6dc)
• Update dependency org.graalvm.buildtools:native-maven-plugin to v0.9.18 (#2425) (cd2ae9f)
• Update dependency org.graalvm.buildtools:native-maven-plugin to v0.9.19 (#2433) (bf94087)
• Update jmh.version to v1.36 (#2415) (0676586)

Python

Changes for google-cloud-bigquery

3.4.1 (2022-12-09)

Documentation

• Add info about streaming quota limits to insert_rows* methods (#1409) (0f08e9a)

Dependencies

• make pyarrow and BQ Storage optional dependencies (e1aa921)

December 15, 2022

You can now access and query Cloud SQL data over a private connection. This feature is generally available (GA).

December 08, 2022

The demo query guide helps you query a public dataset from Google Trends and is now in preview.

December 05, 2022

A weekly digest of client library updates from across the Cloud SDK.

Go

Changes for bigquery/storage/apiv1beta1

1.44.0 (2022-11-30)

Features

• bigquery/datatransfer: Add Location API methods docs: updated comments (22ec3e3)
• bigquery/storage: Add missing_value_interpretations to AppendRowsRequest (2a0b1ae)
• bigquery: Add default partition expiration to Dataset (#7096) (601c77a), refs #7021
• bigquery: Remove code for reservation/apiv1beta1 (#7010) (451acc1)
• bigquery: Start generating proto stubs (#7026) (debc4c7)
• bigquery: Widen retry predicate (#6976) (753b751)

Bug Fixes

• bigquery/storage/managedwriter: Remove old header routing (#6960) (434b407)

Documentation

• bigquery/storage: Remove stale header guidance for AppendRows (9c5d6c8)

December 01, 2022

BigQuery now supports querying Apache Iceberg tables that are created by open source engines. This feature is in preview.

November 28, 2022

BigQuery now supports the following features when you load data:

• ASCII control characters for CSV files.
• Reference file with the expected table schema for creating external tables with Avro, ORC, and Parquet files.

These features are generally available (GA).

November 21, 2022

A weekly digest of client library updates from across the Cloud SDK.

Python

Changes for google-cloud-bigquery

3.4.0 (2022-11-17)

Features

• Add reference_file_schema_uri to LoadJobConfig, ExternalConfig (#1399) (931285f)
• Add default value expression (#1408) (207aa50)
• Add More Specific Type Annotations for Row Dictionaries (#1295) (eb49873)

November 17, 2022

Object tables are now in preview. Object tables are read-only tables containing metadata for unstructured data stored in Cloud Storage. These tables enable you to analyze and perform inference on images, audio files, documents, and other file types by using BigQuery ML and BigQuery remote functions. Object tables extend structured data features such as data security and governance best practices to unstructured data.

Metadata caching is now in preview. Using cached metadata might improve query performance for BigLake tables and object tables that reference large numbers of objects, by allowing the query to avoid listing objects from Cloud Storage.

November 14, 2022

A weekly digest of client library updates from across the Cloud SDK.

Java

Changes for google-cloud-bigquery

2.19.1 (2022-11-08)

Dependencies

• Update dependency com.google.cloud:google-cloud-storage to v2.15.0 (#2402) (aac2711)

2.19.0 (2022-11-07)

Features

• Add getTimestampInstant() method to FieldValue (#2350) (113303f)

Dependencies

• Update dependency com.google.apis:google-api-services-bigquery to v2-rev20221028-2.0.0 (#2393) (d3f6a6b)
• Update dependency com.google.cloud:google-cloud-shared-dependencies to v3.0.6 (#2399) (9de9aa8)
• Update dependency org.graalvm.buildtools:junit-platform-native to v0.9.17 (#2396) (87f8cdd)
• Update dependency org.graalvm.buildtools:native-maven-plugin to v0.9.17 (#2397) (7927350)

The slot estimator helps you manage slot capacity based on historical performance metrics. This feature is now generally available (GA).

November 09, 2022

You can now transfer data from Amazon S3 and Azure Blob Storage to BigQuery using the LOAD DATA statement. This feature is generally available (GA) and includes support for the following features:

• Transfer files that are hive partitioned.
• Load semi-structured JSON source data into BigQuery without providing a schema by using JSON columns in the destination table.
• Encrypt destination tables using customer managed encryption keys.
• Transfer data to US multi-region and US-EAST-4 regions.

November 07, 2022

A weekly digest of client library updates from across the Cloud SDK.

Python

Changes for google-cloud-bigquery

3.3.6 (2022-11-02)

Features

• Reconfigure tqdm progress bar in %%bigquery magic (#1355) (506f781)

Bug Fixes

• Corrects test for non-existent attribute (#1395) (a80f436)
• deps: Allow protobuf 3.19.5 (#1379) (3e4a074)
• deps: Allow pyarrow < 11 (#1393) (c898546)
• deps: Require requests>=2.21.0 (#1388) (e398336)
• Refactor to adapt to changes to shapely dependency (#1376) (2afd278)

Documentation

• Fix typos (#1372) (21cc525)

Miscellaneous Chores

• release 3.3.6 (4fce1d9)

In the Explorer pane, you can now star your projects, datasets, and tables. This feature replaces the pin feature, which formerly allowed you to pin projects to the Explorer pane. This feature is now generally available (GA).

In the Cloud console, the Add data feature lets you access popular ways to search for and ingest data sources that work with BigQuery. For an example, see viewing listings in Analytics Hub. This feature is now generally available (GA).

November 03, 2022

SQL functions for managing wrapped keysets are generally available (GA). You can now perform the following actions natively in BigQuery with fewer risks and steps:

• Create a wrapped keyset
• Rotate a wrapped keyset
• Rewrap a wrapped keyset
• Encrypt and decrypt a column with a wrapped keyset

Included with this release are the following new key management functions:

• KEYS.NEW_WRAPPED_KEYSET
• KEYS.ROTATE_WRAPPED_KEYSET
• KEYS.REWRAP_KEYSET

November 02, 2022

The query execution graph is now in preview. You can use the query execution graph to diagnose query performance issues, and to receive query performance insights.

November 01, 2022

The BigQuery migration assessment is now available for Amazon Redshift in preview. You can use this feature to assess the complexity of migrating from your Amazon Redshift data warehouse to BigQuery.

October 31, 2022

A weekly digest of client library updates from across the Cloud SDK.

Go

Changes for bigquery/storage/apiv1beta1

1.43.0 (2022-10-24)

Features

• bigquery/analyticshub: rename nodejs analyticshub library package name (52dddd1)
• bigquery/storage/managedwriter: Enable field name indirection (#6247) (1969273)
• bigquery/storage/managedwriter: Retry on FailedPrecondition (#6761) (d1a444d)
• bigquery/storage/managedwriter: Support append retries (#6695) (6ae9c67)
• bigquery/storage/managedwriter: Switch to opt-in retry (#6765) (a3e97a6)
• bigquery: Add remote function options to routine metadata (#6702) (d9a437d)

Bug Fixes

• bigquery/storage/managedwriter: Address possible resource leak (#6775) (979440b)
• bigquery: Avoid stack overflow on query param with recursive types (#6890) (854ccfc), refs #6884
• bigquery: Bq connection auth scopes (#6752) (8e09288), refs #6744

Documentation

• bigquery/storage/managedwriter/adapt: Typo in error string (#6729) (bb26153)
• bigquery/storage/managedwriter: Add retry info to package docs (#6803) (81e52e5)

Java

Changes for google-cloud-bigquery

2.18.2 (2022-10-28)

Dependencies

• Remove duplicated deps in the pom. (#2383) (fe164aa)

2.18.1 (2022-10-28)

Dependencies

• Remove version declaration for gson (#2379) (0908652)

2.18.0 (2022-10-27)

Features

• Add executeSelectAsync and Refactor (#2294) (80fa478)

Bug Fixes

• Add --add-opens arg to native-image command (#2369) (8e8b6d7)
• Properly handle external table schema on table update (#2236) (460ef31)

Dependencies

• Update arrow.version to v10 (major) (#2371) (b7873db)
• Update cloud client dependencies (#2362) (0936699)
• Update dependency com.google.api.grpc:proto-google-cloud-bigqueryconnection-v1 to v2.6.0 (#2355) (7bc59a7)
• Update dependency com.google.api.grpc:proto-google-cloud-bigqueryconnection-v1 to v2.7.0 (#2366) (02102d3)
• Update dependency com.google.apis:google-api-services-bigquery to v2-rev20221015-2.0.0 (#2370) (9b796cf)
• Update dependency com.google.cloud:google-cloud-datacatalog-bom to v1.10.0 (#2356) (edb2ca0)
• Update dependency com.google.cloud:google-cloud-shared-dependencies to v3.0.5 (#2361) (51b2258)
• Update dependency com.google.code.gson:gson to v2.10 (#2367) (82e3de5)
• Update dependency org.graalvm.buildtools:junit-platform-native to v0.9.15 (#2352) (b0f172c)
• Update dependency org.graalvm.buildtools:junit-platform-native to v0.9.16 (#2358) (f4e5fc5)
• Update dependency org.graalvm.buildtools:native-maven-plugin to v0.9.15 (#2353) (ac9226c)
• Update dependency org.graalvm.buildtools:native-maven-plugin to v0.9.16 (#2359) (52ec31a)

The max_staleness materialized view option helps you achieve consistently high performance with controlled costs when processing large, frequently changing datasets. This feature is now in preview.

Column-level data masking is now generally available (GA). You can use data masking to selectively obscure column data for groups of users, while still allowing access to the column.

October 27, 2022

Search indexes and the SEARCH() function are now generally available (GA). These enable you to use Google Standard SQL to efficiently pinpoint specific data elements in unstructured text and semi-structured data.

October 26, 2022

The following geography functions are now generally available (GA):

• ST_ISCLOSED: Returns TRUE for a non-empty geography, where each element in the geography has an empty boundary.
• ST_ISRING: Checks if a geography is a linestring and if the linestring is both closed and simple.

October 24, 2022

You can now view BI Engine Top Tables Cached Bytes, BI Engine Query Fallback Count, and Query Execution Count as dashboard metrics for BigQuery. This feature is now in preview.

October 18, 2022

Remote functions, which let you invoke functions from Cloud Functions or Cloud Run in your Google Standard SQL queries, are now generally available (GA).

October 17, 2022

A weekly digest of client library updates from across the Cloud SDK.

Java

Changes for google-cloud-bigquery

2.17.1 (2022-10-10)

Dependencies

• Update cloud client dependencies (#2335) (f8053d7)
• Update cloud client dependencies (#2337) (1194eac)
• Update dependency com.google.api.grpc:proto-google-cloud-bigqueryconnection-v1 to v2.5.6 (#2336) (a86c759)

Python

Changes for google-cloud-bigquery

3.3.5 (2022-10-10)

Bug Fixes

• deps: Allow protobuf 3.19.5 (#1379) (3e4a074)

3.3.5 (2022-10-10)

Bug Fixes

• deps: Allow protobuf 3.19.5 (#1379) (3e4a074)

October 12, 2022

The reporting process for the tabledata.list bytes per minute quota has been updated to more accurately reflect the enforced limit. The limit has not changed.

October 11, 2022

Analytics Hub is now generally available. As an Analytics Hub publisher, you can now view all subscriptions to your listing and remove a subscription from your listing.

You can now use stored procedures for Apache Spark. This feature is in preview.

October 10, 2022

Multi-statement transactions are now generally available (GA).

A weekly digest of client library updates from across the Cloud SDK.

Java

Changes for google-cloud-bigquery

2.17.0 (2022-10-03)

Features

• Add remote function options to routine metadata (#2291) (d30670e)

Dependencies

• Update dependency com.google.api.grpc:proto-google-cloud-bigqueryconnection-v1 to v2.5.5 (#2328) (6e48ec2)
• Update dependency com.google.apis:google-api-services-bigquery to v2-rev20220913-2.0.0 (#2287) (fa33184)
• Update dependency com.google.apis:google-api-services-bigquery to v2-rev20220924-2.0.0 (#2325) (82c2097)
• Update dependency com.google.cloud:google-cloud-shared-dependencies to v3.0.4 (#2327) (188c779)
• Update dependency gcp-releasetool to v1.8.9 (#2326) (52dfd13)
• Update dependency importlib-metadata to v4.13.0 (#2323) (4c7e089)
• Update dependency importlib-metadata to v5 (#2324) (bd43cf4)
• Update dependency org.graalvm.buildtools:junit-platform-native to v0.9.14 (#2288) (959519c)
• Update dependency org.graalvm.buildtools:native-maven-plugin to v0.9.14 (#2289) (3cf7ef8)
• Update dependency org.junit.vintage:junit-vintage-engine to v5.9.1 (#2285) (65fac18)

Python

Changes for google-cloud-bigquery

3.3.4 (2022-09-29)

Bug Fixes

• deps: Require protobuf >= 3.20.2 (#1369) (f13383a)

The ability to use physical bytes for storage billing is now in Preview. For more information, see Dataset storage billing models.

October 05, 2022

You can now explore query results in Colab using Python libraries. This feature is now in preview.

Concurrent connections quotas are now based on the project that initiates the Storage Write API request, not the project containing the BigQuery dataset resource.

October 03, 2022

A weekly digest of client library updates from across the Cloud SDK.

Python

Changes for google-cloud-bigquery

3.3.3 (2022-09-28)

Bug Fixes

• Refactors code to account for a tdqm code deprecation (#1357) (1369a9d)
• Validate opentelemetry span job attributes have values (#1327) (8287af1)

Documentation

• samples: uses function (create_job) more appropriate to the described sample intent (5aeedaa)

BigQuery supports JSON as a native column type. This feature is now generally available (GA).

September 29, 2022

In addition to standard rounding, BigQuery now supports the rounding mode ROUND_HALF_EVEN for parameterized NUMERIC or BIGNUMERIC columns. The ROUND() function also accepts the rounding mode as an optional argument. This feature is now in preview.

September 28, 2022

With Datastream for BigQuery, you can now replicate data and schema updates from operational databases directly into BigQuery. This feature is now in preview.

September 26, 2022

The totalItems field returned by the projects.list API method now returns the number of items per page, rather than an approximate total number of projects across all pages.

A weekly digest of client library updates from across the Cloud SDK.

Go

Changes for bigquery/storage/apiv1beta1

1.42.0 (2022-09-21)

Features

• bigquery/analyticshub: Start generating apiv1 (#6707) (feb7d7d)
• bigquery/datapolicies: Start generating apiv1beta1 (#6697) (f5443e8)
• bigquery/reservation/apiv1beta1: add REST transport (f7b0822)
• bigquery/storage/managedwriter: Define append retry predicate (#6650) (478b8dd)
• bigquery/storage: add proto annotation for non-ascii field mapping (ec1a190)
• bigquery: Add reference file schema option for federated formats (#6693) (3d26091)
• bigquery: Add support for explicit query parameter type (#6596) (d59b5b2), refs #4704

Bug Fixes

• bigquery/connection: integrate gapic-generator-python-1.4.1 and enable more py_test targets (ec1a190)

In the Explorer pane, you can now open tables in Connected Sheets. This feature is now generally available (GA).

September 21, 2022

BigQuery Omni has introduced support for on-demand pricing model (GA) for a limited duration. For more information, see BigQuery Omni Pricing.

You can now view shuffle usage ratios in the admin resource charts. This feature is now in preview.

September 19, 2022

A weekly digest of client library updates from across the Cloud SDK.

Go

Changes for bigquery/storage/apiv1beta1

1.41.0 (2022-09-14)

Features

• bigquery/storage: add location to WriteStream and add WriteStreamView support (6a0080a)

Bug Fixes

• bigquery/storage/managedwriter: Fix incorrect error retention (#6659) (dc02bca)
• bigquery: Parse timestamp query parameter with RFC3339 (#6653) (aabd2d6)

Java

Changes for google-cloud-bigquery

2.16.1 (2022-09-15)

Dependencies

• Update dependency com.google.cloud:google-cloud-shared-dependencies to v3.0.3 (#2274) (4c9952b)

2.16.0 (2022-09-12)

Features

• Add preserveAsciiControlCharacters to CsvOptions (#2143) (856893f)
• Add reference file schema option for federated formats (#2269) (8c488e6)

Bug Fixes

• Socket-timeout at bigquery.it.ITNightlyBigQueryTest: testForTableNotFound (#2260) (a9b5fb2)

Dependencies

• Update dependency com.google.apis:google-api-services-bigquery to v2-rev20220827-2.0.0 (#2261) (3c67d21)
• Update dependency com.google.cloud:google-cloud-datacatalog-bom to v1.9.3 (#2259) (5e30a04)
• Update dependency com.google.cloud:google-cloud-shared-dependencies to v3.0.2 (#2267) (8472fe5)

September 16, 2022

BigQuery Omni now supports the following quota and limit:

• The quota for total query result sizes for a project is now 1 TB per day. For more information, see Query jobs.
• The limit for maximum result size for a query has been increased from 2 MB to 10 GB (preview).

For more information, see Limitations.

September 15, 2022

The BigQuery Data Transfer Service for Google Ads now supports the new Google Ads API. This feature is now in preview.

BigQuery is now available in the Madrid (europe-southwest1), Milan (europe-west8), and Paris (europe-southwest1) regions. The Madrid and Paris regions have the lowest carbon impact.

BigQuery ML is now available in the Madrid (europe-southwest1), Milan (europe-west8), and [Paris (europe-southwest1)]https://cloud.google.com/bigquery/docs/locations#regional-locations) regions. The Madrid and Paris regions have the lowest carbon impact.

September 14, 2022

The Merge is coming! You may experience disruptions in the Ethereum public datasets in BigQuery.

The is_case_insensitive schema option, which allows you to make a dataset and its table names case-insensitive, is now in preview.

JDBC driver update, release 1.3.0 1001

• You can now configure the connector to authenticate the connection using an external account (workforce or workload identity federation).
• You can now configure the connector to use Private Service Connect URLs.
• The connector now supports JDBC transaction APIs. BigQuery supports multi-statement transactions inside a single query, or across multiple queries, when using sessions.
• The connector is now verified to use a default project for datasets. To do this, set the dataset_project_id property in QueryProperties of the connection string to the desired project.
• MATERIALIZED_VIEW has been added to the list of table types when using the getTableTypes function.
• The connector now supports the JSON data type.

ODBC driver update, release 2.5.0 1001

• You can now configure the connector to authenticate the connection using an external account (workforce or workload identity federation), with limited support, using Azure AD and Okta identity providers.
• You can now configure the connector to use Private Service Connect URLs.
• The connector now supports ODBC transaction APIs. BigQuery supports multi-statement transactions inside a single query, or across multiple queries, when using sessions.
• The connector is now verified to use a default project for datasets. To do this, set the dataset_project_id property in QueryProperties of the connection string to the desired project.
• MATERIALIZED_VIEW has been added to the list of table types. To retrieve these table types, configure SQLTables to TABLE_TYPES_ONLY.
• The connector now supports the JSON data type.

September 13, 2022

In Cloud Monitoring, you can view metrics for quota usage and limits of the Storage Write API's concurrent connections and throughput quotas. This feature is now generally available (GA).

September 12, 2022

A weekly digest of client library updates from across the Cloud SDK.

Node.js

Changes for @google-cloud/bigquery

6.0.3 (2022-08-23)

Bug Fixes

• remove pip install statements (#1144) (328f3c0)

Go

Changes for bigquery/storage/apiv1beta1

1.40.0 (2022-09-08)

⚠ BREAKING CHANGES

• bigquery/dataexchange: update BigQuery Analytics Hub API v1beta1 client BREAKING CHANGE: refresh current dataexchange/v1beta1/* directory to include recent change in protos. Removed common directory and use local enum Category fix!: refactor references to Category message docs: improve proto documentation.

Features

• bigquery/dataexchange: update BigQuery Analytics Hub API v1beta1 client BREAKING CHANGE: refresh current dataexchange/v1beta1/* directory to include recent change in protos. Removed common directory and use local enum Category fix!: refactor references to Category message docs: improve proto documentation. (e45ad9a)
• bigquery/storage/managedwriter: Augment reconnection logic (#6609) (6b0ac0c)
• bigquery: Add trace instrumentation support for individual rpcs (#6493) (eedc632)
• bigquery: Improve error when reading null values (#6566) (e9a94c2), refs #2612

Documentation

• bigquery: Add numeric and bignumeric to RowIterator docs (#6560) (bea4028)

Miscellaneous Chores

• bigquery: Release 1.40.0 (#6635) (628deae)

September 09, 2022

The following features are now generally available for ARIMA_PLUS models:

• The HOLIDAY_REGION option can now take more than one region string as input. If you include more than one region string, the union of the holidays in all of the provided regions will be taken into the modeling.
• You can use the new TREND_SMOOTHING_WINDOW_SIZE option to smooth the trend component of the time series by applying a center moving average.

September 06, 2022

Cloud console updates: Improvements that are related to query execution include the following:

• For long-running queries, the Execution details tab is automatically displayed with the timing details of each stage of the query.

• In the query editor, you can now see the query validation message when your query is completed or canceled.

September 01, 2022

Customer-managed encryption keys are now integrated with CMEK organization policies. This feature is generally available (GA).

The slot recommender creates recommendations for customers using on-demand billing and is now generally available (GA).

The Random Forest model is now generally available (GA). For more information, see the random forest sections in the end-to-end user journey page.

August 29, 2022

A weekly digest of client library updates from across the Cloud SDK.

Go

Changes for bigquery/storage/apiv1beta1

1.39.0 (2022-08-23)

Features

• bigquery/storage: allow users to set Apache Avro output format options through avro_serialization_options param in TableReadOptions message Through AvroSerializationOptions, users can set enable_display_name_attribute, which populates displayName for every avro field with the original column name Improved documentation for selected_fields, added example for clarity. (41ab4ec)
• bigquery: add PreserveAsciiControlCharacters support for CSV (#6448) (b7bac2f)
• bigquery: add preview support for default values (#6464) (edc3be5)

Java

Changes for google-cloud-bigquery

2.14.7 (2022-08-23)

Bug Fixes

• table-not-found issue with executeSelect while running long queries (#2222) (4876569)

2.15.0 (2022-08-25)

Features

• add preview support for default values (#2244) (fd3d3c5)

Community contributed UDFs are now generally available in the bigquery-utils GitHub repository and the bigquery-public-data.persistent_udfs public dataset.

Cloud console updates: In the query editor, when you select a function signature from the autocomplete list, you can remove the parameter names quickly by pressing the Backspace or Delete key.

August 22, 2022

A weekly digest of client library updates from across the Cloud SDK.

Go

Changes for bigquery/storage/apiv1beta1

1.38.0 (2022-08-12)

Features

• bigquery/migration: Add MySQL dialect to bigquerymigration v2 client library (370e23e)
• bigquery/storage/managedwriter: improve error communication (#6360) (b30d89d)

Bug Fixes

• bigquery/storage/managedwriter: propagate calloptions to append (#6488) (c65f9da)

Python

Changes for google-cloud-bigquery

3.3.2 (2022-08-16)

Bug Fixes

• deps: require proto-plus >= 1.22.0 (1de7a52)
• deps: require protobuf >=3.19, < 5.0.0 (#1311) (1de7a52)

August 19, 2022

The ALTER TABLE RENAME COLUMN DDL statement, which allows you to rename the columns of a table, is now in preview.

August 17, 2022

You can now set default values on columns in your BigQuery tables. This feature is now in preview.

Cloud console updates: You can now copy BigQuery metadata to your clipboard by using the following options:

• In the Schema view, to copy a table's schema, select any fields, and then click Copy.

• In the Explorer pane, to copy the ID of a resource, click View actions, and then click Copy ID.

Cloud console updates: Improvements include the following:

• Query results are now displayed in resizable columns.

• Tab titles now expand when space is available for longer names.

• Tooltips no longer display text immediately when you hold the pointer over them, avoiding unnecessary distraction.

• In the Explorer pane, you can now access saved queries by expanding your project. The Saved Queries pane is no longer at the bottom of the console.

• In the Explorer pane, you can now find a table by searching for mydataset.mytable.

• In the query editor, you can now press the F1 shortcut key to view more editor shortcuts.

August 16, 2022

Workforce identity federation lets you authenticate and authorize users from external identity providers to access supported Google Cloud products, including BigQuery resources. This feature is now in preview.

August 15, 2022

A weekly digest of client library updates from across the Cloud SDK.

Python

Changes for google-cloud-bigquery

3.3.1 (2022-08-09)

Bug Fixes

• deps: allow pyarrow < 10 (#1304) (13616a9)

An issue was identified in the max_staleness materialized view option. This feature is not available.

BigQuery Omni now supports reservation DDL and access control DCL. This feature is now generally available (GA).

Previously, you could commit up to 100 GB in streaming bytes for every Storage Write API pending mode commit that you triggered in regions other than the US and EU multi-regions. This limit is now 1 TB. For more information, see Storage Write API quotas.

A weekly digest of client library updates from across the Cloud SDK.

Java

Changes for google-cloud-bigquery

2.14.6 (2022-08-12)

Dependencies

• update dependency com.google.cloud:google-cloud-datacatalog-bom to v1.9.2 (#2221) (3292cdd)

2.14.5 (2022-08-12)

Dependencies

• update dependency com.google.apis:google-api-services-bigquery to v2-rev20220806-2.0.0 (#2223) (05d1de1)

2.14.4 (2022-08-08)

Dependencies

• update dependency com.google.cloud:google-cloud-storage to v2.11.3 (#2213) (a293ab5)

August 10, 2022

You can now set default configurations at a project or organization level. This feature is now generally available (GA).

You can now manage query execution priority for Cloud Spanner federated queries. This feature is now generally available (GA).

August 09, 2022

Querying Google Cloud Bigtable external data sources is now generally available (GA).

August 08, 2022

A weekly digest of client library updates from across the Cloud SDK.

Go

Changes for bigquery/storage/apiv1beta1

1.37.0 (2022-08-04)

Features

• bigquery/connection: Add service_account_id output field to CloudSQL properties (1d6fbcc)
• bigquery/storage/managedwriter: refactor AppendResponse (#6402) (c07bca2)
• bigquery: support JSON as a data type (#5986) (835fe4f)

Bug Fixes

• bigquery: include user_email field when requesting job information (#6256) (da42b4e)

Documentation

• bigquery/storage: clarify size limitations for AppendRowsRequest chore: add preferred_min_stream_count to CreateReadSessionRequest chore: add write_stream to AppendRowsResponse (1d6fbcc)

A weekly digest of client library updates from across the Cloud SDK.

Java

Changes for google-cloud-bigquery

2.14.3 (2022-08-05)

Dependencies

• update dependency com.google.apis:google-api-services-bigquery to v2-rev20220730-2.0.0 (#2208) (5165e2b)
• update dependency com.google.cloud:google-cloud-storage to v2.11.2 (#2207) (da5389d)

2.14.2 (2022-08-04)

Dependencies

• update arrow.version to v9 (major) (#2201) (3ec5ef9)
• update dependency com.google.apis:google-api-services-bigquery to v2-rev20220716-2.0.0 (#2202) (c1ca09e)
• update dependency com.google.cloud:google-cloud-bigquerystorage-bom to 2.18.0 (c1ca09e)
• update dependency com.google.cloud:google-cloud-datacatalog-bom to 1.9.1 (c1ca09e)
• update dependency com.google.cloud:google-cloud-shared-dependencies to v3 (c1ca09e)
• update dependency com.google.cloud:google-cloud-storage to 2.11.0 (c1ca09e)
• update dependency com.google.cloud:google-cloud-storage to v2.11.1 (#2194) (45be001)
• update dependency com.google.code.gson:gson to v2.9.1 (#2190) (4bd4539)
• update dependency org.threeten:threeten-extra to v1.7.1 (c1ca09e)

Documentation

• owlbot-java: explaining why not using formatter in pom.xml (#1511) (#2195) (7c45aa5), closes #1502

August 03, 2022

The max_staleness materialized view option helps you achieve consistently high performance with controlled costs when processing large, frequently changing datasets. This feature is now in preview.

August 01, 2022

A weekly digest of client library updates from across the Cloud SDK.

Python

Changes for google-cloud-bigquery

3.3.0 (2022-07-25)

Features

• add destination_expiration_time property to copy job (#1277) (728b07c)

Bug Fixes

• require python 3.7+ (#1284) (52d9f14)

Documentation

• samples: add table snapshot sample (#1274) (e760d1b)
• samples: explicitly add bq to samples reqs, upgrade grpc to fix bug on m1 (#1290) (9b7e3e4)

The trigonometric SQL function CBRT is now generally available (GA). With this function, you can compute the cube root of a value.

The LOAD DATA statement is now available for Preview in Google Standard SQL for BigQuery. You can use the LOAD DATA statement to load data from one or more files into a table.

A weekly digest of client library updates from across the Cloud SDK.

Java

Changes for google-cloud-bigquery

2.14.1 (2022-07-27)

Dependencies

• update dependency org.junit.vintage:junit-vintage-engine to v5.9.0 (#2183) (f8325cf)

July 28, 2022

You can now create BigQuery subscriptions in Pub/Sub to write messages directly to an existing BigQuery table.

July 27, 2022

Inverse trigonometric SQL functions are now generally available (GA). These functions include:

• COT: Compute the cotangent for an angle.
• COTH: Compute the hyperbolic cotangent for an angle.
• CSC: Compute the cosecant for an angle.
• CSCH: Compute the hyperbolic cosecant for an angle.
• SEC: Compute the secant for an angle.
• SECH: Compute the hyperbolic secant for an angle.

July 25, 2022

BigLake is now generally available (GA). You can now create BigQuery ML models using data in Cloud Storage by using BigLake and publish BigLake tables as Analytics Hub listings.

The new Migrate section in the BigQuery documentation helps you migrate to BigQuery. This includes high-level guidance with a migration overview, an introduction to free-to-use tools that help you with each phase of migration, and platform-specific migration guides.

July 20, 2022

Analytics Hub is now available in additional regions across the Americas, Asia Pacific, and Europe. For more information, see Analytics Hub supported regions.

July 14, 2022

Previously, the Storage Write API had a maximum concurrent connection limit of 100 connections for non-multi-regions such as Montreal (northamerica-northeast1). This limit has now been increased to 1,000 connections across all non-multi-regions. For more information, see Storage Write API quotas and limits.

July 12, 2022

You can now select a job type when assigning a folder, organization, or project to a reservation in the Cloud console. This feature is now generally available (GA).

July 11, 2022

The google.cloud.bigquery.reservation.v1beta1.api package is deprecated and will be removed on September 27, 2022. After that date, requests to that package will fail. Data created by using google.cloud.bigquery.reservation.v1beta1.api are accessible by using the google.cloud.bigquery.reservation.v1.api package.

Next steps:

• If you use the API directly, you should switch to google.cloud.bigquery.reservation.v1.api, the GA version of the API, to prevent any impact on your workflow.
• If you only use the Cloud console to manage BigQuery reservations, no action is needed.
• If you use the bq command-line tool to manage BigQuery reservations, upgrade the tool to the latest version.

July 07, 2022

Azure workload identity federation is now available in preview for BigQuery Omni connections. This feature helps you secure data by allowing you to grant Google access to an application you manage in your Azure tenant so that neither you nor Google must manage application client secrets.

July 06, 2022

The APPENDS change history TVF is now in preview. This table-valued function provides a history of table appends over a window of time.

July 01, 2022

An updated version of JDBC driver for BigQuery is now available. This version includes a fix for an issue with connector returning stack overflow in some cases when executing complex long queries.

June 29, 2022

You can now set the view field in the tables.get() API method to indicate which table information is returned. Setting the value to BASIC reduces latency by omitting some storage statistics.

Previously, all BigQuery BI Engine projects had a maximum reservation size per project per location limit of 100 GB. This limit is now 250 GB. For more information, see BI Engine quotas and limits.

June 23, 2022

The BI Engine preferred tables feature lets you limit BI Engine acceleration to a specified set of tables. This feature is now in preview.

June 21, 2022

Query queues are now available in preview for on-demand and flat-rate customers. When query queues are enabled, BigQuery automatically determines the query concurrency rather than setting a fixed limit. Flat-rate customers can override this setting with a custom concurrency target. Additional queries beyond the concurrency target are queued until processing resources become available.

June 15, 2022

Deterministic encryption SQL functions are now generally available (GA). New AEAD encryption functions include DETERMINISTIC_ENCRYPT, DETERMINISTIC_DECRYPT_BYTES, and DETERMINISTIC_DECRYPT_STRING. These functions allow column-level encryption and decryption of data while supporting aggregation and table joins.

June 14, 2022

You can now use the Cloud console to set up VPC service control perimeters to restrict access from BigQuery Omni to external clouds. You can also specify whether you want to grant read or write permission on your external resource. This feature is now generally available (GA).

You can now explore data in Data Studio by using links from your BigQuery query results in the Cloud console. This feature is now generally available (GA).

June 13, 2022

A new system variable, @@dataset_project_id, is now generally available. @@dataset_project_id allows you to set a default project where one is not specified for a dataset in your query. This variable is also available as a Connection Property.

June 10, 2022

Quotas for multi-statement queries have changed. The cumulative time limit for a multi-statement query has increased from 6 hours to 24 hours.

June 08, 2022

Batch and interactive translation services are now generally available (GA), and include support for most major SQL dialects. This release also includes preview availability of SQL object name mapping and metadata extraction tools that you can use to increase the accuracy of your batch translation jobs.

June 06, 2022

You can now attach Resource Manager tags to datasets. This feature is supported in Preview. Tags let you conditionally apply Identity and Access Management (IAM) policies to resources.

The following Storage Read API quotas and limits have changed:

• There is now a limit of 2,000 concurrent ReadRows calls per project in the US and EU multi-regions and 400 concurrent ReadRows calls in other regions.
• The number of data plane requests per user per project per minute has increased from 5,000 to 25,000.

For more information, see Storage Read API quotas and limits.

June 03, 2022

BigQuery Omni now supports Reservation and Access Control DCL. This feature is in Preview.

May 31, 2022

Column-level data masking is now available in preview. You can use data masking to selectively obscure column data for groups of users, while still allowing access to the column. When you use data masking in combination with column-level access control, you can configure a range of access to column data, from full access to no access, based on the requirements of different groups of users.

May 24, 2022

You can now load data into BigQuery using Informatica Data Loader. This feature is generally available. Informatica provides connectors that can ingest data into BigQuery.

May 23, 2022

Metrics for query/statement_scanned_bytes and query/statement_scanned_bytes_billed are no longer delayed for 6 hours in order to smooth reporting over the duration of the job. Values are now reported every 180 seconds without smoothing. For more information about metrics, see Google Cloud metrics.

May 18, 2022

Updated versions of ODBC and JDBC drivers for BigQuery are now available that include enhancements.

May 05, 2022

The new format element %J is generally available (GA) for DATE, TIME, DATETIME, and TIMESTAMP functions. This format element lets you use the ISO 8601 1-based day of the year.

PARSE_DATE, PARSE_TIME, PARSE_DATETIME, and PARSE_TIMESTAMP now support the following date and time format elements: %a, %A, %g, %G, %j, %u, %U, %V, %w, and %W.

May 03, 2022

The following new features are now generally available (GA) for ARIMA_PLUS models:

• You can use ML.EVALUATE to calculate new forecasting accuracy metrics such as MAPE, SMAPE, and MSE.
• You can perform fast model training with little or no loss of forecasting accuracy by using the TIME_SERIES_LENGTH_FRACTION, MIN_TIME_SERIES_LENGTH and MAX_TIME_SERIES_LENGTH options.

To learn how to achieve one hundred times higher scalability with the ARIMA_PLUS model while using the new forecasting accuracy metrics, see the Accelerate ARIMA_PLUS to forecast 1 million time series within hours. You can also read ARIMA_PLUS best practices.

May 02, 2022

Case-insensitive collation support for BigQuery is now available for Preview. Collation determines how strings are sorted and compared in collation-supported operations. If case-insensitive collation is used, case is ignored in comparison and sorting operations.

These operations support collation:

• Several comparison operations
• Join operations
• ORDER BY operations
• GROUP BY operations
• Several scalar and aggregate function operations
• Set operations

The COLLATE function is now available for Preview in Google Standard SQL for BigQuery. With the COLLATE function, you can pass in a STRING and return a STRING with a collation specification.

The DEFAULT COLLATE clause is now available for Preview. With this clause, the default collation specification is applied to all column data types supporting collation. You can use the DEFAULT COLLATE clause in the following DDL statements:

• CREATE SCHEMA and ALTER SCHEMA
• CREATE TABLE and ALTER TABLE

The COLLATE clause is now available for Preview. With this clause, a collation specification is applied to a specific column in a table. You can use the COLLATE clause in the following DDL statements:

• ALTER TABLE ADD COLUMN
• ALTER COLUMN SET DATA TYPE

April 25, 2022

The ability to configure the time travel window is now in Preview. You can specify the duration of the time travel window, from a minimum of two days to a maximum of seven days.

Use the TABLE_STORAGE view to get a snapshot of current storage usage for tables and materialized views. This feature is now in Preview.

BigQuery Admin Resource Charts are now generally available (GA) for on-demand users, enabling administrators to monitor key metrics and troubleshoot issues across the entire organization. Previously, it was only available for reservation users. A new permission, bigquery.jobs.listExecutionMetadata, has been added to make it easier to gain access to the full UI.

April 11, 2022

Starting in July 2022, the projects.list API method will return results in unsorted order. Currently, the API returns the results in sorted order, although this is not a documented behavior of the API.

April 07, 2022

BigLake is now available in Preview. BigLake is a storage engine that allows you to query and unify cross-cloud data lakes and warehouses. Additionally, it provides fine-grained access controls to your tables, allowing you to set access policies on a column or row basis.

BigQuery now supports the creation of search indexes and a SEARCH function. This feature is in Preview. This enables you to use Google Standard SQL to efficiently find data elements in unstructured text and semi-structured data.

April 06, 2022

Analytics Hub is now available in Preview. Analytics Hub is a new service in BigQuery that lets you create secure data exchanges and share analytics assets within and across organizations. This platform allows data providers to publish listings that reference shared datasets. Analytics Hub subscribers can then view and subscribe to these listings.

April 05, 2022

BigQuery Omni now supports cross-cloud transfer. This feature is in Preview. For more information, see Cross-cloud transfer (AWS) and Cross-cloud transfer (Azure).

April 01, 2022

BigQuery ML and Vertex AI Model Registry integration is available in preview. With this integration, BigQuery ML models can be sent to the Vertex AI Model Registry where you can manage the lifecycle of all your ML models. From the Vertex AI Model Registry, you can organize your BigQuery ML models and deploy directly to endpoints.

March 31, 2022

The international public dataset for Data Signals for Google Search Trends is now available in Preview and available in the Google Cloud Marketplace and Analytics Hub.

March 28, 2022

The Wide-and-Deep model is now generally available (GA). For more information, see the Wide-and-Deep sections in the end-to-end user journey page.

March 17, 2022

The BigQuery slot recommender is now available in Preview. The slot recommender creates recommendations for customers using on-demand billing. These recommendations help you to understand the cost and performance tradeoffs of purchasing different amounts of slot capacity.

March 16, 2022

You can now explicitly specify a schema for BigQuery external tables created over Parquet, ORC, and Avro file formats. Previously, the schema was always auto-detected using the last lexicographic file.

March 04, 2022

Session support for BigQuery is now generally available (GA). In addition to the features available in the preview, you can:

• Terminate a session automatically or manually.
• Set a label for all queries in a session.
• Get sessions metadata in INFORMATION_SCHEMA.SESSIONS_BY_PROJECT and INFORMATION_SCHEMA.SESSIONS_BY_USER views.

February 16, 2022

Remote functions are now available for preview. Remote functions allow you to implement your function in other languages than SQL and Javascript, or with libraries or services which are not allowed in BigQuery user-defined functions.

February 15, 2022

The table clones feature in BigQuery is now in Preview. A table clone is a lightweight, writable copy of a table. You are only charged for storing the data in a table clone that differs from its base table.

February 14, 2022

The INFORMATION_SCHEMA.STREAMING_TIMELINE_* views are now generally available (GA).

The QUALIFY clause, which lets you filter the results of analytic functions in Google Standard SQL, is now generally available (GA).

BigQuery ML time series ARIMA_PLUS now trains models 5 times faster than previous training.

BigQuery reliability guide is now available. This guide describes how to build solutions with BigQuery that meet your application's needs for availability, durability, consistency, and data recovery. Topics include the following:

• Import reliability - Managed storage, methods, load jobs, and the Storage Write API
• Query reliability - Slots, reservations, and job optimization.
• Read reliability - Read methods, consistency concerns including quotas and limits, and the Storage Read API.
• Disaster planning - Disaster considerations and their mitigation.

February 10, 2022

BigQuery Omni now supports INFORMATION_SCHEMA.JOBS_* and INFORMATION_SCHEMA.RESERVATION* views. This feature is in Preview. For more information, see View resource metadata (AWS) and View resource metadata (Azure).

February 03, 2022

The BigQuery migration assessment is now available in Preview. Use this feature to assess the complexity of migrating from your current data warehouse to BigQuery.

BigQuery ML Hyperparameter tuning is now generally available (GA). You can use this feature to improve model performance by searching for the optimal hyperparameters when training ML models using CREATE MODEL statements.

To learn more, check out the following topics:

• BigQuery ML Hyperparameter Tuning Overview
• Using the BigQuery ML Hyperparameter Tuning to improve model performance
• End-to-end user journey for each model

February 02, 2022

The WITH RECURSIVE feature has been added to Google Standard SQL for BigQuery and is now in Preview. This feature allows a query in a WITH clause to refer to either itself or to queries defined later in the WITH clause.

January 31, 2022

BigQuery now supports materialized views without aggregation and materialized views with inner join. This feature is generally available (GA).

January 25, 2022

Explainable AI in BigQuery ML is now generally available (GA). This feature helps you understand BigQuery ML prediction or forecasting results at scale. For additional information about explainable AI, see the following:

• Explainable AI documentation
• Blog post: BigQuery Explainable AI helps you interpret your ML models
• Tutorials for regression, classification, and forecasting tasks

January 06, 2022

BigQuery standard SQL now supports the JSON data type for storing JSON data. The JSON data type is in Preview. For more information, see Working with JSON data in Standard SQL.

December 23, 2021

Documentation now includes a series of introductory topics to orient you to BigQuery including:

• What is BigQuery? - Product overview, available tools, and learning resources
• Storage - Infrastructure, ingestion, and optimization
• Analytics - Strategies, SQL queries, and BI tools
• Administration - Resources, workload management, security, and monitoring

In addition, the table of contents is updated to guide you through your staged BigQuery deployment with stages including: Discovery, Get started, Design, Ingest, Analyze, Administer, Secure, and Develop.

December 16, 2021

The row-level security feature now supports administrator access to historical data for tables with row-level access policies.

December 14, 2021

BigQuery BI Engine SQL interface is now generally available.

December 06, 2021

Anomaly detection in BigQuery ML is now generally available (GA). You can use the ML.DETECT_ANOMALIES function with the ARIMA_PLUS model to detect anomalies in time-series data. You can also use this function with the K-means, Autoencoder, or PCA models to detect anomalies in independent and identically distributed (IID) data.

December 03, 2021

The principal component analysis (PCA) model and the autoencoder model are now generally available (GA). You can use these models for common machine learning tasks such as dimensionality reduction, feature embedding, and unsupervised anomaly detection.

For more information, see the PCA and autoencoder sections in the end-to-end user journey page.

December 01, 2021

BigQuery Data Transfer Service now supports Audit Logging, Cloud Logging, and Cloud Monitoring. These features are in preview status.

November 19, 2021

Updated versions of ODBC and JDBC drivers for BigQuery are now available that include enhancements.

November 16, 2021

BigQuery is now available in the Santiago (southamerica-west1) region.

BigQuery ML is now available in the Santiago (southamerica-west1) region.

BigQuery BI Engine is now available in the Santiago (southamerica-west1) region.

BigQuery Data Transfer Service is now available in the Santiago (southamerica-west1) region.

November 12, 2021

BigQuery now supports authorized datasets (General Availability).

November 09, 2021

The following scripting statements have been added to Google Standard SQL for BigQuery.

• CASE: Executes the first list of SQL statements where a boolean expression is TRUE.
• CASE search_expression: Executes the first list of SQL statements where the search expression matches a WHEN expression.
• LABELS: Provides an unconditional jump to the end of the block or loop associated with a label.
• REPEAT: Repeatedly executes a list of SQL statements until the boolean condition at the end of the list is TRUE.
• FOR...IN: Loops over every row in a table expression.

These features are generally available (GA).

November 08, 2021

The following INFORMATION_SCHEMA views now support a DDL column. The value of the column is the DDL statement that can be used to create the resource.

• ROUTINES
• SCHEMATA
• TABLES

This feature is generally available (GA).

November 02, 2021

BigQuery now supports parameterized types. The following parameterized types are supported:

• STRING(L)
• BYTES(L)
• NUMERIC(P) / NUMERIC(P, S)
• BIGNUMERIC(P) / BIGNUMERIC(P, S)

This feature is generally available GA.

October 28, 2021

The table snapshots feature is now generally available (GA). It includes the Cloud console interface and support for creating a table snapshot in a different project from its base table.

October 27, 2021

SQL column-level encryption using Cloud Key Management Service (KMS) is now generally available (GA), letting you encrypt keysets within AEAD encryption functions.

October 25, 2021

BigQuery Omni, a multi-cloud analytics solution, is now generally available.

October 12, 2021

The BigQuery Storage Write API is now generally available (GA). The Storage Write API combines the functionality of high-throughput streaming ingestion and batch loading into a single API.

October 04, 2021

BigQuery Migration Service is now in Preview. It includes the following features:

• Interactive SQL Translator
• Batch SQL Translator

October 01, 2021

BigQuery pricing has changed as follows:

1. BigQuery Storage Read API has moved from a single regional SKU to a set of regional SKUs for bytes scanned. All BigQuery Storage Read API users can now read up to 300 TB of data per month at no charge. For more information, see BigQuery data extraction pricing.

2. BigQuery now charges BigQuery Storage Read API users for network egress. For more information, see BigQuery Storage Read API Network Egress Within Google Cloud.

BigQuery now supports the following geospatial data functions:

• ST_BUFFER: Returns a GEOGRAPHY that represents the buffer around the input GEOGRAPHY. You specify the number of segments to determine how much the resulting geography can deviate from the ideal buffer radius.

• ST_BUFFERWITHTOLERANCE: Returns a GEOGRAPHY that represents the buffer around the input GEOGRAPHY. You specify the tolerance to determine how much the resulting geography can deviate from the ideal buffer radius.

These functions are available as a preview.

September 28, 2021

Table functions are now generally available (GA). With the GA release, authorized table functions are now supported.

September 27, 2021

BigQuery now supports the following geospatial data functions:

• ST_BOUNDINGBOX: Returns a STRUCT that represents the bounding box for a geography.

• ST_EXTENT: Returns a STRUCT that represents the bounding box for a set of geographies.

• S2_COVERINGCELLIDS: Returns an array of S2 cell IDs that cover a geography.

• S2_CELLIDFROMPOINT: Returns the S2 cell ID covering a point geography.

These functions are generally available (GA).

September 21, 2021

When saving query results from the Cloud console to a CSV file, the available download size is now 10 MB. Previously the limit was 16,000 rows. Also, you can now download tables with nested and repeated data to CSV files.

September 17, 2021

BigQuery now supports the following geospatial data functions:

• ST_EXTERIORRING: Returns a linestring geography that corresponds to the outermost ring of a polygon geography.

• ST_INTERIORRINGS: Returns an array of linestring geographies that corresponds to the interior rings of a polygon geography.

• ST_ANGLE: Returns the angle between two intersecting lines.

• ST_AZIMUTH: Returns the azimuth of a line segment formed by two points.

• ST_NUMGEOMETRIES: Returns the number of geometries in a geography.

• ST_GEOMETRYTYPE: Returns the Open Geospatial Consortium (OGC) geometry type that describes a geography as a string.

These functions are generally available (GA).

September 16, 2021

BigQuery ML documentation has been updated with the following improvements:

• The end-to-end user journey now includes an overview of the machine-learning workflow for each available model.
• Each machine learning module now provides an overview document that describes the BigQuery ML behavior and links to additional guidance. New documentation includes the following:
  • Model creation overview
  • Preprocessing overview
  • Hyperparameter tuning overview
  • Model evaluation overview
  • Model inference overview
  • Explainable AI overview
  • Model weights overview
• Improvements to documentation organization and content, as well as the addition of new landing pages.

September 08, 2021

Deleting the metadata for a specific job using the bq command-line tool is now generally available (GA).

Session support for BigQuery is now in Preview. With sessions:

• You can associate your SQL activities in a session across scripts and multi-statement transactions in BigQuery with a unique session identifier.
• You can use session variables (for example, default timezone or dataset) and temporary tables throughout the life of the session and also across scripts and transactions
• When you enable sessions, all actions performed across multiple sessions can be viewed using the SESSION_ID column now available in jobs INFORMATION_SCHEMA views.

August 31, 2021

An updated version of ODBC driver for BigQuery is now available that includes enhancements.

August 30, 2021

Exporting table data in Parquet format is now generally available (GA).

August 23, 2021

BigQuery Admin Resource Charts are now generally available (GA) for reservation users, enabling administrators to more easily monitor and troubleshoot their BigQuery environment. They provide visibility into key metrics such as slot consumption, job concurrency, job execution time, job errors, and bytes processed across the entire organization.

BigQuery Slot Estimator is now in Preview for reservation users. This tool analyzes slot utilization data to help administrators estimate the right number of slots to purchase, and provides insights on how job performance might be impacted by adding or reducing slot capacity for the entire organization or specific reservations.

August 19, 2021

Cloud Spanner federated queries are now generally available (GA).

August 06, 2021

The principal component analysis (PCA) model is now available for preview. For more information, see CREATE MODEL statement for PCA models and the PCA details in the end-to-end user journey.

August 03, 2021

BigQuery now supports the ALTER COLUMN SET DATA TYPE data definition language (DDL) statement. This feature is generally available (GA).

BigQuery now supports the following data definition language (DDL) statement:

• CREATE TABLE LIKE
• CREATE TABLE COPY

This feature is generally available (GA).

BigQuery is now available in the Toronto (northamerica-northeast2) region.

BigQuery ML is now available in the Toronto (northamerica-northeast2) region.

BigQuery Data Transfer Service is now available in the Toronto (northamerica-northeast2) region.

BigQuery BI Engine is now available in the Toronto (northamerica-northeast2) region.

August 02, 2021

An updated version of JDBC driver for BigQuery is now available that includes Enhancements & New Features.

July 28, 2021

The Wide-and-Deep model is now available for preview. 'DNN_LINEAR_COMBINED_CLASSIFIER' and 'DNN_LINEAR_COMBINED_REGRESSOR' create Wide-and-Deep Classifier and Regressor models, respectively.

July 27, 2021

Explainable artificial intelligence (XAI) helps you understand the results that your predictive machine-learning model generates for classification and regression tasks by defining how each feature in a row of data contributed to the predicted result. This feature is now available for preview.

BigQuery now supports the INTERVAL type, which represents a duration or an amount of time. This type is in Preview.

July 26, 2021

Time series models now support holiday effects for weekly time series, in addition to the daily time series that was previously supported. This feature is now generally available (GA).

DML query jobs now return statistics about the number of rows that were inserted, deleted, or updated. For more information, see DmlStats in the Job resource type. In addition, DML statistics are now available in the INFORMATION_SCHEMA.JOBS_BY_* views. This feature is generally available (GA).

July 19, 2021

BigQuery now supports workload management data control language (DCL) statements:

• CREATE CAPACITY
• CREATE RESERVATION
• CREATE ASSIGNMENT
• DROP CAPACITY
• DROP RESERVATION
• DROP ASSIGNMENT

This feature is generally available GA.

BigQuery now supports the following SQL query operators:

• PIVOT operator
• UNPIVOT operator

This feature is generally available (GA).

The end-to-end user journey for BigQuery ML documents an overview of the complete machine-learning flow for each available model including feature preprocessing, model creation, hyperparameter tuning, inference, evaluation, model export, etc.

BigQuery standard SQL now supports the CONTAINS_SUBSTR function. This feature is generally available (GA).

July 07, 2021

BigQuery now supports materialized views without aggregation and materialized views with inner join. This feature is in Preview.

July 01, 2021

An updated version of ODBC driver for BigQuery is now available that includes bug fixes, parameterized data type support, and metadata retrieval performance improvements.

An updated version of JDBC driver for BigQuery is now available that includes bug fixes, parameterized data type support, and job retry improvements.

June 29, 2021

BigQuery is now available in the Delhi (asia-south2) region.

BigQuery ML is now available in the Delhi (asia-south2) region.

BigQuery Data Transfer Service is now available in the Delhi (asia-south2) region.

BigQuery BI Engine is now available in the Delhi (asia-south2) region.

BigQuery now supports multi-statement transactions. These allow you to perform mutating operations, such as inserting or deleting rows, on one or more tables, and either commit or roll back the changes atomically. This feature is in Preview.

June 28, 2021

BigQuery now supports access management data control language (DCL) statements and corresponding views:

• GRANT
• REVOKE
• INFORMATION_SCHEMA.OBJECT_PRIVILEGES view

GRANT and REVOKE statements are generally available (GA). OBJECT_PRIVILEGES table is available in Preview.

BigQuery now supports the following casting features:

• PARSE_BIGNUMERIC
• PARSE_NUMERIC
• Format clause for CAST available for the following data types:
  • String type
  • Date type
  • Datetime type
  • Time type
  • Timestamp type
  • Numeric types
  • Bytes type
• Numeric type INT64 aliases (INT, SMALLINT, INTEGER, BIGINT, TINYINT, BYTEINT)
• ST_GEOGFROM

These features are generally available (GA).

BigQuery now supports the ALTER COLUMN SET OPTIONS data definition language (DDL) statement. This feature is generally available (GA).

Table functions are now available in Preview. These user-defined functions, commonly known as table-valued functions (TVFs), return a table value.

The Google Trends dataset is now available in Preview and available in the Google Cloud Marketplace.

The Google Trends dataset is now in Preview and available in the Google Cloud Marketplace.

Audit logging, Cloud Logging, and Cloud Monitoring for the BigQuery Data Transfer Service are now generally available (GA).

June 25, 2021

BigQuery table snapshots are now in Preview. A table snapshot is a low-cost, read-only copy of a table's data as it was at a particular time. For more information, see Introduction to table snapshots.

June 22, 2021

BigQuery Data Transfer Service now supports Google Merchant Center data transfers for local inventories and regional inventories.

BigQuery ML is releasing the following features for preview:

• The ML.DETECT_ANOMALIES function is now available. This function provides anomaly detection for BigQuery ML. The function runs against time-series data using ARIMA_PLUS models. The function runs against independent and identically distributed (IID) random variables data using AUTOENCODER and KMEANS models.
• The AUTOENCODER model type is now available for CREATE MODEL statements. This is a TensorFlow-based, deep-learning model that supports sparse data representations, and is commonly used in ML tasks such as feature embedding, unsupervised anomaly detection, and non-linear dimensionality reduction. The ML.PREDICT function can use previously built AUTOENCODER models to reduce the dimensionality of query results.
• Hyperparameter tuning is now available and can be used to improve model performance by searching for the optimal hyperparameters when training ML models using CREATE MODEL statements. View the BigQuery ML Hypertuning tutorial to learn how to improve model performance by 40%.

June 21, 2021

Row-level security on table data is now generally available in BigQuery.

BigQuery is now available in the Melbourne (australia-southeast2) region.

BigQuery ML is now available in the Melbourne (australia-southeast2) region.

BigQuery Data Transfer Service is now available in the Melbourne (australia-southeast2) region.

BigQuery BI Engine is now available in the Melbourne (australia-southeast2) region.

June 07, 2021

BigQuery now supports parameterized types. The following parameterized types are supported:

• STRING(L)
• BYTES(L)
• NUMERIC(P) / NUMERIC(P, S)
• BIGNUMERIC(P) / BIGNUMERIC(P, S)

This feature is in Preview.

May 25, 2021

The free trial period for BigQuery BI Engine's SQL interface has been extended to July 15th, 2021. You must enroll to participate in the preview. With this feature, BI Engine now interacts with popular BI tools such as Looker, Tableau, and more, by means of an interactive SQL interface.

May 20, 2021

BigQuery GIS now supports loading geography data from newline-delimited GeoJSON files. This feature is generally available (GA). For more information, see Loading GeoJSON data.

BigQuery GIS now supports the following functions. These functions are generally available (GA).

• ST_STARTPOINT
• ST_ENDPOINT
• ST_POINTN

These functions return a point of a linestring geography as a point geography.

May 19, 2021

BigQuery now supports the ability to rename tables using SQL. See ALTER TABLE RENAME TO. This feature is generally available (GA).

May 18, 2021

The CREATE MODEL statement for training AutoML Tables models is now generally available (GA). AutoML Tables enable you to automatically build state-of-the-art machine learning models on structured data at massively increased speed and scale. For more information, see CREATE MODEL statement for training AutoML Tables models.

May 11, 2021

Updated version of ODBC driver for BigQuery includes bug fixes and install guide improvements.

Updated version of JDBC driver for BigQuery includes bug fixes, service account keyfile support, connection property enhancements, and BigQuery client library updates.

May 10, 2021

BigQuery now supports the following SQL query clauses and operators:

• PIVOT operator
• UNPIVOT operator
• QUALIFY clause

This feature is in Preview.

April 30, 2021

BigQuery now supports the following data definition language (DDL) statements:

• CREATE VIEW with column name list
• ALTER COLUMN DROP NOT NULL constraint

This feature is in GA.

April 21, 2021

BigQuery supports changing an existing non-clustered table to a clustered table and vice versa. You can also update the set of clustered columns of a clustered table. This feature was first documented in October 2020 but was not included in a release note. For more information, see Modifying clustering specification.

April 19, 2021

BigQuery ML is introducing new ARIMA_PLUS models and deprecating the ARIMA model type. While the underlying modeling technique has not changed, the following improvements are now available in ARIMA_PLUS:

• Explainable forecasting via ML.EXPLAIN_FORECAST (tutorial).

• More comprehensive evaluation via ML.ARIMA_EVALUATE (tutorial).

• Multiple ID columns are specifiable via time_series_id_col.

• Additional time series (500,000) for simultaneous forecasting.

• Two new training options: clean_spikes_and_dips and adjust_step_changes.

• Finer data frequency: per_minute.

April 12, 2021

The BigQuery Admin Resource Charts Preview is now available for Reservation users, enabling administrators to more easily monitor and troubleshoot their BigQuery environment. It provides visibility into key metrics such as slot consumption, job concurrency, and job execution time across the entire organization.

April 09, 2021

BigQuery now has better support for loading ENUM and LIST types in Parquet files.

• ENUM logical types can be converted to STRING or BYTES.
• Schema inference is supported for LIST logical types.

For more information, see Loading Parquet data from Cloud Storage.

April 07, 2021

Beginning in early Q3 2021, BigQuery Storage Read API will start charging for network egress. In addition, BigQuery Storage Read API will become available in all locations, with appropriate pricing. Another release note will be issued when these changes take effect.

April 06, 2021

The BigQuery Storage Write API is now in Preview. The Storage Write API is a stream-based API for ingesting data into BigQuery at low cost and high throughput. It provides exactly-once delivery semantics with real-time latency. For more information, see Using the BigQuery Storage Write API.

April 02, 2021

BigQuery standard SQL now supports the ALTER TABLE DROP COLUMN. This feature is in Preview.

The maximum length has been increased from 128 characters to 300 characters for the following BigQuery fields: table column names, column alias names, and user-defined function names.

March 31, 2021

BigQuery standard SQL now supports the following statements for creating, configuring, and deleting datasets:

• CREATE SCHEMA
• ALTER SCHEMA
• DROP SCHEMA

These statements are generally available (GA).

BigQuery standard SQL now supports the TABLESAMPLE operator, which lets you query random subsets of data from large BigQuery tables. For more information, see Table sampling. This feature is in Preview.

BigQuery standard SQL now supports the following JSON functions:

• JSON_EXTRACT_STRING_ARRAY
• JSON_QUERY_ARRAY
• JSON_VALUE_ARRAY

These statements are generally available (GA).

INFORMATION_SCHEMA views for table partitions are now available. This feature is in Preview.

The INFORMATION_SCHEMA.TABLES view now includes a DDL column that can be used to recreate the table. This feature is in Preview.

Support for the BigNumeric type in BigQuery standard SQL is now generally available (GA).

March 24, 2021

BigQuery is now available in the Warsaw (europe-central2) region.

BigQuery ML is now available in the Warsaw (europe-central2) region.

BigQuery Data Transfer Service is now available in the Warsaw (europe-central2) region.

BigQuery BI Engine is now available in the Warsaw (europe-central2) region.

March 11, 2021

BigQuery ML now supports training for DNN/Boosted Tree models in the Iowa (us-central1) region.

March 02, 2021

Updated version of Magnitude Simba ODBC driver includes bug fixes, performance improvements, and enhancements such as support for dynamic SQL and additional DDL and DML keywords.

Updated version of Magnitude Simba JDBC driver includes bug fixes and performance improvements.

February 25, 2021

BigQuery BI Engine now interacts with popular BI tools such as Looker, Tableau, and more, by means of an SQL interface. You must enroll to participate in the preview.

BigQuery materialized views are now generally available (GA). BigQuery materialized views are now generally available (GA). Materialized views are precomputed views that periodically cache the results of a query, enhancing performance and efficiency, and reducing costs, particularly for aggregated queries. For more information, see Introduction to materialized views.

February 24, 2021

The BigQuery Data Transfer Service's 1-hour minimum file age requirement for transfers from Cloud Storage has been eliminated.

February 23, 2021

The BigQuery Data Transfer Service's minimum interval time between recurring transfers from Cloud Storage has been reduced from one hour to 15 minutes.

February 16, 2021

BigQuery now supports exporting table data in Parquet format. This feature is in Preview. For more information, see Parquet export details.

February 12, 2021

BigQuery standard SQL queries on Google Cloud Storage data are now supported by cached query results.

February 02, 2021

Updated version of Magnitude Simba JDBC driver includes bug and security fixes and enhancements for additional DDL keywords and dynamic SQL.

January 29, 2021

Clustered tables now support the DATETIME type for clustering columns. For more information, see Creating and using clustered tables. This feature is generally available.

January 19, 2021

BigQuery is now available in the Iowa (us-central1) region.

The BigQuery Data Transfer Service is now available in the Iowa (us-central1) region.

BigQuery BI Engine is now available in the Iowa (us-central1) region.

BigQuery ML is now available in the Iowa (us-central1) region.

January 14, 2021

BigQuery's Cloud console UI has been updated with many usability improvements for analyzing data, including multi-tab navigation, a new resource panel, and a new SQL editor. These updates are in Preview. For more information, see Using the Cloud console.

December 21, 2020

BigQuery standard SQL now supports the BigNumeric data type for high-precision computations. The BigNumeric data type is in Preview.

December 16, 2020

BigQuery Data Transfer Service is now fully integrated with VPC Service Controls, and can be protected using a service perimeter. Please refer to VPC-SC supported products page for more info.

December 10, 2020

Updated version of Magnitude Simba ODBC driver includes bug fixes and enhancements such as support for BigNumeric data and improved driver logic.

Updated version of Magnitude Simba JDBC driver includes bug and security fixes and enhancements such as support for Java 11, SSL trust store, BigNumeric data, and version-agnostic fully-qualified class names.

November 23, 2020

BigQuery ML integration with AI Platform for Boosted Tree models is now generally available (GA). For more information, see the following documentation:

• CREATE MODEL statement for Boosted Tree models using XGBoost

• ML.FEATURE_IMPORTANCE function

BigQuery ML integration with AI Platform for Deep Neural Network (DNN) models is now generally available (GA). For more information, see CREATE MODEL statement for Deep Neural Network (DNN) models.

Exporting BigQuery ML models to Cloud Storage and using them for online prediction is now generally available (GA). For more information, see Exporting models and the EXPORT MODEL statement.

November 17, 2020

BigQuery column-level security is now generally available. Policy tags can be replicated across locations. For more information, see Introduction to BigQuery column-level security.

October 26, 2020

BigQuery standard SQL now supports the SUBSTRING function. This function is generally available (GA).

October 22, 2020

The ST_GEOGFROMGEOJSON and ST_GEOGFROMTEXT geographic functions support a new make_valid parameter. If set to TRUE, the function attempts to correct polygon issues when importing geography data.

The ST_GEOGFROMTEXT function also supports a new planar parameter. If set to TRUE, the function treats imported WKT geometries as having planar edges.

These new function parameters are in Beta.

Updated version of Magnitude Simba ODBC driver includes improvements to performance, logging, OpenSSL support, and bug fixes.

October 19, 2020

BigQuery Audit Logs stopped using the following checks for redacting resource names for cross-project access and caller identities: The bigquery.jobs.create permission check and the internal setting for a project domain. Please review the documentation at Caller identities and resource names.

October 14, 2020

Dynamic SQL is now generally available (GA). Dynamic SQL lets you generate and execute SQL statements dynamically at runtime. For more information, see EXECUTE IMMEDIATE.

BigQuery standard SQL now supports the following new functions. These functions are generally available (GA).

• ASCII
• CHR
• INITCAP
• INSTR
• LAST_DAY
• LEFT
• OCTET_LENGTH
• REGEXP_EXTRACT with 2 additional parameters (position and occurrence)
• REGEXP_INSTR
• REGEXP_SUBSTR
• RIGHT
• SOUNDEX
• TRANSLATE
• UNICODE

BigQuery now supports the following new statements. These statements are generally available (GA).

• CREATE EXTERNAL TABLE
• DROP EXTERNAL TABLE
• ALTER TABLE ADD COLUMN
• EXPORT DATA
• TRUNCATE TABLE

BigQuery standard SQL now supports DATE arithmetics operators.

The following INFORMATION_SCHEMA views are now generally available (GA).

• TABLES
• TABLE_OPTIONS
• COLUMNS
• COLUMN_FIELD_PATHS
• VIEWS
• ROUTINES
• ROUTINE_OPTIONS
• PARAMETERS
• SCHEMATA
• SCHEMATA_OPTIONS

BigQuery now supports Unicode table names. For more information, see Table naming.

Queries can now have duplicate column names.

October 06, 2020

Authorized user-defined functions (UDFs) are now generally available (GA). Authorized UDFs let you share query results without giving access to the underlying tables. For more information, see Authorized UDFs.

The Cloud console now lets you opt in to search and autocomplete powered by Data Catalog. This feature is in beta.

October 01, 2020

BigQuery table-level access control is now generally available. For more information, see Introduction to table access controls .

September 29, 2020

Time series models now let you change DATA_FREQUENCY from the default value (AUTO_FREQUENCY) when forecasting multiple time series using TIME_SERIES_ID_COL.

September 21, 2020

The following time-unit partitioning features are now Generally Available (GA):

• Creating partitions using hourly, monthly, and yearly time-unit granularities.
• Using a DATETIME column as the partitioning column.

September 18, 2020

Beginning in late 2020, all BigQuery predefined roles with permission to create jobs, including queries, will also be able to get information about their own jobs and cancel their own jobs. This change does not include permission to see jobs created by other users.

September 17, 2020

The BigQuery Data Transfer Service is now available in the following regions: Los Angeles (us-west2), São Paulo (southamerica-east1), South Carolina (us-east1), Hong Kong (asia-east1) and Osaka (asia-northeast2).

The BigQuery Data Transfer Service is now available in the following regions: Los Angeles (us-west2), São Paulo (southamerica-east1), South Carolina (us-east1), Hong Kong (asia-east1) and Osaka (asia-northeast2).

September 11, 2020

You can now use the BQ.JOBS.CANCEL system procedure to cancel a job. For more information, see Canceling jobs.

August 31, 2020

Updated version of Magnitude Simba JDBC driver includes bug fixes and enhancements such as automatically turning on the BigQuery Storage API for anonymous table reads (no charge for temporary table reads).

August 27, 2020

Time series model support is now Generally Available (GA). This release includes a new training option: AUTO_ARIMA_MAX_ORDER.

For more information about time series model support, see the following documentation:

• CREATE MODEL statement for time series models

• ML.FORECAST function

• ML.ARIMA_COEFFICIENTS function

• Single time-series forecasting from Google Analytics data

• Multiple time-series forecasting with a single query

August 26, 2020

INFORMATION_SCHEMA views for BigQuery reservations are now Generally Available (GA).

August 25, 2020

INFORMATION_SCHEMA views for jobs metadata by timeslice are now available.

August 24, 2020

BigQuery support for using service account credentials with scheduled queries is Generally Available (GA).

August 19, 2020

When using consecutive ON/USING clauses, parentheses are now optional and can be omitted. For example, you can use either of the following statements:

• FROM A JOIN (B JOIN C ON B.x = C.y) USING (z)
• FROM A JOIN B JOIN C ON B.x = C.y USING (z)

August 18, 2020

You can now explicitly assign projects to use on-demand pricing with Reservations. See Assign a project to None.

August 17, 2020

Matrix Factorization model support is now Generally Available (GA). For more information, see the following documentation:

• CREATE MODEL statement for Matrix Factorization

• ML.RECOMMEND function

• Using BigQuery ML to predict movie recommendations

• Using BigQuery ML to predict website content for visitors

August 13, 2020

The exports per day (Extract Bytes) default quota has been raised from 10 TB to 50 TB per day.

August 11, 2020

For flat-rate pricing, the minimum slot purchase is now 100 slots. Slots can be purchased in 100-slot increments.

August 06, 2020

BigQuery is now available in the following regions: Oregon (us-west1), Belgium (europe-west1), and Netherlands (europe-west4).

BigQuery ML is now available following regions: Oregon (us-west1), Belgium (europe-west1), and Netherlands (europe-west4).

BigQuery Data Transfer Service is now available following regions: Oregon (us-west1), Belgium (europe-west1), and Netherlands (europe-west4).

BigQuery BI Engine is now available following regions: Oregon (us-west1), Belgium (europe-west1), and Netherlands (europe-west4).

July 31, 2020

Updated version of Magnitude Simba ODBC driver includes performance improvements and bug fixes.

July 27, 2020

INFORMATION_SCHEMA views for streaming metadata are now in alpha. You can use these views to retrieve historical and real-time information about streaming data into BigQuery.

July 24, 2020

BigQuery Data Transfer Service is now available in the following regions: Montréal (northamerica-northeast1), Frankfurt (europe-west3), Mumbai (asia-south1), and Seoul (asia-northeast3).

BigQuery Data Transfer Service is now available in the following regions: Montréal (northamerica-northeast1), Frankfurt (europe-west3), Mumbai (asia-south1), and Seoul (asia-northeast3).

July 16, 2020

BigQuery GIS now supports two new functions, ST_CONVEXHULL and ST_DUMP:

• ST_CONVEXHULL returns the smallest convex GEOGRAPHY that covers the input.
• ST_DUMP returns an ARRAY of simple GEOGRAPHYs where each element is a component of the input GEOGRAPHY.

For more information, see the ST_CONVEXHULL and ST_DUMP reference pages.

July 15, 2020

Data split and validation options are now available for AutoML Table model training.

July 13, 2020

The Standard SQL statement ASSERT is now supported. You can use ASSERT to validate that data matches specified expectations.

July 06, 2020

Updated version of Magnitude Simba ODBC driver. This version includes some performance improvements and bug fixes, and it catches up with the JDBC driver by adding support for user defined functions and variable time zones using the connection string.

July 01, 2020

BigQuery ML now supports time series models as a beta release. For more information, see CREATE MODEL statement for time series models.

June 29, 2020

Flex slots are now generally available (GA).

The BigQuery SLA has been updated to >= 99.99% Monthly Uptime Percentage for all users.

June 26, 2020

Region qualified INFORMATION_SCHEMA views are now in beta.

Starting in mid-August, unqualified INFORMATION_SCHEMA queries for SCHEMATA and SCHEMATA_OPTIONS views will default to returning metadata from the US multi-region. For information about how to specify a region, see region qualifier syntax.

June 16, 2020

BigQuery ML now supports preview integration with AI Platform. The following models are supported in preview:

• AutoML Tables models. For more information, see CREATE MODEL statement for AutoML Tables models.

• Boosted Tree models using XGBoost. For more information, see CREATE MODEL statement for Boosted Tree models.

• Deep Neural Network (DNN) models. For more information, see CREATE MODEL statement for DNN models.

INFORMATION_SCHEMA views for jobs are now generally available (GA).

The Top Brands report for Google Merchant Center Best Sellers exports is now in beta.

June 09, 2020

Clustering for non-partitioned tables is now supported. For more information about clustered tables, see Introduction to clustered tables.

June 08, 2020

BigQuery is now available in the Jakarta (asia-southeast2) region.

BigQuery BI Engine is now available in the Jakarta (asia-southeast2) region.

BigQuery ML is now available in the Jakarta (asia-southeast2) region.

BigQuery Data Transfer Service is now available in the Jakarta (asia-southeast2) region.

June 04, 2020

BigQuery Table ACL is now available as a beta release. For more information, see Introduction to table access controls .

June 02, 2020

You can now purchase BigQuery slots using the bq command-line tool. BigQuery Reservations allows you to purchase slots to take advantage of BigQuery flat-rate pricing and allocate slots for workload management.

A new GIS function, ST_Simplify, is available. ST_Simplify returns a simplified version of the input GEOGRAPHY by replacing sections with straight lines.

Standard SQL view definition bodies can now contain references without project qualifiers, as long as the view is created by the tables.insert REST API or is in the same project used to run the CREATE VIEW DDL query.

Standard SQL user-defined function definition bodies can now contain references to tables and views.

May 21, 2020

The BigQuery Storage API now supports reading small anonymous (cached) tables without any limitations.

May 20, 2020

Happy 10th birthday, BigQuery!

Cloud SQL federated queries are now generally available (GA).

Hourly partitioned tables are now in beta.

Dynamic SQL is now available as a beta release in all BigQuery regions. Dynamic SQL lets you generate and execute SQL statements dynamically at runtime. For more information, see EXECUTE IMMEDIATE.

BigQuery Trial slots are now available in US and EU multi-regions. Trial slots are a limited promotion for qualified customers.

May 13, 2020

Updated versions of Magnitude Simba JDBC drivers have been released.

May 12, 2020

Updated versions of Magnitude Simba ODBC drivers have been released.

May 08, 2020

Next generation BigQuery streaming is now Generally Available (GA).

May 01, 2020

A new function, JSON_EXTRACT_ARRAY, has been added to the list of JSON functions. This function allows you to extract the contents of a JSON document as a string array. For more information, see the JSON_EXTRACT_ARRAY reference section.

The ORDER BY clause now supports the NULLS FIRST and NULLS LAST clauses. These clauses allow you to specify the sort order of null and non-null values. For more information, see the ORDER BY reference section.

April 30, 2020

BigQuery Data Transfer Service is now available in the Taiwan (asia-east1) region.

The BigQuery Data Transfer Service is now available in the Taiwan (asia-east1) region.

April 27, 2020

BigQuery is now available in the Las Vegas (us-west4) region.

BigQuery ML is now available in the Las Vegas (us-west4) region.

BigQuery BI Engine is now available in the Las Vegas (us-west4) region.

BigQuery Data Transfer Service is now available in the Las Vegas (us-west4) region.

April 22, 2020

BigQuery ML now supports exporting BigQuery ML models to Cloud Storage and using them for online prediction. This feature is in beta. For more information, see Exporting models.

April 17, 2020

BigQuery ML now supports Matrix Factorization models for recommendations, as a beta release. For more information, see The CREATE MODEL statement for Matrix Factorization.

April 16, 2020

BigQuery Reservations is now Generally Available (GA). BigQuery Reservations allows you to purchase BigQuery slots to take advantage of BigQuery flat-rate pricing and allocate slots for workload management.

Around the end of April 2020, INFORMATION_SCHEMA (Beta) views for dataset metadata will return metadata about all datasets in a region. Currently, these views return metadata about all datasets in the project across all regions. This upcoming change will also provide support for querying a specific region's metadata (for example, region-us.INFORMATION_SCHEMA.SCHEMATA instead of INFORMATION_SCHEMA.SCHEMATA).

You can replicate this future behavior now by filtering on the SCHEMATA view's LOCATION column (for example, LOCATION = 'US').

April 14, 2020

BigQuery Data Transfer Service now supports Google Merchant Center data transfers for best sellers data.

April 09, 2020

Scheduling queries no longer requires the bigquery.transfers.update permission. The bigquery.jobs.create permission can now be used to schedule queries. See Scheduling queries for details.

April 08, 2020

BigQuery materialized views are now available as a beta release. For more information, see Introduction to materialized views .

April 02, 2020

BigQuery Reservations is now available in all BigQuery regions.

March 31, 2020

INFORMATION_SCHEMA views for BigQuery reservations are now in public alpha.

March 30, 2020

Scripting and stored procedures are now Generally Available.

March 27, 2020

BigQuery Column-level security is now available as a beta release. For more information, see Introduction to BigQuery Column-level security.

March 24, 2020

BigQuery Data Transfer Service is now available in the Northern Virginia (us-east4) region and the Salt Lake City (us-west3) region.

March 23, 2020

BigQuery Data Transfer Service now supports Google Merchant Center data transfers for pricing competitiveness.

March 13, 2020

BigQuery Data Transfer Service now supports the Finland region.

March 11, 2020

BigQuery Data Transfer Service now supports the Zürich region.

March 02, 2020

The BigQuery Storage API is now generally available. For more information, see the Storage API overview.

Support for querying Cloud Storage data in ORC and Parquet format is now Generally Available.

Support for loading and querying externally partitioned data that is stored on Cloud Storage using a default hive partitioning layout is now Generally Available. The data must be in Avro, Parquet, ORC, CSV, or JSON format.

Personally identifiable information (PII) in audit log records for successful reads of resources with restricted access is no longer redacted. For more information, see Caller identities and resource names.

February 27, 2020

Cloud Storage transfers to BigQuery now have the option of appending data or refreshing (overwriting) data.

February 26, 2020

BigQuery Flex slots are now available in beta in all BigQuery Reservations regions. See Workload management and Commitment plans for more information.

BigQuery DML statements no longer have quota limits.

Note: DML statements are counted toward the maximum number of table operations per day and partition modifications per day. However, DML statements will not fail due to these limits. (Updated April 6, 2021)

February 25, 2020

Support for data transfers from Amazon S3 to BigQuery is now generally available.

February 24, 2020

BigQuery is now available in the Salt Lake City (us-west3) region.

BigQuery ML is now available in the Salt Lake City (us-west3) region.

BI Engine is now available in the Salt Lake City (us-west3) region.

February 20, 2020

BigQuery BI Engine is now GA.

Added support for custom queries and views from Data Studio.

Increased reservation size to 100 GB.

Support for Amazon Redshift data warehouse migrations to BigQuery is now generally available.

Support for migrating Amazon Redshift private instances to BigQuery with VPC is also now generally available.

February 18, 2020

Support for tables partitioned on an integer column is now GA. For more information, see Creating and using integer range partitioned tables.

February 14, 2020

INFORMATION_SCHEMA views for live and historical jobs metadata are now in beta.

February 13, 2020

The BigQuery sandbox is now GA.

February 11, 2020

Updated versions of Magnitude Simba ODBC drivers are now available.

January 24, 2020

BigQuery is now available in the Seoul (asia-northeast3) region.

BigQuery ML is now available in the Seoul (asia-northeast3) region.

January 22, 2020

The BigQuery Storage API is now available in all BigQuery regional and multi-regional locations. For more information, see Dataset locations.

January 16, 2020

Updated versions of Magnitude Simba JDBC and ODBC drivers have been released.

January 14, 2020

There is no longer a charge to use Google's ads management connectors and YouTube connectors for the BigQuery Data Transfer Service. Normal BigQuery analysis and storage pricing will continue to apply after any transfers.

BigQuery Reservations is now available in the London (europe-west2) region.

December 20, 2019

The ALTER TABLE and ALTER VIEW Data Definition Language statements are now Generally Available (GA). These statements let you set options on tables and views.

BigQuery Reservations is now available in the Tokyo (asia-northeast1) region.

December 19, 2019

Third party transfers for loading data from external data sources are now generally available GA.

Support for transferring Google Play reports is now generally available GA.

BigQuery ML data preprocessing is now Generally Available (GA). Read about the preprocessing functions and walk through how to use the TRANSFORM clause for feature engineering.

December 13, 2019

BigQuery Reservations is now in beta in US and EU regions. BigQuery Reservations allows you to purchase BigQuery slots to take advantage of BigQuery flat-rate pricing and allocate slots for workload management.

December 11, 2019

Support for tables partitioned on an integer column is now in Beta. For more information, see Creating and using integer range partitioned tables.

December 10, 2019

For users billed on the on-demand pricing model, BigQuery can now burst beyond the default slot capacity to accelerate queries. There is no extra charge associated with query acceleration.

December 09, 2019

Support for Cloud Storage Transfers is now generally available GA.

December 04, 2019

You can now use KMEANS++ to initialize the clusters of a k-means model. KMEANS++ trains a better model than random cluster initialization.

November 26, 2019

INFORMATION_SCHEMA now has updated permissions checks. INFORMATION_SCHEMA is currently in beta.

Previously, queries against INFORMATION_SCHEMA views filtered the results based on the IAM permissions needed to get metadata about the resource. Now, queries return accessDenied (403) errors if users have insufficient IAM permissions to get metadata.

November 22, 2019

Support for persistent standard SQL UDFs is now Generally Available (GA).

You can now use INFORMATION_SCHEMA views to get routine metadata in BigQuery. INFORMATION_SCHEMA is currently in beta.

November 21, 2019

BigQuery ML data preprocessing is now in beta.

BigQuery ML now supports customer-managed encryption keys (CMEK). You can use your own Cloud KMS keys to encrypt ML models.

BigQuery now supports using service account credentials with scheduled queries. This feature is in Beta.

November 20, 2019

BigQuery is now available in the South Carolina (us-east1) region.

BigQuery ML is now available in the South Carolina (us-east1) region.

November 12, 2019

Support for omitting column names in INSERT and MERGE statements is now Generally Available (GA).

October 31, 2019

Support for querying Cloud Storage data in ORC and Parquet format is now in beta.

Support for loading and querying externally partitioned data that is stored on Cloud Storage using a default hive partitioning layout is now in beta. The data must be in Avro, Parquet, ORC, CSV, or JSON format.

October 23, 2019

The BigQuery service is being renamed from:

bigquery-json.googleapis.com

to

bigquery.googleapis.com.

bigquery-json.googleapis.com will become an alias of bigquery.googleapis.com going forward.

This change is largely transparent to users and does not affect runtime traffic or management operations. Risks and mitigation are discussed below.

Calls to the BigQuery API will not be affected.

Timing

The change is a two phase process that we plan to release during November and December 2019.

Impact when using Service Management and Service Usage APIs

Responses from the Service Management API and the Service Usage API for information about the BigQuery service will contain information about bigquery.googleapis.com, even when you request data for bigquery-json.googleapis.com. Other than the name, the responses from the Service Management API and the Service Usage API will be the same. For example, listing enabled services will now return bigquery.googleapis.com, instead of bigquery-json.googleapis.com.

Risks and Mitigation

• During the week of the second phase, alerts on Stackdriver Transparent Service Level Indicators (SLIs) for the bigquery-json.googleapis.com will break. Users should add a duplicate alert for bigquery.googleapis.com. The following are the serviceruntime SLIs where you now need to use bigquery.googleapis.com:

  • serviceruntime.googleapis.com/api/request_count
  • serviceruntime.googleapis.com/api/request_latencies
  • serviceruntime.googleapis.com/api/request_latencies_backend
  • serviceruntime.googleapis.com/api/request_latencies_overhead
  • serviceruntime.googleapis.com/api/request_sizes
  • serviceruntime.googleapis.com/api/response_sizes

• The Google provider for Terraform has been updated to use the new name, bigquery.googleapis.com. Terraform users should upgrade to a Google provider with version >= 2.17.0 before November 4, 2019.

October 09, 2019

The RANGE_BUCKET mathematical function has been added. This function scans through a sorted array and returns the 0-based position of a given point's upper bound. This function can be useful if you need to group your data to build partitions, histograms, business-defined rules, and more.

October 04, 2019

Updated versions of Magnitude Simba JDBC and ODBC drivers have been released.

October 03, 2019

BigQuery Data Transfer Service now supports Search Ads 360 data transfers. Search Ad 360 transfers are in beta.

The ability to use scripting, stored procedures, and temporary tables is now in Beta.

September 30, 2019

Importing TensorFlow models is now GA.

September 27, 2019

• The maximum size of a BI Engine reservation is now 50 GB per project per location. For more information, see Capacity limits.
• We added more options for BI Engine capacity that is bundled with flat-rate pricing. For more information, see Flat-rate pricing.

September 26, 2019

BigQuery Data Transfer Service support for data & schema migration from a Teradata data warehouse to BigQuery has been upgraded. On-demand, one-time transfers are now generally available (GA). Incremental, recurring transfers are in Beta.

September 23, 2019

Support for k-means clustering models is now GA. For more information, see Creating a k-means clustering model.

September 17, 2019

BigQuery is now available in the Frankfurt (europe-west3) region.

BigQuery ML is now available in the Frankfurt (europe-west3) region.

BigQuery BI Engine is now available in the Frankfurt (europe-west3) region.

September 11, 2019

Billing is now enabled for BigQuery BI Engine. You are charged according to the information on the Pricing page.

August 30, 2019

Support for copying datasets, including across regions, is now in Beta.

August 27, 2019

When you create an external table that is linked to a Google Sheets file stored in Google Drive, you can now specify an individual sheet or a cell range. You are no longer limited to the first sheet only.

August 22, 2019

Updated version of Magnitude Simba ODBC driver.

You no longer have to manually re-cluster data in clustered tables. BigQuery now performs automatic re-clustering on all clustered tables.

August 19, 2019

If you stream data into BigQuery without populating the insertId field, you get the following higher quotas in the US multi-region location. These higher quotas are currently in beta.

• Maximum rows per second: 1,000,000
• Maximum bytes per second: 1 GB

For more information, see the Quotas and limits page.

August 07, 2019

You can now query data in BigQuery and Cloud SQL with a federated query. Cloud SQL federated queries are now in beta.

August 05, 2019

The BigQuery Storage API now supports reading data from BigQuery Storage in ARROW format. The BigQuery Storage API is currently in beta.

July 25, 2019

Updated version of Magnitude Simba ODBC driver. This version adds support for the BigQuery Storage API. The structure of the Linux ODBC tar package has also been changed to contain both 32-bit and 64-bit driver versions.

July 19, 2019

The scheduled query feature is now Generally Available.

July 02, 2019

Importing TensorFlow models is now Beta.

July 01, 2019

Support for persistent user-defined functions (UDFs) is now in Beta.

BigQuery Data Transfer Service now supports Google Merchant Center data transfers. Merchant Center transfers are in the beta stage.

June 17, 2019

Support for Amazon Redshift data warehouse migrations to BigQuery is now in Beta.

June 13, 2019

Updated version of Magnitude Simba JDBC driver.

June 10, 2019

BigQuery is moving to the Cloud console. The classic web UI remains available through June 2020. For information on migrating to the Cloud console, see Moving to the Cloud console.

June 03, 2019

Support for omitting column names in INSERT and MERGE statements is now in Beta.

May 29, 2019

BigQuery ML is now Generally Available (GA).

May 28, 2019

The maximum response size for query results has increased from 128 MB to 10 GB.

May 15, 2019

Support for clustering a table by using a GEOGRAPHY column is now Generally Available (GA).

May 14, 2019

BigQuery ML now supports the DROP MODEL DDL statement for deleting models.

May 13, 2019

The concurrent rate limit for on-demand, interactive queries has increased from 50 to 100 concurrent queries.

May 06, 2019

BigQuery ML IAM permissions are now available. These permissions take effect on June 6, 2019. Customers with custom roles should migrate to these permissions no later than June 6. Pre-defined IAM roles and basic roles are not impacted by this change.

April 29, 2019

BigQuery BI Engine is now available in several additional regions:

• Los Angeles
• Montréal
• Northern Virginia
• São Paulo
• Zürich
• Hong Kong
• Mumbai
• Taiwan
• Singapore
• Sydney

For a complete list of BigQuery BI Engine locations, see Supported regions

April 22, 2019

The following quotas have changed:

• The number of copy jobs per project per day has increased from 10,000 to 100,000.
• The number of load jobs per project per day has increased from 50,000 to 100,000.
• The number of export jobs per project per day has increased from 50,000 to 100,000.

April 18, 2019

BigQuery is now available in the Osaka (asia-northeast2) region. For upcoming regional launches, see the Google Cloud blog.

April 10, 2019

BigQuery ML now supports the k-means model type for clustering and customer segmentation.

BigQuery BI Engine is now in beta.

Support for clustered tables is now generally available (GA).

BigQuery standard SQL now supports AEAD encryption functions.

The BigQuery web UI in the Cloud console is now generally available (GA).

The BigQuery Data Transfer Service now supports transferring data into BigQuery from Amazon S3. Amazon S3 transfers are now in beta.

Third party transfers for loading data from external data sources are now in beta.

April 05, 2019

BigQuery now allows you to purchase as few as 500 slots when you enroll in a flat-rate pricing plan. You are also now able to choose either a monthly or a yearly flat-rate pricing plan. The new flat-rate pricing plans are currently in alpha.

The BigQuery Data Transfer Service is now available in beta in the Cloud console.

April 04, 2019

During the beta period, Table permissions were automatically applied to models for custom IAM roles. BigQuery ML will begin enforcing several new IAM permissions on June 6, 2019. Customers who used custom IAM roles during the beta period must reconfigure these roles to use the new BigQuery ML permissions. This change will enable you to manage Models permissions separately from BigQuery ML Table permissions. You can begin redefining your custom roles by the end of April, 2019 when the permissions are released. Pre-defined IAM roles and basic roles are not impacted by this change.

The BigQuery Data Transfer Service for Google Ads has been updated to support v201809 of Google Ads reports.

BigQuery Geographic Information Systems is now generally available (GA).

March 27, 2019

Combined with a special migration agent, the BigQuery Data Transfer Service now provides a database migration service from an on-premises data warehouse system to BigQuery. Support for Teradata migrations is now in beta.

March 18, 2019

BigQuery ML now supports ML.ROC_CURVE and ML.CONFUSION_MATRIX without input data.

The limit on the number of CREATE MODEL queries has increased from 100 to 1,000.

March 12, 2019

Updated version of Simba ODBC and JDBC driver. The Simba JDBC driver now includes support for the BigQuery Storage API.

March 11, 2019

BigQuery is now available in the Zürich (europe-west6) region. For upcoming regional launches, see the Google Cloud blog.

March 01, 2019

The BigQuery web UI in the Cloud console now supports assigning access controls to BigQuery datasets.

When you create a scheduled query, the minimum time interval for custom schedules has changed from three hours to 15 minutes.

February 25, 2019

Support for DML MERGE statements is now generally available (GA).

You can now save query results to Google Drive by using the BigQuery web UI in the Cloud console. Saving query results to Google Drive is currently in beta.

February 22, 2019

The BigQuery Storage API is now in beta.

February 15, 2019

BigQuery is now available in the São Paulo (southamerica-east1) region. For upcoming regional launches, see the Google Cloud blog.

February 13, 2019

The following load job quota has changed:

• The 16 MB maximum block size limit for Avro files has been removed.

February 11, 2019

The following DML quotas have changed:

• The maximum number of combined UPDATE, DELETE, and MERGE statements per day per table has increased from 200 to 1,000.
• The limit of 10,000 combined UPDATE, DELETE, and MERGE statements per day per project has been removed. There is no longer a project-level limit on DML statements.

February 04, 2019

You can now use INFORMATION_SCHEMA views to retrieve resource metadata in BigQuery. INFORMATION_SCHEMA is currently in beta.

The Data Definition Language now supports ALTER TABLE statements to set options on tables and views. ALTER TABLE statements are currently in beta.

January 31, 2019

You can now try BigQuery without providing a credit card or attaching a billing account to your project by using the BigQuery sandbox. The sandbox is now in beta and is available to all Google Cloud customers.

The daily export quota has been changed to measure the actual export file size in bytes, when counted toward your daily export quota. Usage is no longer measured by rounding up to the nearest GB.

The AudienceExplorer table has been deprecated by Google Ad Manager, and has been removed from Google Ad Manager report transformations.

January 29, 2019

BigQuery ML now supports automatic, batch gradient descent, and normal equation optimization strategies for linear regression models.

January 28, 2019

BigQuery is now available in the Los Angeles (us-west2) and Hong Kong (asia-east2) regions. For upcoming regional launches, see the Google Cloud blog.

January 22, 2019

BigQuery audit logging now writes enhanced messages using the BigQueryAuditMetadata format.

December 18, 2018

BigQuery is now available in the Mumbai (asia-south1), and Montréal (northamerica-northeast1) regions. For upcoming regional launches, see the Google Cloud blog.

December 13, 2018

The BigQuery ML ML.WEIGHTS function now supports standardization.

The BigQuery ML ML.PREDICT function now supports thresholds for binary logistic regression models.

December 12, 2018

Raised the maximum length of standard SQL queries to 1 MB.

December 07, 2018

BigQuery is now available in the Taiwan (asia-east1), Finland (europe-north1), and Northern Virginia (us-east4) regions. For upcoming regional launches, see the Google Cloud blog.

December 05, 2018

BigQuery now supports converting Avro logical types when you load Avro data.

You can now set or update a default partition expiration time on a dataset.

December 04, 2018

Updated versions of the Simba ODBC and JDBC drivers.

November 26, 2018

Updated version of the Simba ODBC drivers.

November 13, 2018

BigQuery is now available in the Singapore (asia-southeast1) region. For upcoming regional launches, see the Google Cloud blog.

November 08, 2018

BigQuery ML pricing is now available.

October 29, 2018

BigQuery now supports the bigquery.metadataViewer predefined IAM role.

October 22, 2018

BigQuery is now available in the Sydney (australia-southeast1) region. For upcoming regional launches, see the Google Cloud blog.

October 19, 2018

The BigQuery ML CREATE MODEL statement has increased support for unique values in labels from 10 to 50. Multiclass logistic regression models now support up to 50 unique values for labels.

October 17, 2018

You can now try BigQuery without providing a credit card or attaching a billing account to your project by using the BigQuery sandbox for Firebase. The sandbox is now in beta.

October 11, 2018

When you create a model using the random data split method, the split is now deterministic. Subsequent training runs will produce the same split so long as the underlying input data hasn't changed.

Providing input data to the ML.EVALUATE function is now optional.

October 10, 2018

BigQuery is now available in the London (europe-west2) region. For upcoming regional launches, see the Google Cloud blog.

Support for tables partitioned on a DATE or TIMESTAMP column is now generally available (GA).

Support for using DML statements against partitioned tables is also generally available (GA).

October 09, 2018

The location is no longer required when executing jobs that reference resources outside the US and EU regions.

October 03, 2018

Support for Cloud Storage Transfers is now in Beta.

September 25, 2018

BigQuery standard SQL now supports the GENERATE_UUID function.

September 20, 2018

Updated versions of Simba ODBC and JDBC drivers to support regional resources.

September 19, 2018

BigQuery ML is now available in the Tokyo (asia-northeast1) region.

September 13, 2018

BigQuery ML now supports multiclass logistic regression models and the ML.CONFUSION_MATRIX evaluation function.

September 11, 2018

Scheduling queries is now in Beta.

September 10, 2018

Loading data in ORC format is now GA (Generally Available).

BigQuery Geographic Information Systems is now Beta.

August 27, 2018

Updated version of the Simba JDBC driver.

August 20, 2018

The NUMERIC data type is now generally available.

BigQuery now supports loading Cloud Firestore export data and querying Cloud Firestore data stored in Cloud Storage.

July 31, 2018

The BigQuery Data Transfer Service no longer requires an enrollment step for each data source when you enable the service.

July 30, 2018

BigQuery support for clustered tables is now Beta.

July 25, 2018

BigQuery ML is now Beta.

BigQuery has achieved approximately 5x improvement in TPC-DS 10T power run geomean over the past 18 months.

Data Definition Language (DDL) support in BigQuery is now Generally Available (GA).

July 19, 2018

The BigQuery web UI in the Cloud console is now Beta.

BigQuery support for clustered tables is now Alpha.

July 18, 2018

Support for Google Play reports is now in Beta.

July 11, 2018

BigQuery Data Transfer Service for Google Ads is now GA (generally available).

Google Ads transfers now support API version v201806.

June 28, 2018

BigQuery standard SQL now supports the functions FROM_BASE32 and TO_BASE32.

BigQuery standard SQL now supports templated SQL UDFs.

BigQuery standard SQL now supports the function GENERATE_TIMESTAMP_ARRAY.

June 27, 2018

BigQuery Data Transfer Service support for YouTube Channel reports and YouTube Content Owner reports is now GA (generally available).

June 18, 2018

Loading data in Parquet format is now GA (generally available).

Loading data in ORC format is now Beta.

June 06, 2018

Updated version of the Simba ODBC drivers.

May 25, 2018

The following quotas have changed for load jobs:

• The row and cell size limit for CSV files has changed from 10 MB to 100 MB.
• The row size limit for JSON files has changed from 10 MB to 100 MB.

May 15, 2018

Support for the NUMERIC data type is now in Beta.

May 04, 2018

The following quotas have changed:

• The Maximum number of partitions per partitioned table has changed from 2,500 to 4,000.
• The Maximum number of combined UPDATE, DELETE, and MERGE DML statements per day per table has changed from 96 to 200.

May 01, 2018

Updated version of the Simba JDBC driver.

When you create a Google Ads transfer, the maximum value for the refresh window is now 7 days. Existing transfers are unaffected by this change.

April 17, 2018

BigQuery is now available in the Tokyo (asia-northeast1) region.

BigQuery now includes bigquery.jobs.listAllpermissions. bigquery.jobs.list still allows users to list all jobs in a project, but details and metadata are redacted for jobs submitted by other users.

April 13, 2018

By default, the number of export jobs per project per day has increased from 1,000 to 50,000.

April 10, 2018

Support for DML MERGE statements is now in Beta.

April 06, 2018

Google Ad Manager transfers now support API version v201802.

March 30, 2018

BigQuery standard SQL now supports: the ERROR function to return errors and the SAFE prefix to suppress errors.

BigQuery now allows you to monitor slot usage by job type in Stackdriver.

March 05, 2018

BigQuery data definition language now supports creating a table from the result of a query.

BigQuery support for using DML statements with partitioned tables is now in Beta.

March 01, 2018

Support for loading Parquet files is now in Beta.

Updated version of Simba JDBC driver.

February 16, 2018

Google Ads transfers now support manager accounts (MCCs).

Google Ads transfers now support API version 201710.

February 15, 2018

BigQuery Data Transfer Service run notifications are now in Alpha.

February 08, 2018

BigQuery support for tables partitioned using a TIMESTAMP or DATE column is now in Beta.

Several API methods will begin including a location field in their responses: jobs.insert, jobs.get, jobs.cancel, jobs.query, jobs.getQueryResults, datasets.list, and jobs.list. The location field is currently not required in requests.

For example, the id property will include a location in the following format: "id": "[PROJECT_ID]:[LOCATION].[JOB_ID]". The "id" property should continue to be treated as opaque. Also, the "jobReference" property will begin including a location field in the following format:

"jobReference": {
  "jobId": "[JOB_ID]",
  "location": "[LOCATION]",
  ...

Added documentation on manually changing table schemas.

January 17, 2018

BigQuery data definition language support is now in Beta.

January 08, 2018

Google Ads transfers now support the following reports:

• Age Range Performance Report
• Criteria Performance Report
• Gender Performance Report
• Bid Goal Performance Report
• Budget Performance Report
• Parental Status Performance Report

For more information, see Google Ads Report Transformation.

January 05, 2018

ISOWEEK and ISOYEAR are now supported by the following functions: DATE_TRUNC, DATETIME_TRUNC, TIMESTAMP_TRUNC, DATE_DIFF, and DATETIME_DIFF.

December 11, 2017

You can now exclude removed or deleted items when setting up a Google Ads transfer.

You can now check the Configure Jobs option to allow the BigQuery Data Transfer Service to manage YouTube Channel and Content Owner reporting jobs for you.

Backfill requests for Google Ads do not overwrite Match Tables.

December 08, 2017

Updated versions of Simba ODBC and JDBC releases.

December 04, 2017

The BigQuery web UI supports additional dimensions for filtering your query history.

November 30, 2017

INTERSECT DISTINCT and EXCEPT DISTINCT are now supported set operators in standard SQL.

FARM_FINGERPRINT is now a supported function in legacy SQL.

WEEK(<WEEKDAY>) is now a valid date part for the EXTRACT function.

There is now a quota of 300 concurrent API requests per user.

IAM custom roles are now available for BigQuery.

The _PARTITIONTIME pseudo column is now supported when querying data using a wildcard table over partitioned tables.

November 16, 2017

BigQuery Data Transfer Service is now generally available.

November 14, 2017

High compute queries are being phased out so you are no longer required to opt-in. All high compute queries under tier 100 are billed as tier 1. Running a high compute query does not require setting a threshold flag.

All queries above tier 100 will fail with a RESOURCES_EXCEEDED_PER_BYTE error unless the query is running in a reserved instance. Existing users who have run a high compute query higher than tier 100 in the last 30 days retain the ability to run queries at the higher billing tier until October 31, 2018.

November 02, 2017

Updated version of the BigQuery client library for Python v0.28.

The IGNORE NULLS and RESPECT NULLS modifiers are now supported in the FIRST_VALUE, LAST_VALUE, and NTH_VALUE analytic functions.

The FOR SYSTEM_TIME AS OF expression is now available in the FROM clause.

October 16, 2017

Updated versions of Simba ODBC and JDBC releases.

September 29, 2017

GROUP BY ROLLUP is now supported in standard SQL.

September 28, 2017

The following string functions are now supported in standard SQL:

• NORMALIZE
• NORMALIZE_AND_CASEFOLD

Added support for creating ingestion-time partitioned tables by specifying partitioning options when you write query results or load data into a new destination table.

September 01, 2017

The following analytic functions are now supported in standard SQL:

• PERCENTILE_CONT
• PERCENTILE_DISC

August 18, 2017

Added best practices documentation for BigQuery:

• BigQuery Best Practices

August 17, 2017

Updated versions of Simba ODBC and JDBC releases.

August 11, 2017

The following standard SQL statistical aggregate functions now support the OVER clause — COVAR_SAMP, COVAR_POP, and CORR:

• Statistical-aggregate-functions

August 02, 2017

Announced support for WEEK, ISOWEEK, and ISOYEAR with the standard SQL EXTRACT function:

• EXTRACT

July 27, 2017

Announced availability of snappy codec for Avro compressed data blocks:

• Avro format

July 26, 2017

Announced availability of standard SQL TO_HEX and FROM_HEX functions:

• TO_HEX
• FROM_HEX

July 13, 2017

Announced support for using DML statements to modify tables with required fields:

• DML
• You can also update a table with required fields by using a SELECT statement and writing to a destination table.

June 19, 2017

Announced availability of new public datasets:

• World Bank: Education Data
• World Bank: Global Health, Nutrition, and Population Data
• World Bank: International Debt Data

June 07, 2017

Announced availability of new public dataset:

• NHTSA Traffic Fatality Data

June 06, 2017

Announced availability of new public dataset:

• OpenAQ: Real-time Air Quality Data

June 05, 2017

The following standard SQL statistical aggregate functions now support the OVER clause — STDDEV, STDDEV_POP, STDDEV_SAMP, VAR_POP, VAR_SAMP, VARIANCE:

• Statistical-aggregate-functions

June 02, 2017

Standard SQL now supports the TO_JSON_STRING function.

May 23, 2017

Added troubleshooting documentation for the BigQuery Data Transfer Service:

• Troubleshooting BigQuery Data Transfer Service Transfers

Added troubleshooting documentation for the BigQuery Data Transfer Service:

• Troubleshooting BigQuery Data Transfer Service

May 22, 2017

Announced availability of new public dataset:

• Healthcare Common Procedure Coding System (HCPCS) Level II

May 17, 2017

Announced free tier pricing for storage.

May 12, 2017

Announced availability of new public datasets:

• RxNorm
• United States Census Bureau International Data
• NOAA ICOADS Data

May 10, 2017

Announced availability of new IAM role:

• bigquery.jobUser

May 05, 2017

Announced availability of new public dataset:

• Chicago Taxi Trips Data

May 02, 2017

Announced availability of new public datasets:

• Chicago Crime Data
• EPA Historical Air Quality Data

April 20, 2017

Announced availability of new public dataset:

• United States Census Data

April 05, 2017

Updated versions of JDBC releases.

March 31, 2017

Announced availability of new public dataset:

• 1000 Cannabis Genomes Project

March 16, 2017

Updated limits for maximum rows per second when streaming data into BigQuery.

Standard SQL now supports REVERSE and ARRAY_REVERSE functions for reversing STRING, BYTES, and ARRAY types.

March 15, 2017

Standard SQL now supports several functions that use HyperLogLog++ to estimate the number of unique values in a large dataset.

Standard SQL now supports window, or analytic, mode for ARRAY_AGG and STRING_AGG using the OVER clause.

Updated limits for Avro headers and maximum file size for load jobs.

March 14, 2017

Updated versions of JDBC and ODBC releases.

March 09, 2017

Announced beta availability of BigQuery Data Transfer Service.

March 08, 2017

Announced general availability of Data Manipulation Language for nonpartitioned tables in standard SQL.

Increased daily limits for update and delete statements.

Announced beta availability of Querying Google Cloud Bigtable External Data Sources.

Announced general availability of IGNORE NULLS and RESPECT NULLS modifiers for ARRAY_AGG and APPROX_QUANTILES in standard SQL.

Announced daily limits for copying a table in BigQuery.

Announced availability of new public datasets:

• Bay Area Bike Share Trips Data
• San Francisco Fire Department Service Calls Data
• San Francisco 311 Service Requests Data
• San Francisco Street Trees Data
• San Francisco Police Reports Data

February 01, 2017

Standard SQL now supports:

• Bitwise operators for BYTES.
• The BIT_COUNT function for BYTES.
• The string functions LPAD, RPAD, and REPEAT.
• The FARM_FINGERPRINT fingerprinting function.
• Net functions for working with IP addresses: NET.IP_FROM_STRING, NET.SAFE_IP_FROM_STRING, NET.IP_TO_STRING, NET.IP_NET_MASK, NET.IP_TRUNC, NET.IPV4_FROM_INT64, and NET.IPV4_TO_INT64.

Announced support for copying non-partitioned tables to partitioned tables. For more information, see Copying to partitioned tables.

Announced availability of a new public dataset: New York City Tree Census data.

Announced availability of a new public dataset: US Bureau of Labor Statistics data.

December 22, 2016

Announced availability of custom null markers for CSV load jobs.

Standard SQL now supports the following code point functions: TO_CODE_POINTS, CODE_POINTS_TO_BYTES, CODE_POINTS_TO_STRING.

Standard SQL now supports ORDER BY and LIMIT clauses within ARRAY_AGG, ARRAY_CONCAT_AGG, and STRING_AGG.

Hash functions in standard SQL.

URL functions in standard SQL.

Standard SQL now supports parameterized queries, which help prevent SQL injection when queries are constructed using user input.

Announced availability of a new public dataset: New York City Citi Bike Trips data.

Announced availability of a new public dataset: NYC 311 Service Requests data.

Announced availability of a new public dataset: NYPD Motor Vehicle Collisions data.

If you use OMIT...IF to exclude a portion of a record in a repeating field, and the query also selects other independently repeating fields, BigQuery now returns an error. To avoid this error, migrate your query to standard SQL.

November 30, 2016

Announced availability of JSON functions in standard SQL.

Announced availability of a new public dataset: IRS Form 990 Data.

Announced publication of a new tutorial:

• Visualizing BigQuery Data Using Google Data Studio 360

November 07, 2016

Announced availability of a new public dataset: Medicare Data.

November 02, 2016

Announced availability of a new public dataset: Open Images Data.

November 01, 2016

Announced JDBC drivers.

You can set a SQL variant, either legacy SQL or standard SQL, by setting a query prefix.

Announced availability of a new public dataset: Major League Baseball Data.

Announced availability of standard SQL safe casting functions, which return a null value in cases where normal casting generates an error.

The BigQuery Client Libraries page now links exclusively to one set of libraries: the Google Cloud Client Libraries. Work is underway to update all sample code throughout the BigQuery documentation to use the Google Cloud Client Libraries.

Bracket notation is now supported for the legacy SQL JSON functions.

Additional documentation is available for access control using Identity and Access Management (IAM):

• A table that describes the permissions available in BigQuery.
• A table that describes the permissions required for methods.
• A table that lists all BigQuery IAM roles and the permissions included in each role.
• A section that describes the methods to use for programmatic manipulation of roles.

September 29, 2016

Announced general availability of standard SQL, which is based on the SQL 2011 standard, with extensions for querying nested and repeated data.

New documentation includes:

• Migrating from legacy SQL
• User-Defined Functions

Announced Data Manipulation Language (DML) functionality for standard SQL, which enables you to insert, update, and delete rows and columns in your BigQuery datasets.

For syntax details and examples, see DML Syntax.

Announced BigQuery Stackdriver Monitoring functionality, which enables you to monitor your BigQuery resources, including creating charts and alerts based on BigQuery metrics.

Announced ODBC drivers.

Flat-Rate Pricing gives you unlimited querying for a fixed monthly price.

You can now save and share queries.

August 26, 2016

The updated beta IAM predefined roles for BigQuery are now accessible to all.

August 01, 2016

When loading Avro files, BigQuery no longer creates an unnamed root-level column named "root." To ease the transition to this new schema, projects with the following contents are exempted from this change through a project override:

• Projects that contained federated tables based on Avro files as of August 1, 2016
• Projects that have queried or loaded Avro files in the 30 days prior to August 1, 2016

All affected project owners will be notified and provided the option to remove the project override.

July 27, 2016

Announced support for labeling datasets using the BigQuery API or command-line tool.

June 02, 2016

Announced support for time-partitioned tables.

Announced Beta availability of standard SQL, which is based on the SQL 2011 standard, with extensions for querying nested and repeated data.

Announced a Beta preview of Product Identity and Access Management (IAM) Roles.

May 06, 2016

Announced support for Google Drive:

• You can configure a Google Drive document as a federated data source.
• You can query a Google Drive data source using a temporary table.
• You can save query results to Google Sheets.

Query reference updates:

• Updated PARSE_PACKED_IP, which now returns BYTES.
• Added FROM_BASE64.
• Added SHA1.
• Added TO_BASE64.

March 23, 2016

Announced availability of BigQuery Public Datasets.

Announced Long Term Storage pricing.

Added support for Avro source format for load operations and as an federated data source in the BigQuery API or command-line tool.

Added support for CSV and JSON schema auto-detection for federated data sources in the BigQuery API or command-line tool.

Added support for the BYTES data type.

Updated the interface for creating tables in the BigQuery web UI.

December 15, 2015

Added Custom quotas, which allow you to set project-wide billing alerts that help prevent runaway costs.

Added Query explain, which shows, stage by stage, how BigQuery executes your queries.

Added improvements to the Streaming API:

• You can use the Streaming API in EU locations.
• You can use Template tables to manage related tables used for streaming.
• No more warm-up delay. You can now query your data within a few seconds of the first streaming insertion.

Added support for analyzing Google Cloud Audit logs that enable you to track usage and access.

August 25, 2015

Added support for user-defined functions (UDFs).

Added the ability to query CSV and JSON data directly from Google Cloud Storage. For more information, see federated data sources.

Added ability to cancel a job

Pricing updates:

• Introduced BigQuery Slots, which lets you guarantee resources, regardless of demand on the overall multi-tenant pool.
• Introduced query pricing tiers.

Quota updates: * Increased the concurrent rate limit to 50 queries. * Increased the daily query limit to 100,000 queries.

BigQuery web UI improvements:

• Tables named with a date suffix (for example, when using the TABLE_DATE_RANGE function) display as a collapsed item in the navigation panel.
• User-defined function (UDF) support.
• A new "Format Query" button.

Added streaming buffer statistics for tables that are being actively modified by streaming inserts. These statistics offer better information about table size and availability for copy/extract, and they can also help with debugging.

April 16, 2015

Added ability to load data from Google Cloud Datastore.

Increased the API requests per second, per user quota from 10 to 100 requests per second.

Query reference updates:

• Added COALESCE.
• Added GROUP_CONCAT_UNQUOTED.
• Added support for RIGHT OUTER JOIN and FULL OUTER JOIN.
• Added ROLLUP and GROUPING modifiers.

Added ability to undelete a table using table decorators.

March 25, 2014

Added views. A view is a virtual table defined by a SQL query.

Added JSON functions.

Added table wildcard functions, for a cost-effective way to query data from a specific set of tables.

Improved JOIN support by adding CROSS JOIN and multiple JOIN functionality. For more information, see joins.

Increased quota limits for streaming data into BigQuery.

Added a description property for datasets, tables and fields.

Updated the concurrent rate limit and export request quota in the quota policy.

Added parallel export capability.

Changed the pricing model, lowering rates and adding reserved capacity pricing.

Updated batch query functionality. Batch queries no longer have a 30 minute execution delay. Instead, batch jobs now begin as soon as idle resources are available. If the job hasn't started within 3 hours, the job priority changes to interactive.

September 18, 2013

Added support for streaming inserts.

Added table decorators for time-based table snapshots.

Added new aggregate functions.

June 11, 2013

Added window functions.

Added support for package-based pricing.

Added query caching.

Updated quota policies.

Added support for returning large query results.

March 14, 2013

Added TIMESTAMP data type functionality.

Added GROUP EACH BY functionality for aggregations of more distributed values.

Added JOIN EACH functionality for joining tables of any size.

Added ability to add columns to table schema after the data is already loaded into BigQuery. To add columns to a table schema, pass the new schema to the schema parameter of Tables.update().

October 01, 2012

API: New support for importing JSON files and nested and repeated fields

API: New quota limits

August 28, 2012

API: New batch queries feature allows for queries to be executed between 30 minutes to 3 hours later, but are charged at a lower cost than interactive queries which execute immediately

New tool release: BigQuery Connector for Excel now allows you to make queries to Google BigQuery from Microsft Excel!

August 17, 2012

API: CSV chunking - when importing, you can now specify the allowQuotedNewlines flag to be FALSE in the bq command-line tool, or in the API, and the import will be chunked, or processed in parallel. If you have quoted new lines in your data, be sure to set this flag to be TRUE.

API: Deletion of large tables has been fixed

API: Improved validation of project IDs

July 16, 2012

BigQuery Browser Tool: Added syntax highlighting

BigQuery Browser Tool: Add a drag handle to the separator below the query editor

BigQuery Browser Tool: New favicon

BigQuery Browser Tool: Fixed some keyboard shortcuts in the query editor

API: Project owners can now call bigquery.jobs.get for jobs created by other users

May 30, 2012

API: Introduced new quota enforcement schema. User can now have one (1) large query and one (1) pipeline job (import, export, streaming, etc.) concurrently.

API: Added support for leading underscores in field name

API: Improved error handling

API: Added token based pagination support for list operations. Clients can use token based pagination to page through a single table snapshot. Subsequent changes to the table are not reflected when paging through list results.

API: Improved list operations for tables and datasets

May 11, 2012

API: Table expiration time is now exposed

API: Better errors for field/rows being too long

API: Custom delimiters added

April 20, 2012

BigQuery Browser Tool: New welcome page and more helpful messages when there aren't any queries or jobs to display

API

• Clearer quota error messages
• Copy jobs are now atomic

April 13, 2012

BigQuery Browser Tool

• Add retries for connection errors
• Table view: Field names in schemas are left-aligned
• Table view: Removed "Actions" menu (use the nav panel instead)
• Table view: Added "Query Table" menu, which prefills query builder with a table query
• Query builder has simpler prefill logic and a better help bubble

API: Remove bigquery.jobs.delete

April 05, 2012

BigQuery Browser Tool

• 10MB limit to uploads done through the BigQuery browser tool
• Bug fix for zero row results

API

• API Discovery docs have typed configuration
• Bug fix limiting the amount of data returned in a TableData.List
• Atomic updates of tables for all import jobs
• Join behavior tightened (no outer join, only left most table can be large)

March 09, 2012

BigQuery Browser Tool

• Pressing the tab key in the query box now adds a tab instead of tabbing to the next UI element
• Pressing the enter key in the query box now preserves leading whitespace to the next lines
• Pressing the enter key no longer starts a query but will always add a newline to the box, and CTRL+ENTER runs the query
• Navigation panel no longer forgets scroll location and submenu state when jobs are running
• BigQuery browser tool now has a new URL at http://bigquery.cloud.google.com; it is still possible to access the browser tool using the old URL

March 01, 2012

BigQuery Browser Tool

• Faster BigQuery browser tool load time
• "Max invalid records" support in the BigQuery browser tool

API

• Enable support for VARIANCE, STDDEV, and AVG aggregate functions
• Add job.kind to Jobs:list API response
• Better error messages for quota-related errors

February 01, 2012

BigQuery Browser Tool

• Show previous query results option in the query panel
• The current project view is now highlighted
• "Refresh" option in the projects drop-down menu

API

• Support for loading gzip compressed files from Google Cloud Storage
• Error handling
  • Faster detection of unreadable files
  • Improvements in handline multibyte UTF-8 characters

November 01, 2011

BigQuery Browser Tool

The Web UI now allows JSON schema format to be used in the Create Table dialog.

This is useful in situations where developers may want to use the same schema format for API calls, the bq command-line client, and the Web UI.

API

• The default configuration.load.createDisposition for load jobs is now CREATE_IF_NEEDED
• "load" jobs now allow users to specify a number of rows to skip
  • This is useful for skipping header rows or other metadata that is not parseable when you import CSV files.
  • This value applies to all files in the source list.
• BigQuery can now load data from files that contain delimiters other than the comma
  • Delimiters may be any character expressable as a single Latin-1 byte other than ", , and .
  • Example (using the bq command line tool): bq load --field_delimiter=: newtable gs://bucket/input.txt

October 01, 2011

BigQuery V2 Release

• A new REST API
• A new web user interface
• Support for JOIN statements
• Ability to export a table or query result to a CSV file in Google Cloud Storage
• Support for ACLs on groups of tables (datasets)
• A new object architecture describing tables, groups of tables, and queries