BigQuery Omni lets you run BigQuery analytics on data stored in AWS S3 or Azure blob storage.
Many organizations store data in multiple public clouds. Often, this data ends up being siloed, because it's hard to get insights across all of the data. You don't necessarily want to move or copy the data to a central location, because of cost, time, data governance, or problems caused by data duplication.
By using BigQuery Omni, you don't have to copy the data into Google Cloud. Instead, BigQuery Omni brings the BigQuery analytics engine to your data where it resides.
How it works
BigQuery's architecture separates compute from storage, allowing BigQuery to scale out as needed to handle very large workloads. BigQuery Omni extends this architecture by running the BigQuery query engine in other clouds. As a result, you don't have to physically move data into BigQuery storage. Processing happens where that data already sits.
Query results can be returned to Google Cloud over a secure connection — for example, to be displayed in the Google Cloud Console. Alternatively, you can write the results directly to Amazon S3 storage or Azure blob storage. In that case, there is no cross-cloud movement of the query results.
BigQuery Omni uses standard AWS IAM roles or Azure Active Directory principles to access the data in your subscription. You delegate read or write access to BigQuery Omni, and you can revoke access at any time.
Benefits of BigQuery Omni
Performance. You can get insights faster, because data is not copied across clouds, and queries run in the same region where your data resides.
Cost. You save on network egress costs because the data doesn't move. There are no additional charges to your AWS or Azure account related to BigQuery Omni analytics, because the queries run on clusters managed by Google. You are only billed for running the queries, using the BigQuery pricing model.
Security and data governance. You manage the data in your own AWS or Azure subscription. You don't need to move or copy the raw data out of your public cloud. All computation happens in the BigQuery multi-tenant service which runs within the same region as your data.
Serverless architecture. Like the rest of BigQuery, BigQuery Omni is a serverless offering. Google deploys and manages the clusters that run BigQuery Omni. You don't need to provision any resources or manage any clusters.
Ease of management. BigQuery Omni provides a unified management interface through Google Cloud. BigQuery Omni can use your existing Google Cloud account and BigQuery projects. You can write a standard SQL query in the Cloud Console to query data in AWS or Azure, and see the results displayed in the Cloud Console.
BigQuery Omni limitations include the following:
- You can't create standard tables in BigQuery Omni. BigQuery Omni only supports external tables.
- All limitations for external tables apply to BigQuery Omni external tables.
- The maximum result size for interactive queries is 2 MB.
- Jobs and reservations metadata views are not available for BigQuery Omni tables.
For more information on
INFORMATION_SCHEMAviews, see Introduction to BigQuery INFORMATION_SCHEMA.
- Materialized views for BigQuery Omni external tables are not supported.
- The following SQL statements are not supported
- BigQuery ML statements
- Data definition language (DDL) statements that require data managed in
BigQuery are not supported. For example
CREATE EXTERNAL TABLEor
CREATE SCHEMAare supported but
CREATE MATERIALIZED VIEWis not.
- Data manipulation language (DML) statements.
- Data control language (DCL) statements.
- SQL statements for reserving and assigning slots are not supported.
- Scheduled queries are only supported through the API or CLI.
- The destination table
option is disabled for queries in BigQuery Omni.
EXPORTqueries are allowed.
BigQuery Omni offers flat-rate pricing, which provides a predictable cost for queries. To enable flat-rate pricing, use BigQuery Reservations to purchase slots in the one of the BigQuery Omni regions.
For pricing information, see BigQuery Omni Pricing.