Introduction to connections

BigQuery lets you query data that's stored outside of BigQuery in Google Cloud services like Cloud Storage or Cloud Spanner, or in third-party sources like AWS or Azure. These external connections use the BigQuery Connection API.

For example, suppose that you store details about customer orders in Cloud SQL and data about sales in BigQuery, and you want to join the two tables in a single query. You can create a Cloud SQL connection to the external database by using the BigQuery Connection API. With connections, you never send database credentials as cleartext.

A connection is encrypted and securely stored in the BigQuery connection service. You can give users access to connections by granting them BigQuery connection Identity and Access Management (IAM) roles.

Connection types

BigQuery provides connections for the following external data sources:

• BigLake tables: BigLake connections help to connect the following external data sources while retaining fine-grained BigQuery access control and security:

  • Structured data in Cloud Storage
  • Unstructured data in Cloud Storage
  • Amazon S3 with BigQuery Omni
  • Azure Blob Storage with BigQuery Omni

  For more information, see Introduction to BigLake tables.

• Federated queries: Federated connections establish connections to the following databases before you send the query. Connections handle database credentials for federated queries.

  • Cloud Spanner
  • Cloud SQL

  For more information, see Introduction to federated queries.

• Remote functions: BigQuery remote functions let you implement functions with any supported languages in Cloud Functions or Cloud Run. A BigQuery connection lets you connect with Cloud Functions or Cloud Run and run these functions.

  For more information, see Working with remote functions and Analyze object tables by using remote functions.

• Apache Spark: Stored procedures for Apache Spark let you run stored procedures written in Python using BigQuery. A Spark connection lets you connect to Dataproc Serverless and run the stored procedures for Apache Spark.

  For more information, see Work with stored procedures for Apache Spark.

Note: You don't need a connection to query data in Cloud Bigtable and Google Drive.

Audit logs

BigQuery logs usage and management requests about connections. For more information, see BigQuery audit logs overview.

What's next

• Learn how to query stored data:
  • Query data stored in Amazon S3.
  • Query data stored in Blob Storage.
  • Query structured data stored in Cloud Storage.
  • Query unstructured data stored in Cloud Storage.
  • Query data stored in Spanner.
  • Query data stored in Cloud SQL.
  • Query data using remote functions.
  • Query unstructured data using remote functions.
  • Query data using stored procedures for Apache Spark.
• Learn how to manage connections.
• Learn about external tables.