Create and set up a Cloud resource connection
As a BigQuery administrator, you can create a Cloud resource connection that enables data analysts to perform the following tasks:
- Query structured Cloud Storage data using BigLake tables. BigLake tables enable you to query external data with access delegation.
- Query unstructured data in Cloud Storage using object tables.
- Implement remote functions with any supported languages in Cloud Run functions or Cloud Run.
For more information about connections, see Introduction to connections.
Before you begin
Enable the BigQuery Connection API.
-
To get the permissions that you need to create a Cloud Resource connection, ask your administrator to grant you the following IAM roles:
-
BigQuery Connection Admin (
roles/bigquery.connectionAdmin
) on the project -
Storage Object Viewer (
roles/storage.objectViewer
) on the bucket
For more information about granting roles, see Manage access to projects, folders, and organizations.
You might also be able to get the required permissions through custom roles or other predefined roles.
If you want to query structured data using BigLake tables based on Cloud Storage or unstructured data using object tables, then the service account associated with the connection must also have the Storage Viewer (roles/storage.viewer
) role on the bucket that contains the external data.
-
BigQuery Connection Admin (
- Ensure that your version of the Google Cloud SDK is 366.0.0 or later:
gcloud version
If needed, update the Google Cloud SDK.
- Optional: For Terraform, use Terraform GCP version 4.25.0 or later. You can download the latest version from HashiCorp Terraform downloads.
Location consideration
When you use Cloud Storage to store data files, we recommend that you use Cloud Storage single-region or dual-region buckets for optimal performance, not multi-region buckets.
Create Cloud resource connections
BigLake uses a connection to access Cloud Storage. You can use this connection with a single table or a group of tables.
Select one of the following options:
Console
Go to the BigQuery page.
To create a connection, click
Add, and then click Connections to external data sources.In the Connection type list, select Vertex AI remote models, remote functions and BigLake (Cloud Resource).
In the Connection ID field, enter a name for your connection.
Click Create connection.
Click Go to connection.
In the Connection info pane, copy the service account ID for use in a later step.
bq
In a command-line environment, create a connection:
bq mk --connection --location=REGION --project_id=PROJECT_ID \ --connection_type=CLOUD_RESOURCE CONNECTION_ID
The
--project_id
parameter overrides the default project.Replace the following:
REGION
: your connection regionPROJECT_ID
: your Google Cloud project IDCONNECTION_ID
: an ID for your connection
When you create a connection resource, BigQuery creates a unique system service account and associates it with the connection.
Troubleshooting: If you get the following connection error, update the Google Cloud SDK:
Flags parsing error: flag --connection_type=CLOUD_RESOURCE: value should be one of...
Retrieve and copy the service account ID for use in a later step:
bq show --connection PROJECT_ID.REGION.CONNECTION_ID
The output is similar to the following:
name properties 1234.REGION.CONNECTION_ID {"serviceAccountId": "connection-1234-9u56h9@gcp-sa-bigquery-condel.iam.gserviceaccount.com"}
Terraform
Append the following section into your main.tf
file.
## This creates a cloud resource connection. ## Note: The cloud resource nested object has only one output only field - serviceAccountId. resource "google_bigquery_connection" "connection" { connection_id = "CONNECTION_ID" project = "PROJECT_ID" location = "REGION" cloud_resource {} }Replace the following:
CONNECTION_ID
: an ID for your connectionPROJECT_ID
: your Google Cloud project IDREGION
: your connection region
Grant access to the service account
To create remote functions, you must grant required roles to Cloud Run functions or Cloud Run.
To connect to Cloud Storage, you must give the new connection read-only access to Cloud Storage so that BigQuery can access files on behalf of users.
Select one of the following options:
Console
We recommend that you grant the connection resource service account the
Storage Object Viewer IAM role
(roles/storage.objectViewer
), which lets the service account access
Cloud Storage buckets.
Go to the IAM & Admin page.
Click
Add.The Add principals dialog opens.
In the New principals field, enter the service account ID that you copied earlier.
In the Select a role field, select Cloud Storage, and then select Storage Object Viewer.
Click Save.
gcloud
Use the gcloud storage buckets add-iam-policy-binding
command:
gcloud storage buckets add-iam-policy-binding gs://BUCKET \ --member=serviceAccount:MEMBER \ --role=roles/storage.objectViewer
Replace the following:
BUCKET
: the name of your storage bucket.MEMBER
: the service account ID that you copied earlier.
For more information, see Add a principal to a bucket-level policy.
Terraform
Append the following section to your main.tf
file.
## This grants permissions to the service account of the connection created in the last step. resource "google_project_iam_member" "connectionPermissionGrant" { project = "PROJECT_ID" role = "roles/storage.objectViewer" member = format("serviceAccount:%s", google_bigquery_connection.connection.cloud_resource[0].service_account_id) }
Share connections with users
You can grant the following roles to let users query data and manage connections:
roles/bigquery.connectionUser
: enables users to use connections to connect with external data sources and run queries on them.roles/bigquery.connectionAdmin
: enables users to manage connections.
For more information about IAM roles and permissions in BigQuery, see Predefined roles and permissions.
Select one of the following options:
Console
Go to the BigQuery page.
Connections are listed in your project, in a group called External connections.
In the Explorer pane, click your project name > External connections > connection.
In the Details pane, click Share to share a connection. Then do the following:
In the Connection permissions dialog, share the connection with other principals by adding or editing principals.
Click Save.
bq
You cannot share a connection with the bq command-line tool. To share a connection, use the Google Cloud console or the BigQuery Connections API method to share a connection.
API
Use the
projects.locations.connections.setIAM
method
in the BigQuery Connections REST API reference section, and
supply an instance of the policy
resource.
Java
Before trying this sample, follow the Java setup instructions in the BigQuery quickstart using client libraries. For more information, see the BigQuery Java API reference documentation.
To authenticate to BigQuery, set up Application Default Credentials. For more information, see Set up authentication for client libraries.
What's next
- Learn about different connection types.
- Learn about managing connections.
- Learn about BigLake tables.
- Learn how to create BigLake tables.
- Learn how to upgrade external tables to BigLake tables.
- Learn about object tables and how to create them.
- Learn how to implement remote functions.