By default, BigQuery encrypts customer content stored at rest. BigQuery handles and manages this default encryption for you without any additional actions on your part. First, data in a BigQuery table is encrypted using a data encryption key. Then, those data encryption keys are encrypted with key encryption keys, which is known as envelope encryption. Key encryption keys do not directly encrypt your data but are used to encrypt the data encryption keys that Google uses to encrypt your data. For more information, see Key management.
If you want to control encryption yourself, you can use customer-managed encryption keys (CMEK) for BigQuery. Instead of Google managing the key encryption keys that protect your data, you control and manage key encryption keys in KMS. This topic provides details about this technique.
Learn more about encryption options on Google Cloud.
Before you begin
Decide whether you are going to run BigQuery and KMS in the same Google Cloud project, or in different projects. For documentation example purposes, the following convention is used:
- project_id is the project ID of the project running BigQuery
- project_number is the project number of the project running BigQuery
- kms_project_id is the project ID of the project running KMS (even if this is the same project running BigQuery)
BigQuery is automatically enabled in new projects. If you are using a pre-existing project to run BigQuery, enable the BigQuery API.
For the Google Cloud project that runs KMS:
- Enable the KMS API.
- Create a key ring and a key as described in
Creating key rings and keys. Create the key ring
in a location that matches the location of your BigQuery
dataset:
-
Any multi-regional dataset should use a multi-regional key ring from a
matching location. For examples, a dataset in region
US
should be protected with a key ring from regionus
, and a dataset in regionEU
should be protected with a key ring from regioneurope
. -
Regional datasets should use a matching regional keys. For example,
a dataset in region
asia-northeast1
should be protected with a key ring from regionasia-northeast1
. -
The
global
region is not supported for use with BigQuery.
-
Any multi-regional dataset should use a multi-regional key ring from a
matching location. For examples, a dataset in region
Encryption specification
KMS keys used to protect your data in BigQuery are AES-256 keys. These keys are used as key encryption keys in BigQuery, in that they encrypt the data encryption keys that encrypt your data.
Grant encryption and decryption permission
Use the Google Cloud Console to determine the BigQuery service account ID, and provide the service account with the appropriate role to encrypt and decrypt using KMS.
Determine the service account ID
Classic UI
Go to the BigQuery web UI.
Click the down arrow icon
next to your project name in the navigation and then click Customer-Managed Encryption.
A user dialog opens to show you the service account that requires encryption and decryption permission:
Click Copy to copy the service account ID to your clipboard and then click OK to close the user dialog.
CLI
You can use the bq show
command with the --encryption_service_account
flag to determine the service account ID:
bq show --encryption_service_account
The command displays the service account ID:
ServiceAccountID ------------------------------------------------------------- bq-project_number@bigquery-encryption.iam.gserviceaccount.com
Assign the Encrypter/Decrypter role
Assign the Cloud KMS CryptoKey Encrypter/Decrypter
role to the
BigQuery system service account that you copied to your
clipboard. This account is of the form
bq-project_number@bigquery-encryption.iam.gserviceaccount.com
Console
Open the Security page in the Cloud Console.
Select your project and click Continue.
Identify the encryption key to which you want to add the role.
- If the
bq-project_number@bigquery-encryption.iam.gserviceaccount.com
service account isn't already on the members list, it doesn't have any roles assigned to it. Click Add member and enter the email address of the service account,bq-project_number@bigquery-encryption.iam.gserviceaccount.com
. - If the service account is already on the members list, it has
existing roles. Click the current role drop-down list for the
bq-project_number@bigquery-encryption.iam.gserviceaccount.com
service account.
- If the
Click the drop-down list for Role, click Cloud KMS, and then click the Cloud KMS CryptoKey Encrypter/Decrypter role.
Click Add or Save to apply the role to the
bq-project_number@bigquery-encryption.iam.gserviceaccount.com
service account.
CLI
You can use the gcloud
command-line tool to assign the role:
gcloud kms keys add-iam-policy-binding \ --project=kms_project_id \ --member serviceAccount:bq-project_number@bigquery-encryption.iam.gserviceaccount.com \ --role roles/cloudkms.cryptoKeyEncrypterDecrypter \ --location=kms_key_location \ --keyring=kms_key_ring \ kms_key
Replace kms_project_id with the ID of your Google Cloud project that is running KMS, replace project_number with the project number (not project ID) of your Google Cloud project that is running BigQuery, and replace kms_key_location, kms_key_ring, kms_key with location, key ring and key names of your KMS key.
Key resource ID
The resource ID for the KMS key is required for CMEK use, as shown in the examples of this topic. This key is in the form:
projects/project_id/locations/location/keyRings/key_ring/cryptoKeys/key
For information on how to retrieve the key resource ID, see Key resource ID.
Create a table protected by KMS
Create an empty table protected by KMS
To create a table that is protected by KMS:
Classic UI
Click the down arrow icon
next to your dataset name in the BigQuery web user interface and then click Create new table.
On the Create table page, fill in the information needed to create an empty table with a schema definition. Before you click Create Table, set the encryption type and specify the KMS key to use with the table:
- Click on the drop-down list for Encryption Type and select Customer-Managed Encryption.
- For Customer-Managed Encryption Key, enter the resource ID for the key.
Click Create Table.
CLI
You can use the bq
command-line tool with the --destination_kms_key
flag
to create the table. The --destination_kms_key
flag specifies the
resource ID of the key to use with the table.
To create an empty table with a schema:
bq mk --schema name:string,value:integer -t \ --destination_kms_key projects/project_id/locations/location/keyRings/key_ring/cryptoKeys/key \ mydataset.newtable
Alternatively, you can use a DDL statement:
bq query --use_legacy_sql=false " CREATE TABLE mydataset.newtable (name STRING, value INT64) OPTIONS( kms_key_name='projects/project_id/locations/location/keyRings/key_ring/cryptoKeys/key' ) "
To create a table from a query:
bq query --destination_table=mydataset.newtable \ --destination_kms_key projects/project_id/locations/location/keyRings/key_ring/cryptoKeys/key \ "SELECT name,count FROM mydataset.babynames WHERE gender = 'M' ORDER BY count DESC LIMIT 6"
For more information about the bq
command-line tool, see
bq command-line tool.
Go
Python
Protect a new table with a customer-managed encryption key by setting the
Table.encryption_configuration
property to an EncryptionConfiguration
object before creating the table.
Query a table protected by a KMS key
No special arrangements are required to query a table protected by KMS. BigQuery stores the name of the key used to encrypt the table content and will use that key when a table protected by KMS is queried.
All existing tools, the BigQuery console, and the bq
command-line interface run the same way as with default-encrypted
tables, as long as BigQuery has access to the KMS
key used to encrypt the table content.
Protect query results with KMS key
Classic UI
Click the Compose query button in the BigQuery web UI.
Enter a valid BigQuery SQL query in the New Query text area.
Click Encryption Type and select Customer-Managed Encryption.
For Customer-Managed Encryption Key, enter the resource ID for the key.
Click Run Query.
CLI
Specify the flag --destination_kms_key
to protect the destination table or
query results (if using a temporary table) with your KMS key.
The --destination_kms_key
flag specifies the
resource ID of the key to use with the destination or
resulting table.
Optionally use the --destination_table
flag to specify the destination for
query results. If --destination_table
is not used, the query results will
be written to a temporary table.
To query a table:
bq query \ --destination_table=mydataset.newtable \ --destination_kms_key projects/project_id/locations/location/keyRings/key_ring/cryptoKeys/key \ "SELECT name,count FROM mydataset.babynames WHERE gender = 'M' ORDER BY count DESC LIMIT 6"
For more information about the bq
command-line tool, see
bq command-line tool.
Go
Python
Load a table protected by KMS
To load a data file into a table that is protected by KMS:
Console
Protect a load job destination table with a customer-managed encryption key by specifying the key when you load the table.
- Open the BigQuery web UI in the Cloud Console.
Go to the BigQuery web UI - In the navigation panel, in the Resources section, expand your project and select a dataset.
- On the right side of the window, in the details panel, click Create table.
- Enter the options you want to use for loading the table, but before you click Create table, click Advanced options.
- Under Encryption, select Customer-managed key.
Click the Select a customer-managed key dropdown and select the key to use. If you don't see any keys available, enter a key resource ID.
Click Create table.
CLI
Protect a load job destination table with a customer-managed encryption
key by setting the --destination_kms_key
flag.
bq --location=location load \ --autodetect \ --source_format=format \ --destination_kms_key projects/project_id/locations/location/keyRings/key_ring/cryptoKeys/key \ dataset.table \ path_to_sourceFor example:
bq load \ --autodetect \ --source_format=NEWLINE_DELIMITED_JSON \ --destination_kms_key projects/project_id/locations/location/keyRings/key_ring/cryptoKeys/key \ test2.table4 \ gs://cloud-samples-data/bigquery/us-states/us-states.json
Go
Python
Protect a load job destination table with a customer-managed encryption
key by setting the
LoadJobConfig.destination_encryption_configuration
property to an EncryptionConfiguration
and load the table.
Stream into a table protected by KMS
You can stream data into your CMEK-protected BigQuery table without specifying any additional parameters. Note that this data is encrypted using your KMS key in the buffer as well as in the final location. Before using streaming with a CMEK table, review the requirements on key availability and accessibility.
Learn more about streaming at Streaming data into BigQuery.
Change a table from default encryption to KMS protection
CLI
You can use the bq cp
command with the --destination_kms_key
flag
to copy a table protected by default encryption into a new table, or into
the original table, protected by KMS. The
--destination_kms_key
flag specifies the resource ID
of the key to use with the destination table.
To copy a table that has default encryption to a new table that has KMS protection:
bq cp \ --destination_kms_key projects/project_id/locations/location/keyRings/key_ring/cryptoKeys/key \ sourceDataset.sourceTableId destinationDataset.destinationTableId
In you want to copy a table that has default encryption to the same table with KMS protection:
bq cp -f \ --destination_kms_key projects/project_id/locations/location/keyRings/key_ring/cryptoKeys/key \ sourceDataset.sourceTableId sourceDataset.sourceTableId
If you want to change a table from KMS protection to default
encryption, copy the file to itself by running bq cp
without using the
--destination_kms_key
flag.
For more information about the bq
command-line tool, see
bq command-line tool.
Go
Python
Before trying this sample, follow the Python setup instructions in the
BigQuery Quickstart Using Client Libraries
.
For more information, see the
BigQuery Python API reference documentation
.
Protect the destination of a table copy with a customer-managed
encryption key by setting the QueryJobConfig.destination_encryption_configuration
property to an EncryptionConfiguration
and copy the table.
Determine if a table is protected by KMS
In the BigQuery web UI, click the blue arrow to the left of your dataset to expand it, or double-click the dataset name. This displays the tables and views in the dataset.
Click the table name.
Click Details. The Table Details page displays the table's description and table information.
If the table is protected by KMS, the Customer-Managed Encryption Key field will display the key resource ID.
Change the KMS key for a BigQuery table
To change the KMS key of an existing CMEK-protected table, you
can run an ALTER TABLE
query, use the API, or use the bq
command-line tool.
There are two ways to modify the KMS key using the API and
command-line tool: update
or cp
. If you use update
, you can change the
KMS key used for a KMS-protected table. If you use cp
,
you can change the KMS key used for a CMEK-protected table,
change a table from default encryption to CMEK-protection, also change a table
from CMEK-protection to default encryption. An advantage of update
is it is
faster than cp
and it allows the use of
table decorators.
Classic UI
Go to the BigQuery web UI.
Click Compose query.
Type your DDL statement into the New Query text area. For the
kms_key_name
value, specify the resource ID of the key that you want to use to protect the table.#standardSQL ALTER TABLE mydataset.mytable SET OPTIONS ( kms_key_name="projects/[PROJECT_ID]/locations/[LOCATION]/keyRings/[KEYRING]/cryptoKeys/[KEY]" )
CLI
You can use the bq cp
command with the --destination_kms_key
flag
to change the key for a table protected by KMS. The
--destination_kms_key
flag specifies the resource ID
of the key to use with the table.
bq update \ --destination_kms_key projects/project_id/locations/location/keyRings/key_ring/cryptoKeys/key \ -t dataset_id.table_id
Go
Python
Change the customer-managed encryption key for a table by changing the
Table.encryption_configuration
property to a new EncryptionConfiguration
object and update the table.
Set a dataset default key
You can set a dataset-wide default KMS key that will apply to all newly-created tables within the dataset, unless a different KMS key is specified when you create the table. The default key does not apply to existing tables. Changing the default key does not modify any existing tables and applies only to new tables created after the change.
You can apply, change, or remove a dataset default key by
specifying the default key in the
EncryptionConfiguration.kmsKeyName
field when you call thedatasets.insert
ordatasets.patch
methodsspecifying the default key in the
--default_kms_key
flag when you run thebq mk --dataset
command.
Remove BigQuery's access to the KMS key
You can remove BigQuery's access to the KMS key at any time, by revoking the IAM permission for that key.
If BigQuery loses access to the KMS key, the user experience can suffer significantly and data loss may occur:
Data in these CMEK-protected tables can no longer be accessed —
query
,cp
,extract
, andtabledata.list
will all fail.No new data can be added to these CMEK-protected tables.
Even once access is granted back, the performance of queries to these tables can be degraded for multiple days.
Impact of KMS key rotation
BigQuery does not automatically rotate a table encryption key when the KMS key associated with the table rotates. Existing tables continue to use the key version with which they were created. New tables will use the current key version.
Limitations
BigQuery access to the KMS key
A KMS key is considered available and accessible by BigQuery if:
- the key is enabled
- the BigQuery service account has encrypt and decrypt permissions on the key
The following sections describe impact to streaming inserts and long-term inaccessible data when a key is inaccessible.
Impact to streaming inserts
The KMS key must be available and accessible for at least 24 consecutive hours in the 48-hour period following a streaming insertion request. If the key is not available and accessible, the streamed data may not be fully persisted and can be lost. For more information on streaming inserts, see Streaming data into BigQuery.
Impact to long-term inaccessible data
As BigQuery provides managed storage, long-term inaccessible data is not compatible with BigQuery's architecture. If the KMS key of a given BigQuery table is not available and not accessible for 60 consecutive days, BigQuery may choose to delete the table and its associated data. Before the data is deleted, BigQuery will send an email to the email address associated with the billing account at least 7 days before the deletion.
Using table decorators
When data in a table protected by KMS is replaced via write
disposition WRITE_TRUNCATE
for load
, cp
, or query
operations, the
table becomes inaccessible for query via
table decorators depending on the snapshot
decorator time.
Assuming a table was replaced at time T
and snapshot decorator snapshot_time
is for a time less than T
, the following table shows whether you can query for
snapshot_time
:
Encryption type before T |
Encryption type after T |
snapshot_time |
---|---|---|
KMS encrypted | KMS encrypted | Cannot query |
Default encrypted | KMS encrypted | Can query |
KMS encrypted | Default encrypted | Cannot query |
Note that similar logic applies to
<time2>
when a range decorator
is used.
Frequently asked questions
Who needs permission to the KMS key?
With customer-managed encryption keys, specifying permissions repeatedly is not required. As long as the BigQuery service account has permission to use the KMS key to encrypt and decrypt, anyone with permission to the BigQuery table can access the data — even if they don't have direct access to the KMS key.
Which service account is used?
The BigQuery service account associated with the Google Cloud project of the table is used to decrypt that table's data. The BigQuery service accounts are unique for each project. For a job that writes data into a KMS-protected anonymous table, the job's project's service account is used.
As an example, consider three CMEK-protected tables: table1
, table2
, and
table3
. To query data from {project1.table1, project2.table2}
with
destination table {project3.table3}
:
- Use the
project1
service account forproject1.table1
- Use the
project2
service account forproject2.table2
- Use the
project3
service account forproject3.table3
In what ways can BigQuery use my KMS key?
BigQuery will use the KMS key to decrypt data in
response to a user query, for example,
tabledata.list
or
jobs.insert
.
BigQuery can also use the key for data maintenance and storage optimization tasks, like data conversion into a read-optimized format.
What cryptography libraries are used?
BigQuery relies on KMS for CMEK functionality. KMS uses Tink for encryption.
How to get more help?
If you have questions that are not answered here, reach out to cmek-feedback@google.com, see BigQuery support, or see KMS support.
Troubleshooting errors
The following describes common errors and recommended mitigations.
Error | Recommendation |
---|---|
Please grant Cloud KMS CryptoKey Encrypter/Decrypter role | The BigQuery service account associated with your project does not have sufficient Cloud IAM permission to operate on the specified KMS key. Follow the instructions in the error or in this documentation to grant the proper Cloud IAM permission. |
Existing table encryption settings do not match encryption settings specified in the request | This can occur in scenarios where the destination table has encryption
settings that do not match the encryption settings in your request. As
mitigation, use write disposition TRUNCATE to replace the table,
or specify a different destination table. |
This region is not supported | The region of the KMS key does not match the region of the
BigQuery dataset of the destination table. As a mitigation,
select a key in a region that matches your dataset, or load into a dataset
that matches the key region.
|