Using customer-managed encryption keys

This page describes how to use a Cloud Key Management Service (Cloud KMS) encryption key with Cloud Dataflow. A customer-managed encryption key (CMEK) enables encryption of data at rest with a key that you can control through Cloud KMS. You can create a batch or streaming pipeline that is protected with a CMEK or access CMEK-protected data in sources and sinks.

You can also use Cloud HSM, a cloud-hosted Hardware Security Module (HSM) service that allows you to host encryption keys and perform cryptographic operations in a cluster of FIPS 140-2 Level 3 certified HSMs. For more information on additional quotas for Cloud HSM, see Cloud KMS Quotas.

For more information, see encryption options on Google Cloud Platform.

Beta

This feature is in a pre-release state and might change or have limited support. For more information, see the product launch stages.

Note: You currently cannot use CMEK with Cloud Dataflow Shuffle and Streaming Engine. Only Persistent Disks attached to Cloud Dataflow workers and used for Persistent Disk-based shuffle and streaming state storage have CMEK support.

Before you begin

1. Verify that you have the Apache Beam SDK for Java >= 2.13.0 or the Apache Beam SDK for Python >= 2.13.0.

   For more information, see Installing the Apache Beam SDK.

2. Decide whether you are going to run Cloud Dataflow and Cloud KMS in the same Google Cloud Platform (GCP) project or in different projects. This page uses the following convention:

   • PROJECT_ID is the project ID of the project that is running Cloud Dataflow.
   • PROJECT_NUMBER is the project number of the project that is running Cloud Dataflow.
   • KMS_PROJECT_ID is the project ID of the project that is running Cloud KMS.

   For information about GCP project IDs and project numbers, see Identifying projects.

3. On the GCP project that you want to run Cloud KMS:

   1. Enable the Cloud KMS API.

   2. Create a key ring and a key as described in Creating symmetric keys. Cloud KMS and Cloud Dataflow are both regionalized services. Create the key ring in a location that matches the location of your Cloud Dataflow worker instances:

      • For a Cloud Dataflow job specified with region, select keys for use in the same region.
      • For a Cloud Dataflow job specified with zone, select keys for use in the same region.

Caution: For best results, do not use global keys with your Dataflow pipelines. Instead, use regional keys.

Granting Encrypter/Decrypter permissions

Note: If you use the Google Cloud Platform Console and the Create job from template page, a prompt appears to grant the encrypt and decrypt permissions to your Compute Engine service account and Cloud Dataflow service account as part of your job creation flow. If you follow that prompt to grant the permissions, you can skip this section, which describes how to manually set these permissions.

1. Assign the Cloud KMS CryptoKey Encrypter/Decrypter role to the Cloud Dataflow service account. This grants your Cloud Dataflow service account the permission to encrypt and decrypt with the CMEK you specify. If you use the Google Cloud Platform Console and the Create job from template page, this permission is granted automatically and you can skip this step.

   Use the gcloud command-line tool to assign the role:

   gcloud projects add-iam-policy-binding KMS_PROJECT_ID \
   --member serviceAccount:[e-mail] \
   --role roles/cloudkms.cryptoKeyEncrypterDecrypter
   

   Replace KMS_PROJECT_ID with the ID of your GCP project that is running Cloud KMS, and replace PROJECT_NUMBER with the project number (not project ID) of your GCP project that is running the Cloud Dataflow resources.

2. Assign the Cloud KMS CryptoKey Encrypter/Decrypter role to the Compute Engine service account. This grants your Compute Engine service account the permission to encrypt and decrypt with the CMEK you specify.

   Use the gcloud command-line tool to assign the role:

   gcloud projects add-iam-policy-binding KMS_PROJECT_ID \
   --member serviceAccount:[e-mail] \
   --role roles/cloudkms.cryptoKeyEncrypterDecrypter
   

   Replace KMS_PROJECT_ID with the ID of your GCP project that is running Cloud KMS, and replace PROJECT_NUMBER with the project number (not project ID) of your GCP project that is running the Compute Engine resources.

Create a pipeline that is protected by Cloud KMS

When you create a batch or streaming pipeline, you can select a Cloud KMS key to encrypt the pipeline state. The pipeline state is the data that is stored by Cloud Dataflow in temporary storage.

Command-line interface

To create a new pipeline with pipeline state that is protected by a Cloud KMS key, add the relevant flag to the pipeline parameters. The following example demonstrates running a word count pipeline with Cloud KMS.

Java

Cloud Dataflow does not support creating default Cloud Storage paths for temporary files when using a Cloud KMS key. You must specify gcpTempLocation.

mvn compile exec:java -Dexec.mainClass=org.apache.beam.examples.WordCount \
  -Dexec.args="--inputFile=gs://dataflow-samples/shakespeare/kinglear.txt \
               --output=gs://STORAGE_BUCKET/counts \
               --runner=DataflowRunner --project=PROJECT_ID \
               --gcpTempLocation=gs://STORAGE_BUCKET/tmp \
               --dataflowKmsKey=KMS_KEY"
  -Pdataflow-runner

Python

Cloud Dataflow does not support creating default Cloud Storage paths for temporary files when using a Cloud KMS key. You must specify temp_location.

python -m apache_beam.examples.wordcount \
  --input gs://dataflow-samples/shakespeare/kinglear.txt \
  --output gs://STORAGE_BUCKET/counts \
  --runner DataflowRunner \
  --project PROJECT_ID \
  --temp_location gs://STORAGE_BUCKET/tmp/ \
  --dataflow_kms_key=KMS_KEY

Cloud Console

1. Open the Cloud Dataflow monitoring UI.
   Go to the Cloud Dataflow Web UI
2. Select Create job from template.
3. In the Encryption section, select Customer-managed key.

Note: The drop-down menu Select a customer-managed key only shows keys with the regional scope global or the region you selected in the Regional endpoint drop-down menu.

The first time you attempt to run a job with a particular Cloud KMS key, your Compute Engine service account and/or Cloud Dataflow service account might not have been granted the permissions to encrypt and decrypt using that key. In this case, a warning message appears to prompt you to grant the permission to your service account.

Encryption of pipeline state artifacts

Data that a Cloud Dataflow pipeline reads from user-specified data sources is encrypted, except for the data keys that the user specified for key-based transforms.

Data keys used in key-based operations, such as Windowing, Grouping, and Joining, are not protected by CMEK encryption. If these keys contain personally identifiable information (PII), you need to hash or otherwise transform the keys before they enter the Cloud Dataflow pipeline. The values of the key-value pairs are in-scope for CMEK encryption.

Job metadata is not encrypted with Cloud KMS keys. Job metadata includes the following:

• User-supplied data, such as Job Names, Job Parameter values, and Pipeline Graph
• System-generated data, such as Job IDs and IP addresses of workers

Encryption of pipeline state locations

The following storage locations are protected with Cloud KMS keys:

• Persistent Disk attached to Cloud Dataflow workers and used for Persistent Disk-based shuffle and streaming state storage.
• Cloud Storage buckets that store temporary export or import data. Cloud Dataflow only supports default keys set by the user on the bucket level.
• Cloud Storage buckets used to store binary files containing pipeline code. Cloud Dataflow only supports default keys set by the user on the bucket level.

Currently, Cloud Dataflow Shuffle and Streaming Engine state cannot be protected by a CMEK and are encrypted by a Google-managed key. If you want all of your pipeline state to be protected by CMEKs, do not use these optional features.

Verifying Cloud KMS key usage

You can verify if your pipeline uses a Cloud KMS key using the GCP Console or the gcloud command-line tool.

Console

1. Open the Cloud Dataflow monitoring UI.
   Go to the Cloud Dataflow Web UI
2. Select your Cloud Dataflow job to view job details.
3. In the Job Summary section, the key type is listed in the Encryption type field.

   

CLI

Run the describe command using the gcloud tool:

gcloud dataflow jobs describe JOB_ID

Search for the line that contains serviceKmsKeyName. This information shows that a Cloud KMS key was used for Cloud Dataflow pipeline state encryption.

You can verify Cloud KMS key usage for encrypting sources and sinks by using the Cloud Storage and BigQuery GCP Console pages and tools or through viewing your Cloud KMS audit logs.

Removing Cloud Dataflow's access to the Cloud KMS key

You can remove Cloud Dataflow's access to the Cloud KMS key by using the following steps:

1. Revoke Cloud KMS CryptoKey Encrypter/Decrypter role to the Cloud Dataflow service account using the GCP Console or programmatically.
2. Revoke Cloud KMS CryptoKey Encrypter/Decrypter role to the Compute Engine service account using the GCP Console or programmatically.
3. Optionally, you can also destroy the key version material to further prevent Cloud Dataflow and other services from accessing the pipeline state.

Although you can destroy the key version material, you cannot delete keys and key rings. Key rings and keys do not have billable costs or quota limitations, so their continued existence does not impact costs or production limits.

Cloud Dataflow jobs periodically validate whether the Cloud Dataflow service account can successfully use the given Cloud KMS key. If an encrypt or decrypt request fails, the Cloud Dataflow service halts all data ingestion and processing as soon as possible and immediately begins cleaning up the GCP resources attached to your job.

Using GCP sources and sinks that are protected with Cloud KMS keys

Cloud Dataflow can access sources and sinks that are protected by Cloud KMS keys without you having to specify the Cloud KMS key of those sources and sinks, as long as you are not creating new objects. If your Cloud Dataflow pipeline might create new objects in a sink, you must define pipeline parameters that specify the Cloud KMS keys for that sink and pass this Cloud KMS key to appropriate I/O connector methods.

For Cloud Dataflow pipeline sources and sinks that do not support CMEK managed by Cloud KMS, such as Confluent Kafka hosted on GCP or Amazon Simple Storage Service (S3), the Cloud Dataflow CMEK settings are irrelevant.

Cloud KMS key permissions

When accessing services that are protected with Cloud KMS keys, verify that you have assigned the Cloud KMS CryptoKey Encrypter/Decrypter role to that service. The accounts are of the following form:

• Cloud Storage: service-{project_number}@gs-project-accounts.iam.gserviceaccount.com
• BigQuery: bq-{project_number}@bigquery-encryption.iam.gserviceaccount.com
• Cloud Pub/Sub: service-{project_number}@gcp-sa-pubsub.iam.gserviceaccount.com

Cloud Storage

If you want to protect the temporary and staging buckets that you specified with the TempLocation/temp_location and stagingLocation/staging_location pipeline parameters, see setting up CMEK-protected Cloud Storage buckets.

BigQuery

Java

Use the with_kms_key() method on return values from BigQueryIO.readTableRows(), BigQueryIO.read(), BigQueryIO.writeTableRows(), and BigQueryIO.write().

You can find an example in the Apache Beam GitHub repository.

Python

Use the kms_key argument in BigQuerySource and BigQuerySink.

You can find an example on the Apache Beam GitHub repository.

Cloud Pub/Sub

Cloud Dataflow handles access to CMEK-protected topics by using your topic CMEK configuration.

To read from and write to CMEK-protected Cloud Pub/Sub topics, see Cloud Pub/Sub instructions for using CMEK.

Pricing

You can use Cloud KMS encryption keys with Cloud Dataflow in all Cloud Dataflow regional endpoints where Cloud KMS is available.

This integration does not incur additional costs beyond the key operations, which are billed to your GCP project. Each time the Cloud Dataflow service account uses your Cloud KMS key, the operation is billed at the rate of Cloud KMS key operations.

For more information, see Cloud KMS pricing details.