Using customer-managed encryption keys

This page describes how to use Customer Managed Encryption Keys (CMEK) to protect Cloud Composer environments. Enabling CMEK will allow you to provide your own encryption keys to encrypt/decrypt environment data.

Regions where CMEK is not supported

Cloud Composer does not support CMEK in the following regions:

  • Seoul (asia-northeast3)
  • Salt Lake City (us-west3)
  • Las Vegas (us-west4)

Prerequisites

  1. Enable the Artifact Registry API.

        gcloud services enable artifactregistry.googleapis.com
      

  2. If you want your environment to run inside a VPC Service Controls perimeter, you must add the Cloud Key Management Service and Artifact Registry APIs to the perimeter.

Configuring CMEK support for Cloud Composer

Console

Use the following steps to configure CMEK encryption in the Cloud Console during environment creation:

  1. In the Cloud Console, go to the Create environment page. Open the Create environment page

  2. Expand the Networking, Airflow config overrides, and additional features section.

  3. Below Data encryption select Customer-managed key.

  4. Under Select a customer-managed key, select your key from the dropdown menu.

  5. If additional setup is required, a message will appear to inform you. You will then have the option of opening a wizard to guide you through the process.

After the environment has been created, you can verify its encryption configuration:

  1. Go to the Environment list page. Open the Environment list page

  2. On the top right of the page, below the "refresh" button, click the button for Column display options.

  3. From the Column display options dropdown menu, select Data encryption.

  4. A new column should now be visible, showing the data encryption for each environment as either "Google-managed key" or "Customer-managed key".

  5. To verify that your customer-managed key is available, select your new environment from the list to reach the Environment details page. Select the Environment configuration tab and look for the Data encryption row.

gcloud

Open the Cloud Console and use the commands below to configure CMEK encryption for Cloud Composer:

  1. Set project variables.

    export project=PROJECT_ID
    export location=LOCATION
    export keyRing=KEY_RING
    export keyName=KEY_NAME
    export keyProject=${project} # Change if you are using a key from another project.
    export projectNumber=$(gcloud projects describe ${project} | grep projectNumber | cut -f2 -d" " | sed "s/'//g" )
    
  2. Create a CMEK key in KMS (if one is not already available).

    gcloud config set project ${project}
    gcloud kms keyrings create ${keyRing} --location=`${location} --project ${keyProject}`
    gcloud kms keys create ${keyName} --location=${location} \
    --keyring=${keyRing} --purpose=encryption `--project ${keyProject}`
    

    You must create a CMEK key in the same region where your environments will be located.

  3. Grant permissions to the Composer Service Agent service account.

    gcloud kms keys add-iam-policy-binding ${keyName} \
      --location ${location} \
      --keyring ${keyRing} \
      --member=serviceAccount:$(gcloud beta services identity create \
      --service=composer.googleapis.com 2>&1 | awk '{print $4}') \
      --role=roles/cloudkms.cryptoKeyEncrypterDecrypter \
      --project ${keyProject}
    
  4. Grant permissions to the Artifact Registry service account.

    gcloud kms keys add-iam-policy-binding ${keyName} \
      --location ${location} \
      --keyring ${keyRing} \
      --member=serviceAccount:$(gcloud beta services \
    identity create --service=artifactregistry.googleapis.com 2>&1 | awk \
    '{print $4}') --role=roles/cloudkms.cryptoKeyEncrypterDecrypter
    
  5. Grant permissions to the GKE service account.

    gcloud kms keys add-iam-policy-binding ${keyName} \
      --location ${location} --keyring ${keyRing} \
      --member=serviceAccount:$(gcloud beta services identity create \
      --service=container.googleapis.com 2>&1 | awk '{print $4}') \
      --role=roles/cloudkms.cryptoKeyEncrypterDecrypter
    
  6. Grant permissions to the Pub/Sub service account.

    gcloud kms keys add-iam-policy-binding ${keyName} \
      --location ${location} \
      --keyring ${keyRing} \
      --member=serviceAccount:$(gcloud beta services identity create \
      --service=pubsub.googleapis.com 2>&1 | awk '{print $4}') \
      --role=roles/cloudkms.cryptoKeyEncrypterDecrypter
    
  7. Grant permissions to the Cloud Storage service account.

    gsutil kms authorize -k projects/${keyProject}/locations/${location}/keyRings/${keyRing}/cryptoKeys/${keyName}
    
  8. Grant permissions to the Compute Engine service account.

    gcloud kms keys add-iam-policy-binding ${keyName} 
    --location ${location}
    --keyring ${keyRing}
    --member=serviceAccount:service-${projectNumber}@compute-system.iam.gserviceaccount.com
    --role=roles/cloudkms.cryptoKeyEncrypterDecrypter

  9. Create a Cloud Composer environment.

    gcloud composer environments create ${envname} \
      --location=${location} \
      --kms-key projects/${keyProject}/locations/${location}/keyRings/${keyRing}/cryptoKeys/${keyName}
    

CMEK for Cloud Composer logs

If you expect your logs can contain sensitive data, you may also choose to redirect Cloud Composer logs the CMEK-encrypted Cloud Storage bucket via the Logs Router. This will prevent your logs from being sent to Monitoring. If you need support from Google Cloud Support, you will need to grant Google support engineers access to the Cloud Composer logs stored in Cloud Storage.

gcloud

  1. Create a new Cloud Storage bucket (e.g. composer-logs-${location}-${envname}).

    gsutil mb -l ${location} gs://${bucket_name}
    
  2. Encrypt it with your CMEK key.

    gsutil kms encryption -k projects/${project}/locations/${location}/keyRings/${keyRing}/cryptoKeys/${keyName} gs://${bucket_name}
    
  3. Create a new log sink.

    gcloud logging sinks create composer-log-sink-${envname} storage.googleapis.com/${bucket_name}
      --log-filter "resource.type=cloud_composer_environment AND resource.labels.environment_name=${envname} AND resource.labels.location=${location}"
    
  4. Grant the appropriate role to the service account for this bucket (shown in the result of the previous command).

    gcloud projects add-iam-policy-binding ${project} --member="serviceAccount:${serviceAccountNumber}@gcp-sa-logging.iam.gserviceaccount.com" --role='roles/storage.objectCreator' --condition=None
    
  5. Exclude the logs for your new environment from Monitoring.

    gcloud beta logging sinks update _Default --add-exclusion name=${envname}-exclusion,filter="resource.type=cloud_composer_environment AND resource.labels.environment_name=${envname} AND resource.labels.location=${location}"
    
  6. Add organization-level CMEK encryption to the Logs Router.

    gcloud beta logging cmek-settings describe --organization=[ORGANIZATION_ID]
    gcloud kms keys add-iam-policy-binding \
      --project=[KMS_PROJECT_ID] \
      --member [SERVICE_ACCOUNT_ID]@gcp-sa-logging.iam.gserviceaccount.com \
      --role roles/cloudkms.cryptoKeyEncrypterDecrypter \
      --location=[KMS_KEY_LOCATION] \
      --keyring=[KMS_KEY_RING] \
      [KMS_KEY]
    gcloud beta logging cmek-settings update \
      --organization=[ORGANIZATION_ID] --kms-key-name=[KMS_KEY_NAME]
    

Cloud Composer data protected with Google-provided encryption keys

Cloud Monitoring does not support CMEK encryption, which means that the names of your Cloud Composer environments and DAGs will be stored in the Monitoring database in encrypted form using Google encryption keys.

The Cloud Composer backend database also stores the environment metadata in encrypted form using Google encryption keys.