Using Cloud EKM with VPC

This topic is about using Cloud External Key Manager (Cloud EKM) to create and manage external keys accessed solely via a Virtual Private Cloud (VPC) network.

Limitations

Currently, accessing your Cloud EKM keys via a VPC is available in Cloud KMS regional locations.

Terminology

• EKM connection

  The Cloud KMS resource used to configure a connection to your external key manager. In the Google Cloud console, this is referred to as an EKM via VPC connection.

• VPC project

  The project that holds the VPC resource used to connect to your external key manager.

• Key projects

  The projects that hold EKM connection resources and Cloud EKM keys in Cloud KMS. A key project can be the same as a VPC project, but it is not required.

Before you begin

After you complete the steps below, you can begin using Cloud EKM keys to protect your data.

Create a new project

1. In the Google Cloud console, go to the Manage Resources page.

   Go to the Manage Resources page

2. Create a new Google Cloud project or select an existing project.

   Important: The name you use must be between 4 and 30 characters. When you type the name, the form will suggest a project ID, which you can edit. The project ID you use must be between 6 and 30 characters, with a lowercase letter as the first character. You can use a dash, lowercase letter, or digit for the remaining characters, but the last character cannot be a dash. You should be aware that some resource identifiers (such as project IDs) might be retained beyond the life of your project. For this reason, avoid storing sensitive information in resource identifiers.
   ...see naming guidelines

3. Make sure that billing is enabled for your Cloud project. Learn how to check if billing is enabled on a project.

You can learn more about Cloud EKM pricing.

Enable Cloud KMS

1. Enable the Cloud Key Management Service API for the project.

   Enable the Cloud Key Management Service API

2. Make a note of your project's Cloud EKM service account. In the following example, replace PROJECT_NUMBER with your Google Cloud project's project number. This information is also visible each time you use the Google Cloud console to create a Cloud EKM key.

   [e-mail]
   

Ensure gcloud CLI is up to date

If you're going to use the Google Cloud CLI, ensure that it's up-to-date with the following command:

gcloud

gcloud components update

Prepare a VPC network

There are two options when setting up a VPC network:

• Auto mode network
• Custom mode network

By default, new projects contain an auto mode network that is pre-populated with firewall rules. If the VPC network will not be used for production purposes, the default auto mode network is the quickest way to get started.

If your external key manager is running on-premises and you will connect to it via hybrid connectivity, you should use a custom mode network because it provides control over the subnet IP address ranges.

Follow these steps for setting up your VPC:

1. Enable Private Google Access

   The external key manager must verify the OIDC token contained in each request. To verify the token, it needs to retrieve the OAuth2 public key from the www.googleapis.com domain name. If the external key manager runs in Google Cloud and does not have access via the internet (e.g. a VM without an external IP or blocked by a firewall), follow the instructions for configuring private Google access.

2. Firewall configuration for IP range 35.199.192.0/19

   Requests from Cloud EKM will come from this range. Create both ingress and egress allow firewall rules for TCP for the port on which the external key manager is listening.

Set up hybrid connectivity

If the external key manager is running on-premise, use a hybrid connectivity solution to connect the VPC with your on-premise network. Once you have set up connectivity, follow these additional steps:

1. Enable Private Google Access

   The external key manager must verify the OIDC token contained in each request. To verify the token, it needs to retrieve the OAuth2 public key from the www.googleapis.com domain name. If the external key manager runs on-premise and does not have access via the internet, follow the instructions for configuring private Google access for on-premises hosts.

2. Firewall configuration for IP range 35.199.192.0/19

   Requests from Cloud EKM will come from this range. Configure your on-premise network firewall or similar equipment to permit TCP traffic on the port on which the external key manager is listening.

3. Ensure that your VPC has a return route to IP range 35.199.192.0/19

   Your on-premises network must have a route for the 35.199.192.0/19 destination. For information about how to meet this requirement, see return route strategies for on-premise targets.

Return route strategies for on-premise targets

• For Cloud VPN tunnels that use static routing, manually create a route in your on-premises network whose destination is 35.199.192.0/19 and whose next hop is the Cloud VPN tunnel. For Cloud VPN tunnels that use policy-based routing, configure the Cloud VPN's local traffic selector and the on-premises VPN gateway's remote traffic selector to include 35.199.192.0/19.

• For Cloud VPN tunnels that use dynamic routing or for Cloud Interconnect, configure a custom route advertisement for 35.199.192.0/19 on the BGP session of the Cloud Router that manages the tunnel or VLAN attachment.

Set up your external key manager

Follow the instructions from your EKM provider to set up your EKM.

Create a Service Directory service endpoint

Create and configure a Service Directory service endpoint in your VPC project that points to the private IP address and port of your your external key manager. If you are using a load balancer in front of multiple EKM replicas, use the load balancer's IP address and port. Ensure the network field of your Service Directory service endpoint is populated.

Authorize Cloud EKM to access your VPC

For each key project, you must authorize Cloud EKM to access your VPC on that project's behalf, even if the key project and the VPC project are the same. By authorizing access, keys in your key project can use the VPC in your VPC project.

1. Ensure a Cloud EKM service account exists for the project.

   gcloud

   gcloud beta services identity create \
     --service=cloudkms.googleapis.com \
     --project=KEY_PROJECT_ID
   

2. Grant the servicedirectory.viewer and servicedirectory.pscAuthorizedService in your VPC project to [e-mail] For help with getting your project ID and number, see Creating and managing projects.

   gcloud

   gcloud projects add-iam-policy-binding VPC_PROJECT_ID \
     --member='serviceAccount:[e-mail]' \
     --role='roles/servicedirectory.viewer'
   
   gcloud projects add-iam-policy-binding VPC_PROJECT_ID \
     --member='serviceAccount:[e-mail]' \
     --role='roles/servicedirectory.pscAuthorizedService'
   

Create an external key

You perform these steps on the external key management partner system. The exact steps vary by external key management partner. You can review the list of supported Cloud EKM partners.

1. If necessary, request access from your external key management partner to participate.

2. Create a symmetric or asymmetric key in the external key management partner system or select an existing key.

   Create the key in a region near the Google Cloud region you plan to use for Cloud EKM keys. This helps reduce the network latency between your Google Cloud project and the external key management partner. Otherwise, you may experience an increased number of failed operations. Review Cloud EKM and regions for more information.

3. Make a note of the external key's URI or key path. You need this information to create a Cloud EKM key.

Connect your EKM to Cloud EKM

To connect your external key manager to Cloud EKM, create an EKM connection in your key project.

Note: All Cloud KMS keys associated with an EKM connection will need to reside in the same Google Cloud location as the connection.

Note: EKM connections with the same name can exist in different locations, so you must always specify the location.

Console

1. Go to the Key Management page in the Google Cloud console.

   Go to the Key Management page

2. Click KMS infrastructure in the upper right corner.

3. Click Create connection next to EKM via VPC connections.

4. In the Name field, enter the name for your connection.

5. From the Location dropdown, select a location for the EKM connection. Note that any Cloud KMS keys associated with this connection must be in the same region as the connection.

6. In the Service Directory field, enter the value of the Service Directory service created in the Create a Service Directory service endpoint section. Note that the service directory service must be in the same region as the connection.

7. In the Hostname field, add the host name for your external key manager.

8. In Certificates, click Add certificate to upload one or more X.509 server certificates for your external key manager. Certificates must be in DER format.

9. Click Create.

gcloud CLI

To use Cloud KMS on the command line, first Install or upgrade to the latest version of Google Cloud CLI.

gcloud beta kms ekm-connections create EKM_CONNECTION \
    --location LOCATION \
    --service-directory-service SERVICE_DIRECTORY_SERVICE \
    --hostname HOSTNAME\
    --server-certificates-files SERVER_CERTIFICATE_FILES

Replace EKM_CONNECTION with a name for the connection. Replace LOCATION with the Cloud KMS location for the connection. Note that any Cloud KMS keys associated with this connection must be in the same region as the connection. Replace SERVICE_DIRECTORY_SERVICE with the resource ID of the Service Directory service for your connection. Replace HOSTNAME with the hostname of your external key manager. Replace SERVER_CERTIFICATE_FILES with a comma-separated list of files containing X.509 server certificates for your external key manager. Certificates must be in DER format.

For information on all flags and possible values, run the command with the --help flag.

API

These examples use curl as an HTTP client to demonstrate using the API. For more information about access control, see Accessing the Cloud KMS API.

curl "https://cloudkms.googleapis.com/v1/projects/PROJECT_ID/locations/LOCATION_ID/ekmConnections" \
    --request "POST" \
    --header "authorization: Bearer TOKEN" \
    --header "content-type: application/json" \
    --header "x-goog-user-project: PROJECT_ID" \
    --data "{\"name\": {\"EKM_CONNECTION_NAME\": {}}}"

See the EkmConnection.create API documentation for more information.

Certificate status

Once you've uploaded a certificate for your EKM connection, you can check the overall certificate status of the EKM connection as well as the status of each certificate from the KMS infrastructure page.

EKM connections have an overall status in the Certificate status column of each connection. If an EKM connection has a status other than Active, we recommend updating the certificate(s) on your EKM connection.

Both EKM connections and individual certificates can have the following status:

• Active: The certificate is valid and is not nearing its expiration date.
• Expiring within 30 days: The certificate is valid, but has an expiration date in the next 30 days.
• Expired: The certificate is expired and is no longer valid. We recommend updating any expired certificates.
• Not valid yet: The certificate is not active. This can happen if the certificate's start date is in the future.

If your certificate is no longer valid, update your EKM connection in the Google Cloud console.

Console

1. Go to the KMS infrastructure page in the Google Cloud console.
   Go to the KMS infrastructure page
   

2. Click the name of the EKM via VPC connection with the certificate that needs to be updated.

3. Click Edit connection.

4. Click Add certificate to upload one or more X.509 server certificates for your external key manager. Certificates must be in DER format.

5. Remove the expired certificates. Hover over the expired certificate and select the Delete icon on the right.

6. Click Update connection to update the EKM via VPC connection.

Create a Cloud EKM key

You can now use the EKM connection to create Cloud KMS keys and link them to the keys in your external key manager using the Google Cloud console or the Google Cloud CLI.

Create a symmetric encryption key

Note: The Google Cloud location of the EKM connection created above must match the Google Cloud location used here when creating a key.

Console

1. Go to the Key Management page in the Google Cloud console.

   Go to the Key Management page

2. Click the name of the key ring for which you will create a key.

3. Click Create key.

4. In What type of key do you want to create?, choose Externally managed key.

5. In the Key name field, enter the name for your key.

6. In Externally managed key connection type, choose via VPC.

7. Click the EKM via VPC connection dropdown and select the EKM connection you wish to use for this key.

   Note, if you do not have the EkmConnection.list permission, you will need to manually enter the connection resource name.

8. In the Key path field, enter the path to your external key.

9. Click Create.

gcloud CLI

To use Cloud KMS on the command line, first Install or upgrade to the latest version of Google Cloud CLI.

gcloud kms keys create KEY \
    --keyring KEY_RING \
    --location LOCATION \
    --purpose "encryption" \
    --default-algorithm "external-symmetric-encryption" \
    --protection-level "external-vpc" \
    --skip-initial-version-creation \
    --crypto-key-backend EKM_CONNECTION

Replace KEY with a name for the new key. Replace KEY_RING with the name of the existing key ring where the key will be located. Replace LOCATION with the Cloud KMS location for the key ring. Replace EKM_CONNECTION with the name of the existing EKM connection to use for creating this key.

For information on all flags and possible values, run the command with the --help flag.

Create an asymmetric signing key

Note: The Google Cloud location of the EKM connection created above must match the Google Cloud location used here when creating a key.

Console

1. Go to the Key Management page in the Google Cloud console.

   Go to the Key Management page
   

2. Click the name of the key ring for which you will create a key.

3. Click Create key.

4. For the type of key, select Externally managed key.

5. Enter a name for the key.

6. Set Purpose to Asymmetric sign.

7. In the Algorithm field, select a valid asymmetric signing algorithm that's supported by your external key management partner. After the asymmetric key is created, the algorithm cannot be modified.

8. In Externally managed key connection type, choose via VPC.

9. Click the EKM via VPC connection dropdown and select the EKM connection you wish to use for this key.

   If you do not have the EkmConnection.list permission, you will need to manually enter the connection resource name.

10. In the Key path field, enter the external key's URI.

11. Optionally, add labels for the key. You can learn more about labeling keys.

12. Click Create.

gcloud CLI

To use Cloud KMS on the command line, first Install or upgrade to the latest version of Google Cloud CLI.

gcloud kms keys create KEY \
    --keyring KEY_RING \
    --location LOCATION \
    --purpose asymmetric-signing \
    --protection-level "external-vpc" \
    --skip-initial-version-creation \
    --default-algorithm ec-sign-p256-sha256
    --crypto-key-backend EKM_CONNECTION

Replace KEY with a name for the new key. Replace KEY_RING with the name of the existing key ring where the key will be located. Replace LOCATION with the Cloud KMS location for the key ring. Replace EKM_CONNECTION with the name of the existing EKM connection to use for creating this key. Set the --default algorithm to a valid asymmetric signing algorithm that is supported by your external key management partner.

For information on all flags and possible values, run the command with the --help flag.

Create a key version

Create a key version pointing to a EKM key, specified by its ekm-connection-key-path. The ekm-connection-key-path is the path that's used to access the key on the external key manager. Refer to your EKM provider's documentation for more information on finding the key path.

This process is the same for both symmetric encryption keys and asymmetric signing keys.

Console

1. Go to the Key Management page in the Google Cloud console.

   Go to the Key Management page

2. Click the name of the key ring that contains the key for which you will create a new key version.

3. Click the key for which you will create a new key version.

4. In the header, click Rotate Key.

5. In the prompt, enter the key path for the new version.

6. Click Rotate Key to confirm.

gcloud CLI

To use Cloud KMS on the command line, first Install or upgrade to the latest version of Google Cloud CLI.

gcloud kms keys versions create \
    --key KEY \
    --keyring KEY_RING \
    --location LOCATION \
    --ekm-connection-key-path EKM_CONNECTION_KEY_PATH
    --primary

Replace KEY with a name for the key. Replace KEY_RING with the name of the existing key ring where the key will be located. Replace LOCATION with the Cloud KMS location for the key ring. Replace EKM_CONNECTION_KEY_PATH with the path to your key on your external key manager.

Key versions are numbered sequentially.

For information on all flags and possible values, run the command with the --help flag.

After the key version is successfully created, you can use it just as you would use any other Cloud KMS key version.

Rotate or create a new external key version

First, rotate the external key material on your external key manager. If that results in a new key path, you need to rotate or create a new Cloud EKM key version with the new path.

For symmetric encryption keys, rotate the Cloud EKM key and specify the new key path from your external key manager. For asymmetric keys, create a new key version and specify the new key path. Rotating or creating a new key version causes all newly-created data protected with that key to be encrypted with new key material. Data protected with previous key material is not re-encrypted. As a result, your external key manager must continue to make previous key material available to be used.

If the key material in the external key management partner system does not change, but the key path changes, you can update the key's external key path without rotating the key.

Console

1. Go to the Key Management page in the Google Cloud console.
   Go to the Key Management page
   

2. Select the key ring, then select the key.

3. Select Rotate key for symmetric keys, or Create version for asymmetric keys.

4. Enter the new key path, then select Rotate key for symmetric keys or Create version for asymmetric keys.

The new key version becomes the primary version.

gcloud

To update the key, create a new key version that contains the updated external key path and make this version the primary key version. Use the same name, key ring, and location you used when creating the key.

This process is the same for both symmetric and asymmetric keys.

gcloud kms keys versions \
  create \
  --key KEY_NAME \
  --keyring KEY_RING_NAME \
  --location LOCATION \
  --ekm-connection-key-path NEW_EKM_CONNECTION_KEY_PATH \
  --primary

Update the key path for a key version without rotation

It is also possible to update the key path without rotating the key. This is useful when you do not wish to change the key material, but the key path changes due to a configuration change to your external key manager. Updating the key path without rotation is only permitted if the new key path hosts the same key material as the original key path.

If the key material has been rotated in the external key management partner system, you must rotate the key instead.

Console

1. Go to the Key Management page in the Google Cloud console.
   Go to the Key Management page
   

2. Select the key ring, then select the key and version.

3. Click More then View key path.

4. Click Update key path.

5. Enter the new key path, then click Save.

gcloud

To update the key version's path, use the gcloud kms versions update command, specify the key version to update, and set the --ekm-connection-key-path flag to the new path.

gcloud kms keys versions update KEY_VERSION \
  --key KEY \
  --keyring KEY_RING \
  --location LOCATION \
  --ekm-connection-key-path ekm-connection-key-path

For example, the following command updates the path for version 2 of key example-key to be v0/example_key:

gcloud kms keys versions update 2 \
  --key example-key \
  --keyring example-keyring \
  --location us-west1 \
  --ekm-connection-key-path v0/example_key

Disable or destroy an external key

To temporarily disable the association between a Cloud EKM key and an external key, you can disable the Cloud EKM key or key version. Disabling all key versions is recommended. Disabling a key takes effect within three hours.

When you disable a key, you should also revoke access to the key. IAM operations are consistent within seconds. Also consider revoking the Google Cloud service account's access in the external key management partner system.

To permanently remove the association between a Cloud EKM key and an external key, you can schedule the Cloud EKM key version for destruction. After a 24-hour period, the key is destroyed. Destroying a key is permanent. After the key version is destroyed, you can no longer encrypt data or decrypt data that was encrypted with the Cloud EKM key version. You cannot recreate a Cloud EKM key version that has been destroyed, even if you use the same external key URI or key path. When destroying external key material, we recommend first destroying the key or key version in Google Cloud and then, only after 24 hours, destroying the key material in the external key manager.

Note: Disabling or destroying a key or key version in Cloud KMS does not modify the key in the external key management partner system.

Troubleshooting errors

In addition to the errors listed in the Cloud EKM error reference, EKMs accessed via VPC might experience additional errors.

Input errors

The following table describes errors caused by incorrect input and troubleshooting suggestions:

google.rpc.Status.message	violation[1].type(Error domain)	Troubleshooting
Permission denied when accessing the Service Directory. Ensure the Cloud EKM service account has access to the Service Directory resource in the VPC project.	SD_RESOURCE_PERMISSION_DENIED	Follow the steps in Authorize Cloud EKM to access your VPC to authorize Cloud EKM to access your VPC resource. Also, refer to the Service Directory troubleshooting guide.

External key management system errors

The following table describes EKM system errors and troubleshooting suggestions:

google.rpc.Status.message	violation[1].type(Error domain)	Troubleshooting
Unable to use the Service Directory entry provided for the external key manager. The data was incomplete or was not found in the Service Directory service.	SD_RESOURCE_MALFORMED	If you manage your own EKM: Ensure the network field of your Service Directory endpoint is populated. Ensure the IP address and Port are set correctly for your endpoint. If your EKM is managed by a separate provider: Contact your EKM provider to ensure the network Service Directory endpoints are correctly set.