Customer-managed encryption keys

Setup

This page discusses customer-managed encryption keys and how they are used in Cloud Storage. For other encryption options, see Data Encryption Options.

Overview

If you need more control over key operations than what the standard Cloud Storage encryption allows, you can use customer-managed encryption keys. These keys are created and managed using Cloud Key Management Service (Cloud KMS), and you store the keys as software keys, in an HSM cluster, or externally. You can use customer-managed encryption keys on individual objects, or configure your bucket to use a key by default on all new objects added to a bucket.

When using a customer-managed encryption key, an object is encrypted with the key by Cloud Storage at the time it's stored in a bucket, and the object is automatically decrypted by Cloud Storage when the object is served to requesters.

When is the key used?

When you apply a customer-managed encryption key to an object, Cloud Storage uses the key when encrypting:

  • The object's data.
  • The object's CRC32C checksum.
  • The object's MD5 hash.

Cloud Storage uses standard server-side keys to encrypt the remaining metadata for the object, including the object's name. Thus, if you have sufficient permission, you can perform actions such as reading most metadata, listing objects, and deleting objects even after you've disabled or destroyed the associated customer-managed encryption key.

Service agents

Each project has a special Cloud Storage service account called a service agent that performs encryption and decryption with customer-managed encryption keys. Once you give the service agent access to an encryption key, that service agent encrypts:

When adding or rewriting an object in Cloud Storage, if you have both a default key set on your bucket and a specific key included in your request, Cloud Storage uses the specific key to encrypt the object.

When a requester wants to read an object encrypted with a customer-managed encryption key, they simply access the object as they normally would. During such a request, the service agent automatically decrypts the requested object as long as:

  • The service agent still has permission to decrypt using the key.
  • You have not disabled or destroyed the key.

If one of these conditions is not met, the service agent does not decrypt the data, and the request fails.

Restrictions

The following restrictions apply when using customer-managed encryption keys:

  • You cannot encrypt an object with a customer-managed encryption key by updating the object's metadata. Include the key as part of a rewrite of the object instead.

    • gcloud storage uses the objects update command to set encryption keys on objects, but the command rewrites the object as part of the request.
  • You must create the Cloud KMS key ring in the same location as the data you intend to encrypt. For example, if your bucket is located in US-EAST1, any key ring used for encrypting objects in that bucket must also be created in US-EAST1.

    • For most dual-regions, you must create the Cloud KMS key ring in the associated multi-region. For example, if your bucket is located in the pair US-EAST1, US-WEST1, any key ring used for encrypting objects in that bucket must be created in the US multi-region.

    • For the ASIA1, EUR4, and NAM4 predefined dual-regions, you must create the key ring in the same predefined dual-region.

      For available Cloud KMS locations, see Cloud KMS locations.

  • Cloud KMS encryption and decryption rates are subject to a quota.

  • The CRC32C checksum and MD5 hash of objects encrypted with customer-managed encryption keys are not returned when listing objects with the JSON API.

    • When appropriate, some tools, such as gcloud storage, perform an additional metadata GET request on each object encrypted with a customer-managed encryption key in order to retrieve the CRC32C and MD5 information. These additional requests can make listing substantially slower than listing objects encrypted with standard Cloud Storage encryption.
  • Cloud Storage cannot use the decryption portion of asymmetric keys stored in Cloud KMS to automatically decrypt relevant objects in the same manner that customer-managed encryption keys do.

Relation to customer-supplied encryption keys

In addition to customer-managed encryption, Cloud Storage offers Customer-Supplied Encryption Keys as a way of controlling your data encryption. You can encrypt different objects in a single bucket with different encryption methods, but note that:

  • A single object can only be encrypted by one of these methods at a time.

  • If you have a default customer-managed key set for your bucket and specify a customer-supplied key in a request, Cloud Storage uses the customer-supplied key to encrypt the object.

Key management

This section discusses considerations when rotating keys, replacing keys, and disabling or destroying key versions.

Key rotation

Cloud KMS supports both automatic and manual key rotation to a new version. After rotating a key, Cloud Storage uses the new version for all operations that encrypt using the key, such as:

  • Object uploads when the destination bucket uses the key as its default encryption key.

  • Object upload, copy, and rewrite operations that specifically use the key in the operation.

Previous versions of the key are not disabled or destroyed, so Cloud Storage can still decrypt existing objects that were previously encrypted using those versions.

Key replacement

Use the following guidelines when replacing the key you use to encrypt Cloud Storage objects with a new key:

  1. Check your buckets to see which use the key as their default encryption key. For these buckets, replace the old key with a new key.

    This ensures that all objects written to the bucket use the new key going forward.

  2. Inspect your source code to understand which requests use the key in ongoing operations, such as setting bucket configurations and uploading, copying, or rewriting objects. Update these instances to use the new key.

  3. Check for objects, in all of your buckets, encrypted with the old key. Use the Rewrite Object method to re-encrypt each object with the new key.

  4. Disable all versions of the old key. After disabling old key versions, monitor client and service logs for operations that fail due to a version becoming unavailable.

Disabling or destroying a key version

  • When you disable or destroy a specific key version, you cannot decrypt any object that is currently encrypted with that key version.

    For example, you cannot download, copy, or rewrite the object, and attempting to do so results in an error.

    • If you disable a key version, you can re-enable it. Once re-enabled, you can access objects that were encrypted by that key version.

    • If you destroy a key version, downloads of objects encrypted with that version are never possible again.

    Before disabling or destroying a key version, you should identify all objects, in all buckets, that were encrypted using the specific key version. Once identified, use the Rewrite Object method to re-encrypt each object using a new key version, a whole new key, or server-side keys.

  • When you disable or destroy the primary version of a key, you cannot use the key for encryption until you have a new primary version. For example, without a primary version:

    • You cannot specify the key as part of an object upload, copy, or rewrite.

    • You cannot upload, copy, or rewrite objects to a bucket that has the key set as its default encryption key unless you specify a different, valid key as part of the operation.

    Once you have a primary version for your key, operations that use the key to encrypt objects succeed.

    Before disabling or destroying a key version that is the primary version of the key, you should first stop using it as the primary version. You can do so by either:

    • Replacing it with a new primary version, typically by performing a key rotation.
    • Removing instances where you use the key for encryption. When you do so, Cloud Storage uses server-side keys for encryption instead.

Key versions and locked objects

If a key version encrypts an object that is locked, either because the object is stored in a bucket with a locked retention policy or because the object has its own locked retention configuration, the key version can only be destroyed if the following conditions are met:

  • The encrypted object's retention expiration time must be in the past.
  • The encrypted object must not have any object holds placed on it.

Once all relevant objects have met these conditions, it's possible to destroy the key version, even without deleting the objects. If you do so, affected object data becomes permanently inaccessible.

What's next