Cloud HSM architecture

This content was last updated in November 2023 and represents the status quo as of the time that it was written. Google's security policies and systems may change going forward, as we continually improve protection for our customers.

To help you meet corporate and compliance regulations, Cloud HSM lets you generate your encryption keys and perform cryptographic operations in FIPS 140-2 Level 3 certified hardware security modules (HSM).

This paper describes Cloud HSM architecture, including how the hardware is managed and keys are attested and created.

Overview

Cryptographic operations include encrypting data at rest, protecting the private keys for Certificate Authority Service, and protecting data encryption keys so that they can be stored together with the encrypted data. Cloud HSM uses Marvell LiquidSecurity HSMs (models CNL3560-NFBE-2.0-G and CNL3560-NFBE-3.0-G) with the firmware versions 3.4 build 09. For more information about our certification, see Certificate #4399.

Cloud HSM is fully managed so that you can protect your workloads without the operational overhead of managing an HSM cluster. The service provides the following advantages:

  • Global availability
  • A consistent and unified API
  • Automatic scaling based on your use
  • Centralized management and regulatory compliance

Cloud HSM is available in every Google Cloud region around the globe, including multi-regions that span larger geographies. When you enable Cloud HSM, you can create and use HSM-backed keys to protect your data, including data that you store in other Google Cloud services, such as BigQuery, Cloud Storage, and Persistent Disk.

Because Cloud HSM and the HSM hardware are managed by Google, you don't need to manage the HSM-backed keys in production. When you use Cloud HSM, your data is strictly isolated from other tenants and services in Google Cloud. The Cloud HSM data plane API, which is part of the Cloud Key Management Service API, lets you manage HSM-backed keys programmatically.

Cloud HSM supports HSM-backed customer-managed encryption keys (CMEK) wherever CMEK keys are supported across Google Cloud. For example, you can encrypt data in Cloud Storage buckets or Cloud SQL tables using a Cloud HSM key that you manage.

Cloud HSM management

Within Cloud HSM, clusters of HSMs are maintained by Google site reliability engineers (SREs) and technicians who work in each Google Cloud data center location. Google handles physical security, logical security, infrastructure, capacity planning, geo-expansion, and data center disaster-recovery planning.

Abstraction of HSM hardware

Typically, applications communicate directly with HSMs using both PKCS#11 and a cluster management API. This communication requires that you maintain specialized code for workloads that use or manage HSM-backed keys.

Cloud HSM abstracts communication away from the HSM by proxying requests for HSM-backed keys through the Cloud Key Management Service API. The abstraction reduces the need for HSM-specific code. Cloud HSM inherits tight integration with Cloud KMS.

Tight integration with Cloud KMS provides substantial security benefits. The Cloud Key Management Service API significantly reduces the breadth of the HSM interface available, reducing risk in the case of a customer security breach. For example, an attacker would be unable to wipe entire HSMs. By default, attempts to destroy individual keys are mitigated through a default 24-hour safety period. You can set the constraints/cloudkms.minimumDestroyScheduledDuration organization policy to enforce a minimum length for the scheduled for destruction duration for new keys and the constraints/cloudkms.disableBeforeDestroy organization policy to delete key versions only when they are disabled. For more information, see Control key version destruction.

You can control access to HSM resources using Identity and Access Management (IAM). IAM configuration is less likely to suffer from misconfigurations and bugs than a custom HSM solution.

Cloud HSM architecture diagram.

Strict geographic separation, by design

In Cloud HSM, you can choose to make keys globally available or to enforce strict geographic restrictions on keys that require restrictions.

Often, HSMs are divided into partitions, so that a single physical device can operate as multiple logical devices. You might use partitions to reduce deployment costs in cases where you need to separate HSM administration and keys.

Each Cloud HSM regional location is associated with a separate wrapping key. The wrapping key is cloned onto a partition in each HSM in the location, but never leaves the HSM in the location. Cloning lets HSMs in the same region to serve the same set of customer keys, and ensures that HSMs outside the region cannot serve those keys.

Cloud HSM also creates multi-regions using wrapping keys. All customer keys for a multi-region are wrapped using a wrapping key present on a partition in all locations that constitute the multi-region. The service uses the same hardware for multi-regions, but provides the same strong isolation between regions and multi-regions that exists between different regions.

Diagram of Cloud HSM geography.

The regionalization scheme requires that wrapping keys are only replicated to appropriate partitions. Each configuration change must be approved by multiple members of the Cloud HSM team before it becomes active. Data center technicians can't access an HSM configuration, runtime, or storage in the field.

Centralized management

In a traditional data center that hosts HSMs, management of the HSMs and their resources is entirely separate from management of other cryptographic resources. Cloud HSM is tightly integrated into Google Cloud, allowing you to seamlessly manage your Cloud HSM resources. For example, you can manage the following:

  • You manage your HSM-backed resources alongside your other keys in Cloud KMS and externally managed keys in Cloud External Key Manager (Cloud EKM).
  • You manage access to HSM-backed resources within IAM.
  • Cost reporting for cryptographic operations using HSM-backed keys are reported in Cloud Billing.
  • You can use HSM-backed keys transparently in all Google Cloud services that support encrypting resources using CMEK. CMEK integrations require the CMEK key and the data it encrypts to be located in compatible geographic locations. Because of the strict geographic restriction of the Cloud HSM keys, all encryption and decryption of the CMEK data are also geographically restricted.
  • Administrative operations on HSM-backed resources are always logged at the API layer in Cloud Audit Logs. You can choose to enable data-access logging as well. For more information, see Cloud KMS audit logging information.
  • Google partners directly with the HSM manufacturer to keep the hardware and software on each HSM updated, and to find and fix issues in real time. In the event of a zero-day exploit on the HSM, Google can selectively disable affected code paths on affected HSM clusters until the exploit is fixed.

Developer and user experience

Because Google is responsible for HSM management, Cloud HSM offers significant benefits to developers and end users.

HSMs at Google scale

When you rely on hardware that exists on-premises or in data centers, the hardware can create a performance bottleneck or become a single point of failure. Cloud HSM is designed to be extremely resilient to unpredictable workloads and hardware failures. The Cloud HSM backend uses a pool of HSMs in each region to ensure high availability and scalability. This pool of HSMs lets Cloud HSM also provide high throughput. For more information, see Monitor and adjust Cloud KMS quotas.

All customer keys are stored wrapped with a regional wrapping key in the Cloud KMS database and can only be unwrapped by an HSM in the region as part of a cryptographic operation. This wrapping has the following benefits:

  • A key's durability is not tied to a specific HSM or subset of HSMs in a region.
  • Each Cloud HSM customer experiences the full scale and availability of the Cloud HSM clusters that serve their keys.
  • Cloud HSM can handle a much larger set of keys that can be stored on an HSM.
  • Adding or replacing an HSM is rapid and secure.

Unified API design

Cloud HSM and Cloud KMS share a common management and data plane API. The internal details of communicating with an HSM are abstracted from the caller.

Consequently, no code changes are required to update an existing application that uses software keys in Cloud KMS to support HSM-backed keys. Instead, you update the resource name of the key to use.

PKCS#11 support

You can use Cloud Key Management Service API to connect your existing applications to Cloud HSM to manage cryptographic keys. The PKCS#11 library lets you use HSM-backed keys to sign your binaries and serve TLS web sessions.

Security and regulatory compliance

Cloud HSM has obtained compliance with numerous regulations, including FedRAMP High, C5:2020, and OSPAR. In addition, Cloud HSM helps you enforce regulatory compliance for your workloads in the cloud.

Cryptographic key attestation

Each time you generate or import a Cloud HSM key, the HSM generates an attestation statement that is signed with a signing key that is associated with the partition. The statement contains information about your key's attributes. The signing key is backed by certificate chains rooted in both Google and the HSM manufacturer. You can download the attestation statement and certificates to verify the statement's authenticity and validate properties of the key and the HSM that generated or imported it.

The certificate chain lets you check the following:

  • The HSM hardware and firmware are genuine.
  • The HSM partition and HSM are managed by Google.
  • The HSM is in the FIPS mode of operation.

The content of the attestation statement lets you check the following:

  • The key is not extractable.
  • The key was generated for your CryptoKeyVersion.
  • The public key in an asymmetric key pair corresponds to an HSM-backed private key.
  • The key material of an imported symmetric key matches the value you wrapped.

Secure key import directly into HSMs

You can securely import existing keys into Cloud HSM to maintain a backup of your key material outside of Google Cloud, or to simplify migrating certain workloads to Google Cloud. The key-import process does not allow Google any direct access to the unwrapped key material. Cloud HSM provides you with an attestation statement for the HSM-generated wrapping key to validate that no access occurred.

Because key import potentially creates security and compliance risk by allowing users to bring keys from unknown sources, separate IAM roles allow fine-grained control over who can import keys into a project. Imported keys can be distinguished by the attestation statement that the HSM generates on import.

For more information, see Importing a key into Cloud Key Management Service.

Strict security procedures safeguard HSM hardware

As mandated by FIPS 140-2 level 3, HSM devices have built-in mechanisms to help protect against, and provide evidence of, physical tampering.

In addition to the assurances provided by the HSM hardware itself, the infrastructure for Cloud HSM is managed according to Google infrastructure security design overview.

Documented, auditable procedures protect the integrity of each HSM during provisioning, deployment, and in production:

  • All HSM configurations must be verified by multiple Cloud HSM SREs before the HSM can be deployed into a data center.
  • After an HSM is put into service, configuration change can only be initiated and verified by multiple Cloud HSM SREs.
  • An HSM can only receive firmware that is signed by the HSM manufacturer.
  • HSM hardware is not directly exposed to any network.
  • Servers that host HSM hardware are prevented from running unauthorized processes.

The duties for system operators are defined in standard operating procedures. System operators are prevented from accessing, using, or extracting customer key material while performing their duties.

Service and tenant isolation

The Cloud HSM architecture ensures that HSMs are protected from malicious or inadvertent interference from other services or tenants.

An HSM that is part of this architecture accepts requests only from Cloud HSM, and the Cloud HSM service accepts requests only from Cloud KMS. Cloud KMS enforces that callers have appropriate IAM permissions on the keys that they attempt to use. Unauthorized requests don't reach HSMs.

HSM-backed keys are also subject to quotas for cryptographic operations. These quotas protect your ability to run your workloads by helping to prevent inadvertent or malicious attempts to overload the service. The default quotas, 3,000 QPM for asymmetric cryptographic operations and 30,000 QPM for symmetric cryptographic operations, are based on observed usage patterns. The quotas are significantly below the service capacity and can be increased upon request.

Request flows

This section demonstrates how the architectural highlights above apply in practice by showing the steps for different types of requests. These flows emphasize the Cloud HSM portions. For more information about steps common to all keys, see the Cloud Key Management Service deep dive.

Creating keys

When you create an HSM-backed key, the Cloud Key Management Service API doesn't create the key material, but requests that the HSM create it.

An HSM can only create keys in locations that it supports. Each partition on an HSM contains a wrapping key corresponding to a Cloud KMS location. The wrapping key is shared among all partitions that support the Cloud KMS location. The key-creation process looks like the following:

  1. The Google Front End Service (GFE) routes the key creation request to a Cloud KMS server in the location that corresponds to the request.
  2. The Cloud Key Management Service API verifies the caller's identity, the caller's permission to create keys in the project, and that the caller has sufficient write request quota.
  3. The Cloud Key Management Service API forwards the request to Cloud HSM.
  4. Cloud HSM directly interfaces with the HSM. The HSM:
    1. Creates the key and wraps it with the location-specific wrapping key.
    2. Creates the attestation statement for the key and signs it with the partition signing key.
  5. After Cloud HSM returns the wrapped key and attestation to Cloud KMS, the Cloud Key Management Service API wraps the HSM-wrapped key according to the Cloud KMS key hierarchy, then writes it to the project.

This design ensures that the key can't be unwrapped or used outside of an HSM, can't be extracted from the HSM, and exists in its unwrapped state only within specified locations.

The following diagram shows the differences when creating Cloud HSM keys and software keys in Cloud KMS.

HSM key creation diagram.

Cryptographic operations

When you perform a cryptographic operation in Cloud KMS, you don't need to know whether you are using an HSM-backed or software key. When the Cloud Key Management Service API detects that an operation involves a HSM-backed key, it forwards the request to an HSM in the same location. The following are the steps for a cryptographic operation:

  1. The GFE routes the request to a Cloud KMS server in the appropriate location. The Cloud Key Management Service API verifies the caller's identity, the caller's permission to access the key and perform the operation, and the project's quota for cryptographic operations.
  2. The Cloud Key Management Service API retrieves the wrapped key from the datastore and decrypts one level of encryption using the Cloud KMS master key. The key is still wrapped with the HSM wrapping key for the KMS location.
  3. The Cloud Key Management Service API detects that the protection level is HSM and sends the partially unwrapped key, along with the inputs to the cryptographic operation, to Cloud HSM.
  4. Cloud HSM directly interfaces with the HSM. The HSM completes the following operations:
    1. Checks that the wrapped key and its attributes have not been modified.
    2. Unwraps the key and loads it into the HSM storage.
    3. Performs the cryptographic operation and returns the result.
  5. The Cloud Key Management Service API passes the result back to the caller.

Cryptographic operations using HSM-backed keys are performed entirely within an HSM in the configured location, and only the result is visible to the caller.

This diagram shows the difference between creating Cloud HSM keys and software keys in Cloud KMS.

HSM encryption operations diagram.

CMEK integrations

CMEK and Cloud HSM together allow you to protect your data in select Google Cloud services with HSM keys. Configuring a CMEK-enabled service to use Cloud HSM keys is as simple as choosing a key with an HSM protection level when following the service-specific instructions.

When a caller reads or writes data to a CMEK-enabled service, the caller doesn't need direct permission to use the key, and the caller doesn't need to know whether the key is stored in an HSM.

The flow for a CMEK operation is very similar to that for a normal cryptographic operation with the following exceptions:

  • The request from the CMEK-enabled service is initiated within Google's network, and doesn't need to traverse the GFE.
  • The Cloud Key Management Service API verifies that the service account for the CMEK-enabled service has proper permissions to use the key. The Cloud Key Management Service API doesn't validate permissions on the end user of the CMEK-enabled service.

Cloud HSM is Google Cloud's hardware key management service. It offers a number of distinct advantages to users looking to protect their data at rest with HSM keys. The service was designed with the principles of locked-down API access to the HSMs, effortless scale, and tight regionalization of the keys.

Cloud HSM offers CMEK- support for the most important services and presence of Cloud HSM in every Google Cloud region (including multi-regions and global). The service is designed to make it easy for you to protect your sensitive data, wherever it may be, with a key that's protected by FIPS 140-2 Level 3 devices.

What's next

To learn more, explore the following resources: