Using a Cloud HSM key for TLS offloading with NGINX

Stay organized with collections Save and categorize content based on your preferences.

This guide provides instructions for setting up NGINX to use a Cloud HSM key for TLS offloading on Debian 11 (Bullseye). These instructions may have differences based on your OS or Linux distribution.

Use cases

Using a Cloud HSM key with NGINX for TLS offloading helps address the following enterprise security needs:

  • You want your NGINX web server to offload TLS cryptographic operations to Cloud HSM.
  • You do not want to store your certificate’s private key on the Compute Engine instance’s local file system that is hosting your web application.
  • You need to meet regulatory requirements where public facing applications need their certificates to be protected by an HSM that has FIPS 140-2 Level 3 certification.
  • You want to use NGINX to create a reverse proxy with TLS termination to protect your web application.

Before you begin

Before continuing, complete the steps in Using a Cloud HSM key with OpenSSL.

Once OpenSSL setup is complete, ensure that a recent version of nginx is installed:

sudo apt-get update
sudo apt-get install libengine-pkcs11-openssl opensc nginx

Security configuration recommendations

Secure your instance that is hosting NGINX with the following recommendations:

  1. Follow the instructions for creating and enabling service accounts for instances to host NGINX.

    1. Assign the following roles:
      • roles/cloudkms.signer
      • roles/cloudkms.viewer
  2. Configure organization policies as follows to limit external IPs and creation of service account keys.

    • constraints/compute.vmExternalIpAccess
    • constraints/iam.disableServiceAccountKeyCreation
  3. Create a custom subnet that enables private Google access.

  4. Configure firewall rules.

  5. Create a Linux VM and configure it as follows:

    • Select the correct service account that you created earlier.
    • Select the network that you created earlier.
      • Add appropriate labels for any firewall rules.
      • Ensure the subnet has the "external IP" field set to none.
  6. Grant your identity the IAP-Secured Tunnel User (roles/iap.tunnelResourceAccessor) role on the instance.

Configuration of Cloud KMS-hosted signing key

The next sections detail steps needed to create and configure a Cloud KMS-hosted signing key.

Creating a Cloud KMS-hosted signing key

Create a Cloud KMS EC-P256-SHA256 signing key in your Google Cloud project, in the key ring that you previously configured for OpenSSL:

gcloud kms keys create NGINX_KEY \
  --keyring "KEYRING" --project "PROJECT" \
  --location "LOCATION" --purpose "asymmetric-signing" \
  --default-algorithm "ec-sign-p256-sha256" --protection-level "hsm"

SSH into your VM using IAP

SSH into your VM using IAP with the following command:

gcloud compute ssh INSTANCE \
  --zone ZONE --tunnel-through-iap

If you run into an issue, confirm you used the --tunnel-through-iap flag. Also, confirm that you have the IAP-Secured Tunnel User (roles/iap.tunnelResourceAccessor) role on the instance for the identity authenticated with gcloud CLI.

Creating a certificate with OpenSSL

For a production environment, create a certificate signing request (CSR). Learn more by reading the example to generate a CSR. Provide the CSR to your certificate authority (CA) so that they can create a certificate for you. Use the certificate provided by your CA in the subsequent sections.

For example purposes, you can generate a self-signed certificate with the Cloud KMS-hosted signing key. To do so, OpenSSL lets you use PKCS #11 URIs instead of a regular path, identifying the key by its label (for Cloud KMS keys, that is equivalent to the CryptoKey name).

openssl req -new -x509 -days 3650 -subj '/CN=test/' -sha256 -engine pkcs11 \
  -keyform engine -key pkcs11:object=NGINX_KEY > CA_CERT

If the command fails, PKCS11_MODULE_PATH may have been set incorrectly, or you might not have the correct permissions to use the Cloud KMS signing key. You should now have a certificate that looks like this:

-----BEGIN CERTIFICATE-----
...
...
...
-----END CERTIFICATE-----

Install your certificate for NGINX

Run the following commands to create a location to place your public certificate:

sudo mkdir /etc/ssl/nginx
sudo mv CA_CERT /etc/ssl/nginx

Configure your environment to use the PKCS #11 library

The next sections detail steps needed to prepare and test your environment.

Prepare library configurations for NGINX

Allow NGINX to log its PKCS #11 engine operations with the library with the following:

sudo mkdir /var/log/kmsp11
sudo chown www-data /var/log/kmsp11

Create an empty library configuration file with the appropriate permissions for NGINX.

sudo touch /etc/nginx/pkcs11-config.yaml
sudo chmod 744 /etc/nginx/pkcs11-config.yaml

Edit the empty config file and add the needed configuration as shown in the following snippet:

# cat /etc/nginx/pkcs11-config.yaml
---
tokens:
  - key_ring: "projects/PROJECT/locations/LOCATION/keyRings/KEYRING"
log_directory: "/var/log/kmsp11"

Test your OpenSSL configuration

Run the following command:

openssl engine -tt -c -v pkcs11

You should see output similar to the following:

(pkcs11) pkcs11 engine
 [RSA, rsaEncryption, id-ecPublicKey]
     [ available ]
     SO_PATH, MODULE_PATH, PIN, VERBOSE, QUIET, INIT_ARGS, FORCE_LOGIN

Configure NGINX to use Cloud HSM

Allow TLS offloading by editing a few NGINX files. First, edit the /etc/nginx/nginx.conf file in two places to add a few directives to configure NGINX to use PKCS #11.

After the event block and before the http block, add the following directives:

ssl_engine pkcs11;
env KMS_PKCS11_CONFIG=/etc/nginx/pkcs11-config.yaml;

In the same /etc/nginx/nginx.conf file, configure SSL directives to use your certificate and its private key in Cloud HSM. In the http block add the following attributes:

ssl_certificate "/etc/ssl/nginx/CA_CERT";
ssl_certificate_key "engine:pkcs11:pkcs11:object=NGINX_KEY";
ssl_protocols TLSv1.2 TLSv1.3; # Consider changing the default to only TLS1.2 or newer

# Consider defining the `ssl_ciphers` to use ciphers approved by your security teams and handle
# appropriate client compatibility requirements.

Your /etc/nginx/nginx.conf file should look like the following:

user www-data;
worker_processes auto;
pid /run/nginx.pid;
include /etc/nginx/modules-enabled/*.conf;

events {
        worker_connections 768;
        # multi_accept on;
}

ssl_engine pkcs11;
env KMS_PKCS11_CONFIG=/etc/nginx/pkcs11-config.yaml;

http {

        #...
        #...

        # SSL configuration
        ssl_certificate "/etc/ssl/nginx/CA_CERT";
        ssl_certificate_key "engine:pkcs11:pkcs11:object=NGINX_KEY";
        ssl_protocols TLSv1.2 TLSv1.3; # Dropping SSLv3, ref: POODLE
        # ssl_ciphers YOUR_CIPHERS
        ssl_prefer_server_ciphers on;

        #...
        #...

}

Configure NGINX to listen for TLS traffic

Edit the /etc/nginx/sites-enabled/default file to listen for TLS traffic. Uncomment the SSL configuration in the server block. The resulting change should look like the following example:


server {
        listen 80 default_server;
        listen [::]:80 default_server;

        # SSL configuration
        listen 443 ssl default_server;
        listen [::]:443 ssl default_server;

        # ...
        # ...
}

Provide environment variables to the NGINX service

Run the following command:

sudo systemctl edit nginx.service

In the resulting editor, add the following lines and replace the LIBPATH with the value for the location where you installed libkmsp11.so:

[Service]
Environment="GRPC_ENABLE_FORK_SUPPORT=1"
Environment="KMS_PKCS11_CONFIG=/etc/nginx/pkcs11-config.yaml"
Environment="PKCS11_MODULE_PATH=LIBPATH/libkmsp11-1.0-linux-amd64/libkmsp11.so"

After you configure these values, you will need to run the following command to make them available:

sudo systemctl daemon-reload

Restart NGINX with TLS Offloading

Run the following command so that NGINX restarts and uses the updated configuration:

sudo systemctl start nginx

Test NGINX uses TLS offloading to your Cloud HSM

Use the openssl s_client to test connection to your NGINX server by running the following command:

openssl s_client -connect localhost:443

The client should complete SSL handshake and pause. The client is awaiting input from you as shown following:

# completes SSL handshake
# ...
# ...
# ...
    Verify return code: 18 (self signed certificate)
# ...
    Max Early Data: 0
---
read R BLOCK

# When the client pauses, it’s waiting for instructions.
# Have the client get the index.html file in the root path (“/”), by typing the following:

GET /

# Press enter.
# You should now see the default NGINX index.html file.

Your audit logs should now show operations to your NGINX_KEY key. To view the logs, navigate to Cloud Logging in your cloud console. In the project you've been using, add the following filter:

resource.type="cloudkms_cryptokeyversion"

After running the query, you should see asymmetric key operations to your NGINX_KEY key.

Optional configurations

You may need to create an external TCP/UDP load balancer to expose your NGINX server with an external IP.

If you need to use NGINX as a reverse proxy with load balancing, consider updating the NGINX configuration file. Learn more about configuring NGINX as a reverse proxy by reading All-Active HA for NGINX Plus on the Google Cloud Platform.

Next steps

You now have configured your NGINX server to use TLS offloading to Cloud HSM.