Using Vault on Compute Engine for Secret Management

This tutorial shows how to use Vault, an open source tool for secret management, with Google Cloud Platform (GCP). Vault employs a secret-sharing scheme to seal and unseal its ability to decrypt the Vault contents. It supports multiple provider backends, and a variety of methods for authentication, storage, and auditing.

By using the Vault GCP Auth Plugin Backend, you can allow applications running on GCP to seamlessly authenticate to Vault using service accounts and GCP roles and permissions created within your organization. In the tutorial, you'll see how to deploy Vault to Google Compute Engine and how to configure the Vault GCP Auth Plugin Backend.

Architecture of Vault in a GCP environment

Objectives

  • Deploy Vault to Compute Engine using Terraform.
  • Initialize and unseal Vault from keys stored in Google Cloud Storage.
  • Configure the Vault GCP Auth Plugin Backend.
  • Create a signed JSON web token (JWT) and retrieve the Vault authentication token.
  • Use the Vault authentication token to read and write secrets.

Costs

This tutorial uses billable components of GCP, including:

The Pricing Calculator estimates the cost of this environment at around $14 per day.

Before you begin

  1. Sign in to your Google account.

    If you don't already have one, sign up for a new account.

  2. Select or create a Cloud Platform project.

    Go to the Manage resources page

  3. Enable billing for your project.

    Enable billing

  4. Enable the Google Compute Engine, Google Kubernetes Engine, Cloud KMS, and Cloud IAM APIs.

    Enable the APIs

Deploying Vault using Terraform

Terraform is an infrastructure automation tool used in this tutorial to provision the following resources:

  • TLS certificates for securing the Vault API.
  • The service account for the Vault Compute Engine instance.
  • The Cloud Identity & Access Management (IAM) policy bindings that specify how Vault can interact with Cloud Storage, Cloud IAM, and Cloud KMS.
  • The Vault instance template and startup script that are used to install Vault.
  • The managed instance group for the instance template.
  • The Cloud Storage bucket for the Vault storage backend.
  • The Cloud Storage bucket for Vault assets like the unseal keys and TLS certificates and keys.

These resources can also be created using standard Linux tools like openssl and the Google Cloud SDK or the Cloud Platform Console. But using Terraform gives you a single tool to provision the majority of resources used in this tutorial. You can easily parameterize Terraform with variables and write infrastructure as code. Terraform also lets you preview changes using the terraform plan command.

This tutorial uses the terraform-google-vault module in the example code to automate the creation of the Vault TLS keys and certificates and to store encrypted copies of them in a Cloud Storage bucket. These keys and certificates are later decrypted automatically by the Vault instance startup script.

After the instance is started, the startup script initializes Vault and stores an encrypted copy of the unseal keys and root token to a Cloud Storage bucket. In this tutorial, you decrypt those keys and use them to manually unseal Vault.

  1. Go to Cloud Shell.

    Open Cloud Shell

  2. Make a local clone of the GitHub repository with the Terraform module and example code.

    git clone https://github.com/GoogleCloudPlatform/terraform-google-vault
    

  3. Change to the directory that contains the example code:

    cd examples/vault-on-gce
    

  4. Create the Cloud KMS KeyRing and key for asset encryption:

    gcloud kms keyrings create vault --location global

    gcloud kms keys create vault-init --location global --keyring vault --purpose encryption

  5. Create the terraform.tfvars file with the project ID, bucket name, and KeyRing name. Notice that the bucket name is based on your project ID and the string "vault".

    export GOOGLE_PROJECT=$(gcloud config get-value project)
    cat - > terraform.tfvars <<EOF
    project_id = "${GOOGLE_PROJECT}"
    storage_bucket = "${GOOGLE_PROJECT}-vault"
    kms_keyring_name = "vault"
    EOF
    

  6. Configure your Cloud Shell environment to use the latest version of Terraform by installing it with the helper script:

    curl -sL https://goo.gl/yZS5XU | bash
    source ${HOME}/.bashrc
    

  7. Use Terraform commands to deploy Vault to Compute Engine:

    terraform init
    terraform plan
    terraform apply
    

    After a few minutes, the Vault instance is ready to be configured.

Initializing Vault

Now that Vault is installed, you must unseal it before you perform any other configuration or actions. Unsealing is done by decrypting the unseal keys that are in Cloud Storage and manually entering them using the Vault command-line tool. After you unseal Vault, you can configure the GCP Auth Plugin Backend, audit log, and roles using the root authentication token.

  1. Use SSH to connect to the Vault instance:

    gcloud compute ssh \
        $(gcloud compute instances list --limit=1 --filter=name~vault- --uri) -- sudo bash
    

  2. Export Vault environment variables:

    export VAULT_ADDR=https://127.0.0.1:8200
    export VAULT_CACERT=/etc/vault/vault-server.ca.crt.pem
    export VAULT_CLIENT_CERT=/etc/vault/vault-server.crt.pem
    export VAULT_CLIENT_KEY=/etc/vault/vault-server.key.pem
    

    By default, Vault uses a secret-sharing scheme that requires three out of five unseal keys to be provided manually.

    As noted earlier, the unseal keys and root authentication token for Vault are generated when the instance is created. The keys and token are encrypted and stored in one of the Cloud Storage buckets created by Terraform for Vault assets. See the Cloud KMS documentation for details on storing secrets in Cloud Storage.

  3. Obtain the unseal keys from Cloud Storage and decrypt them using Cloud KMS:

    export GOOGLE_PROJECT=$(gcloud config get-value project)
    gcloud kms decrypt \
        --location=global  \
        --keyring=vault \
        --key=vault-init \
        --plaintext-file=/dev/stdout \
        --ciphertext-file=<(gsutil cat gs://${GOOGLE_PROJECT}-vault-assets/vault_unseal_keys.txt.encrypted)
    

    The output looks like the following:

    Unseal Key 1: oO1UNH4TPVZRFuGWUa9D0eciJ2LMMgi2PYxm/bLL/lt0
    Unseal Key 2: +4q3O9LT46p22uTcDTYZyIVvVt+mxhB8OQ87vZFc3pkp
    Unseal Key 3: tFnuYrDD1Xgkec3wFXhk93wIjEfq3kCOD34i16MkE+pl
    Unseal Key 4: DFQhkl344Z+jpwr9L/looYjNYPAh8/UKGF5fXAO2Vj0W
    Unseal Key 5: XOQVAZCKt6njWcF6IAP19ER1WnRqhH5MllyvcywBLtaw
    Initial Root Token: 8d9b6907-0386-c422-cad8-624ceba2d0ae
    

  4. Start the process of unsealing.

    Vault enters a sealed state when it is started, and the contents cannot be accessed until you unseal it. This tutorial installs Vault with the default settings, so that at least three unseal keys are required in order to manually unseal Vault.

    vault unseal
    

  5. When you're prompted, enter one of the unseal keys.

  6. Repeat the vault unseal command two more times, for a total of three times. After each command, enter another one of the unseal keys. (Use a different key each time.)

    After you've entered the vault unseal command the third time, Vault is unsealed.

  7. Verify that Vault is unsealed:

    vault status
    

    If Vault is unsealed, you'll see output like the following:

    Sealed: false
    Key Shares: 5
    Key Threshold: 3
    Unseal Progress: 0
    Unseal Nonce:
    Version: 0.8.1
    Cluster Name: vault-cluster-3ee8bd03
    Cluster ID: a7651e75-2d69-2a0e-2ece-285a60364c17
    

  8. Authenticate to Vault with the root token, replacing [ROOT_TOKEN] with the Initial Root Token value that was printed by the kms decrypt command you ran earlier:

    vault auth [ROOT_TOKEN]
    

    The vault auth command writes the file ${HOME}/.vault-token that the vault command will use to authenticate to Vault.

  9. Enable the Vault syslog audit backend to send the Vault audit logs to Stackdriver:

    vault audit-enable syslog
    

Configuring the Vault GCP Auth Plugin Backend

  1. Enable the Vault GCP Auth Plugin Backend:

    vault auth-enable gcp
    

  2. Set the backend authentication credentials:

    vault write auth/gcp/config credentials="$(cat /etc/vault/gcp_credentials.json)"
    

  3. Create a Vault role named dev-role that has the default policy. For the name of the service account, use vault-admin plus the ID of your GCP project.

    GOOGLE_PROJECT=$(gcloud config get-value project)
    SERVICE_ACCOUNT=vault-admin@${GOOGLE_PROJECT}.iam.gserviceaccount.com
    vault write auth/gcp/role/dev-role \
        type="iam" \
        project_id="${GOOGLE_PROJECT}" \
        policies="default" \
        service_accounts="${SERVICE_ACCOUNT}"
    

For information about advanced Vault configuration, see the Vault Policies documentation.

Authenticating to Vault using Cloud IAM

The Vault GCP Auth Plugin Backend uses a JSON web token (JWT) that's signed by the service account to obtain a Vault authentication token. The signing request is a JSON document that specifies the Vault role, service account name, and expiration time. This JSON request is included with the Cloud IAM sign-jwt command to obtain the JWT used to authenticate with Vault.

  1. Create the JSON signing request document:

    GOOGLE_PROJECT=$(gcloud config get-value project)
    SERVICE_ACCOUNT=vault-admin@${GOOGLE_PROJECT}.iam.gserviceaccount.com
    cat - > login_request.json <<EOF
    {
        "aud": "vault/dev-role",
        "sub": "${SERVICE_ACCOUNT}",
        "exp": $((EXP=$(date +%s)+600))
    }
    EOF
    

    The request specifies the Vault role, service account name, and expiration time. In this example, the expiration time is set 10 minutes from the time the JSON document was created.

  2. Request a signed JWT:

    JWT_TOKEN=$(gcloud beta iam service-accounts sign-jwt login_request.json \
        signed_jwt.json \
        --iam-account=${SERVICE_ACCOUNT} && cat signed_jwt.json)
    

  3. Use the JWT token to authenticate to Vault:

    vault write -field=token auth/gcp/login role=dev-role jwt=${JWT_TOKEN} > ~/.vault-token
    

  4. Test access by writing and reading a value to the Vault cubbyhole:

    vault write /cubbyhole/hello value=world
    vault read /cubbyhole/hello
    

    If Vault is working as expected, you see output like the following:

    Key     Value
    ---     -----
    value   world
    

You now have a Vault instance that you can authenticate with using an IAM service account. This lets you use the Vault command line tool and API in your applications to read and write secrets.

Cleaning up

After you've finished the tutorial, you can clean up the resources you created on GCP so you won't be billed for them in the future.

Deleting the Vault instance

Close the SSH session and then run the following command to remove all resources created by Terraform. This also removes the Cloud Storage bucket that contains the Vault data.

terraform destroy

What's next

Send feedback about...