Edit on GitHub
Report issue
Page history

Injecting secrets into Google Kubernetes Engine Pods using HashiCorp Vault and Google Cloud Auth

Author(s): @soeirosantos ,   Published: 2021-08-09

Romulo Santos

Contributed by the Google Cloud community. Not official Google documentation.

In this tutorial, you use the Agent Sidecar Injector to read secrets from HashiCorp Vault and inject secrets into Pods running in Google Kubernetes Engine. This tutorial uses the Google Cloud Auth method with a trust relationship configured based on Workload Identity to authenticate with Vault.

This tutorial emulates a common use case in which you have an external Vault server providing secrets management as a service to multiple clients and applications running in different platforms and environments. In this case, you connect your workloads running in GKE to this Vault server to read secrets.

Before you begin

For this tutorial, you need a Google Cloud project. You can create a new one, or select a project that you have already created.

  1. Create a Google Cloud project.
  2. Enable billing for your project.
  3. For the commands in this tutorial, you use the gcloud command-line interface. To install the Cloud SDK, which includes the gcloud tool, follow these instructions.
  4. To install the Vault Agent Injector, you use the Helm command-line interface. Follow the instructions to install Helm.

Costs

This tutorial uses billable components of Google Cloud, including the following:

Use the pricing calculator to generate a cost estimate based on your projected usage.

Start a Vault server in dev mode

In this section, you use a Compute Engine VM instance to start a Vault server in dev mode for experimentation. There is nothing special about this choice; a VM instance is a quick and convenient way to simulate a situation in which the Vault server is external to your Kubernetes cluster. Never run a server in dev mode in production.

  1. Define some variables that you use during the tutorial:

    gcp_project=[YOUR_PROJECT_NAME]
    cluster_name=[YOUR_CLUSTER_NAME]
    cluster_region=us-east1
    
    gcloud config set project $gcp_project
    
  2. Create a file named startup-script.sh with the instructions to download and run Vault in dev mode:

    cat << EOF > startup-script.sh
    #!/bin/bash
    
    set -e
    
    wget https://releases.hashicorp.com/vault/1.7.3/vault_1.7.3_linux_amd64.zip
    sudo apt install unzip -y
    unzip vault_1.7.3_linux_amd64.zip
    
    ./vault server -dev -dev-root-token-id="root" \
        -dev-listen-address="0.0.0.0:8200"  &
    EOF
    

    This file runs during the VM instance startup.

  3. Change permissions for the file to make it executable:

    chmod +x startup-script.sh
    
  4. Create a VM instance using the startup script to start a Vault server:

    gcloud compute instances create vault-dev \
      --tags vault-dev \
      --metadata-from-file=startup-script=startup-script.sh \
      --image-family=ubuntu-1804-lts \
      --image-project=ubuntu-os-cloud \
      --machine-type=e2-small \
      --zone="$cluster_region"-b #change the zone accordingly
    
  5. Add a firewall rule to allow access to the Vault port 8200:

    gcloud compute firewall-rules create vault-dev \
      --target-tags vault-dev --direction INGRESS \
      --allow tcp:8200
    
  6. Check whether your Vault server is up:

    VAULT_DEV_PUBLIC_IP=$(gcloud compute instances describe vault-dev \
      --zone ${cluster_region}-b \
      --format="json" \
      --format="value(networkInterfaces.accessConfigs[0].natIP)")
    
    export VAULT_ADDR=http://"$VAULT_DEV_PUBLIC_IP":8200
    
    vault status
    # omitting output
    
    export VAULT_TOKEN=root #the vault token is defined in the startup script
    
    vault secrets list
    # omitting output
    

    It may take a minute for the installation and initialization to complete after the machine has started.

Create the GKE cluster

  1. Create the GKE cluster:

    gcloud beta container clusters create "$cluster_name" \
      --release-channel regular \
      --enable-ip-alias \
      --region "$cluster_region" \
      --workload-pool="${gcp_project}".svc.id.goog
    
  2. Check the Kubernetes configuration:

    kubectl config current-context
    
  3. If your current context doesn't match the cluster that you just created, or if you need to connect to it again after changing your kubeconfig, use the get-credentials command to configure your local access to the cluster:

    gcloud container clusters get-credentials "$cluster_name" --region "$cluster_region"
    

Enable and configure the Google Cloud Auth method

To configure the Google Cloud Auth method, you need to create a Google service account that will be used by the Vault server to validate the incoming login requests. Instead of using existing roles, this tutorial configures a custom role to make sure that the service account is properly privileged with only the required permissions.

  1. Create the role:

    gcloud iam roles create VaultAuthRole \
      --title=vault-auth-role \
      --stage=GA \
      --description="Role used for the Vault Auth Method" \
      --project "$gcp_project" \
      --permissions=iam.serviceAccounts.get,iam.serviceAccountKeys.get,compute.instances.get
    
  2. Create the service account:

    gcloud iam service-accounts create vault-auth-gsa --display-name="vault-auth-gsa"
    
  3. Apply the policy:

    gcloud projects add-iam-policy-binding "$gcp_project" \
      --member "serviceAccount:vault-auth-gsa@${gcp_project}.iam.gserviceaccount.com" \
      --role projects/"$gcp_project"/roles/VaultAuthRole
    
  4. Create the key:

    gcloud iam service-accounts keys create key.json --iam-account=vault-auth-gsa@"$gcp_project".iam.gserviceaccount.com
    
  5. Enable and configure the Google Cloud Auth method in Vault:

    vault auth enable gcp
    vault write auth/gcp/config credentials=@key.json
    

Configure Workload Identity and respective Vault role

To configure Workload Identity, you need a Google service account that will be impersonated by a Kubernetes service account to access Google Cloud APIs and to establish the trust relationship with Vault.

  1. Create the service account:

    gcloud iam service-accounts create my-service-gsa --display-name="my-service-gsa"
    
  2. Create the policy bindings:

    gcloud iam service-accounts add-iam-policy-binding \
      --role roles/iam.workloadIdentityUser \
      --member "serviceAccount:${gcp_project}.svc.id.goog[staging/my-service-ksa]" \
      my-service-gsa@${gcp_project}.iam.gserviceaccount.com
    
    gcloud projects add-iam-policy-binding "$gcp_project" \
      --member "serviceAccount:my-service-gsa@${gcp_project}.iam.gserviceaccount.com" \
      --role roles/iam.serviceAccountTokenCreator
    

    The first policy binding allows the Kubernetes service account my-service-ksa in the namespace staging to impersonate the Google service account my-service-gsa. The second policy binding allows the Google service account to sign JWTs to perform the Vault login. The only permission necessary for the Vault login is the iam.serviceAccounts.signJwt permission. You may consider creating a custom role with fewer privileges as you did for the service account used to configure the Vault server.

  3. Create a Kubernetes namespace and service account to match the workload identity configured in the previous step:

    kubectl create namespace staging
    
    kubectl create serviceaccount my-service-ksa -n staging
    
    kubectl annotate serviceaccount my-service-ksa \
      iam.gke.io/gcp-service-account=my-service-gsa@"$gcp_project".iam.gserviceaccount.com \
      -n staging
    
  4. Check whether the workload identity is working as expected:

    kubectl run gcloud -it --rm --restart=Never \
      --serviceaccount my-service-ksa \
      -n staging \
      --image google/cloud-sdk:slim gcloud auth list
    

    This command launches a Pod and checks with the command gcloud auth list whether the active (and only) identity corresponds to the Google service account email address previously configured. You should see the Google service account my-service-gsa@"$gcp_project".iam.gserviceaccount.com as active.

  5. Create the Vault role that binds the Google service account granting access to the secrets:

    vault write auth/gcp/role/my-service-stg-role \
      type="iam" \
      policies="stg" \
      max_jwt_exp="3600" \
      bound_service_accounts="my-service-gsa@${gcp_project}.iam.gserviceaccount.com"
    

    The policy stg doesn't exist yet. This is handled in the secrets step.

  6. Check whether the Vault login is working as expected:

    kubectl run vault -it --rm --restart=Never \
      --serviceaccount my-service-ksa \
      -n staging \
      --image vault \
      -- vault login -address="$VAULT_ADDR" -method=gcp service_account="my-service-gsa@${gcp_project}.iam.gserviceaccount.com" role="my-service-stg-role"
    

    This command launches a Pod and performs a vault login command from within the Pod pointing to your Vault server. You should see a successful response containing a Vault token.

Inject secrets in a Pod

In this section, you start writing some secrets to Vault. For that, you mount a kv-v2 secrets engine and create a Vault policy that allows your Vault role to read from this secrets backend.

  1. Set up the secrets engine:

    vault secrets enable -path=my-service/secrets kv-v2
    
    vault kv put my-service/secrets/stg/database/config username="db-username" password="db-password"
    
  2. Check the data just written:

    vault kv get my-service/secrets/stg/database/config
    
  3. Create the stg policy:

    vault policy write stg - <<EOF
    path "my-service/secrets/data/stg/database/config" {
      capabilities = ["read"]
    }
    EOF
    

    The stg policy allows reading only to the my-service/secrets/stg path, preventing that the role (and application) from having access to secrets that are not necessary.

  4. Install the Vault Agent Injector using Helm, as described in the Vault Agent Injector installation documentation:

    1. Add the HashiCorp Helm repository:

      helm repo add hashicorp https://helm.releases.hashicorp.com
      
    2. Check whether you have access to the Helm chart:

      helm search repo hashicorp/vault
      

      You should see output similar to this:

      NAME            CHART VERSION    APP VERSION    DESCRIPTION
      hashicorp/vault 0.13.0           1.7.3          Official HashiCorp Vault Chart
      
    3. Create the namespace:

      kubectl create ns vault-agent-injector
      
    4. Install the Vault Agent Injector:

      helm install vault hashicorp/vault --namespace vault-agent-injector --set="server.enabled=false" --set="injector.externalVaultAddr=$VAULT_ADDR" --set "injector.authPath=auth/gcp"
      

      For more information about configuration options, see the Vault Helm documentation.

  5. Create a Deployment resource with the Agent Injector annotations:

    kubectl apply -f - <<EOF
    apiVersion: apps/v1
    kind: Deployment
    metadata:
      name: my-service
      namespace: staging
      labels:
        app: my-service
    spec:
      selector:
        matchLabels:
          app: my-service
      replicas: 1
      template:
        metadata:
          annotations:
            vault.hashicorp.com/agent-inject: "true"
            vault.hashicorp.com/agent-inject-status: "update"
            vault.hashicorp.com/auth-type: "gcp"
            vault.hashicorp.com/auth-config-type: "iam"
            vault.hashicorp.com/auth-config-service-account: "my-service-gsa@${gcp_project}.iam.gserviceaccount.com"
            vault.hashicorp.com/role: "my-service-stg-role"
            vault.hashicorp.com/agent-inject-secret-database-config.txt: "my-service/secrets/data/stg/database/config"
            vault.hashicorp.com/agent-inject-template-database-config.txt: |
              {{- with secret "my-service/secrets/data/stg/database/config" -}}
              postgresql://{{ .Data.data.username }}:{{ .Data.data.password }}@postgres:5432/wizard
              {{- end -}}
          labels:
            app: my-service
        spec:
          serviceAccountName: my-service-ksa
          containers:
            - name: my-service
              image: nginx
    EOF
    
  6. Check whether the Pod is running:

    kubectl get po -n staging
    

    The output should be similar to the following:

    NAME                         READY   STATUS    RESTARTS   AGE
    my-service-d5dcfb55f-w8g89   2/2     Running   0          22s
    

    If the Pod is stuck in the Init status, check the vault-agent-init container logs to identify the problem:

    # run this only if you need to debug
    kubectl logs my-service-d5dcfb55f-w8g89 vault-agent-init -n staging
    
  7. Check the my-service container and verify that the secrets have been injected:

    kubectl exec \
      $(kubectl get pod -l app=my-service -n staging -o jsonpath="{.items[0].metadata.name}") \
      -c my-service -n staging -- cat /vault/secrets/database-config.txt
    
    postgresql://db-username:db-password@postgres:5432/wizard%
    

    For more information about the available annotations, configuration options, and features, see the Agent Injector documentation.

Cleaning up

  1. Delete the GKE cluster:

    gcloud container clusters delete "$cluster_name" --region "$cluster_region"
    
  2. Delete the Compute Engine VM instance and firewall rule:

    gcloud compute instances delete vault-dev --zone "$cluster_region"-b
    gcloud compute firewall-rules delete vault-dev
    
  3. Delete the custom role and service accounts:

    gcloud iam roles delete VaultAuthRole --project "$gcp_project"
    gcloud iam service-accounts delete vault-auth-gsa@"$gcp_project".iam.gserviceaccount.com
    gcloud iam service-accounts delete my-service-gsa@"$gcp_project".iam.gserviceaccount.com
    

Submit a tutorial

Share step-by-step guides

Submit a tutorial

Request a tutorial

Ask for community help

Submit a request

View tutorials

Search Google Cloud tutorials

View tutorials

Except as otherwise noted, the content of this page is licensed under the Creative Commons Attribution 4.0 License, and code samples are licensed under the Apache 2.0 License. For details, see our Site Policies. Java is a registered trademark of Oracle and/or its affiliates.