Storing service account keys in Hashicorp Vault

Storing service account secrets in Hashicorp Vault

This feature allows you to store Google service account certs for Apigee Hybrid in Hashicorp Vault, an external secret manager. External secret managers allow you to manage how data is stored in Kubernetes, including managing data residency and fine grained access controls.

If you are not using Workload Identity on GKE clusters or Workload Identity Federation on EKS and AKS, your Apigee hybrid components need to authenticate Google service accounts to be able to perform their tasks. There are three methods for storing and referring to Google service account keys in Apigee hybrid:

Service account cert files (.json files) stored on a hard drive. Refer to these in your overrides with the serviceAccountPath configuration property. For example:
```
logger:
  serviceAccountPath: service-accounts/myhybridorg-apigee-logger.json
```
Service account cert files (.json files) stored on a hard drive. Refer to these in your overrides with the serviceAccountPath configuration property. See About service accounts.
Service account certs stored in a Kubernetes secret. Refer to these in your overrides with the serviceAccountRef configuration property. See Storing data in a Kubernetes secret.
Service account certs stored in Hashicorp Vault, explained in this guide. Refer to these in your overrides with the serviceAccountSecretProviderClass configuration property.

Set up to store service account secrets in Vault

Install CSI driver and Vault provider

If you haven't already installed the CSI driver on your cluster using Helm, follow the instructions in Secrets Store CSI Driver: Installation. For more information, see Installing the Vault CSI provider in the Vault documentation.

See Apigee hybrid supported platforms and versions for the minimum CSI Driver versions supported by Apigee hybrid.

Create Vault secrets, policies, and roles

Use the Vault UI or APIs to create secrets and grant permissions for the Kubernetes service accounts used by Apigee hybrid to read those secrets.

Create the organization and environment-specific secrets in the following format:

Secret Key	Secret Data
secret/data/apigee/orgsakeys	{ "cassandraBackup": "*", "cassandraRestore": "", "connectAgent": "", "logger": "", "mart": "", "metrics": "", "mint": "", "udca": "", "watcher": "**" }
secret/data/apigee/envsakeys-`ENV_NAME`	{ "runtime": "*", "synchronizer": "", "udca": "**". }

Secret Key

Secret Data

secret/data/apigee/orgsakeys

{
  "cassandraBackup": "***",
  "cassandraRestore": "***",
  "connectAgent": "***",
  "logger": "***",
  "mart": "***",
  "metrics": "***",
  "mint": "***",
  "udca": "***",
  "watcher": "***"
}

secret/data/apigee/envsakeys-ENV_NAME

{
  "runtime": "***",
  "synchronizer": "***",
  "udca": "***". 
}

Replace the "***" in each pair with the contents of the .json file for the google service account corresponding to the apigee component. apigee-cassandra-backup and apigee-cassandra-restore both use the apigee-cassandra service account. For example:

{
  "cassandraBackup": "{
    "type": "service_account",
    "project_id": "myhybridorg",
    "private_key_id": "PRIVATE_KEY_ID",
    "private_key": "-----BEGIN PRIVATE KEY-----\nPRIVATE_KEY_TEXT\n-----END PRIVATE KEY-----\n",
    "client_email": "apigee-cassandra@myhybridorg.iam.gserviceaccount.com",
    "client_id": "123456789012345678901",
    "auth_uri": "https://accounts.google.com/o/oauth2/auth",
    "token_uri": "https://oauth2.googleapis.com/token",
    "auth_provider_x509_cert_url": "https://www.googleapis.com/oauth2/v1/certs",
    "client_x509_cert_url": "https://www.googleapis.com/robot/v1/metadata/x509/apigee-cassandra%40myhybridorg.iam.gserviceaccount.com",
    "universe_domain": "googleapis.com"
  }",
  "cassandraRestore":...
...
}

Grant access to the organization secret. Create a text file named orgsakeys-auth-policy.txt with the following contents:
```
path "secret/data/apigee/orgsakeys" {
  capabilities = ["read"]
}
```
Within Vault, create a policy which grants access to the organization secret:
```
vault policy write apigee-orgsakeys-auth orgsakeys-auth-policy.txt
```
For each environment, create a text file named envsakeys-ENV_NAME-auth-policy.txt with the following contents:
```
path "secret/data/apigee/envsakeys-ENV_NAME" {
  capabilities = ["read"]
}
```
Repeat this step for each environment.
Within Vault, create a policy which grants access to the environment secret:
```
vault policy write apigee-envsakeys-ENV_NAME-auth envsakeys-ENV_NAME-auth-policy.txt
```
Repeat this step for each environment.

Create a script called generate-encoded-sas.sh with the following contents:

# generate-encoded-sas.sh

ORG=$APIGEE_ORG            # Apigee organization name
ENVS=$APIGEE_ENV_LIST      # comma separated env names, for example: dev,prod

ORG_SHORT_NAME=$(echo $ORG | head -c 15)
ENCODE=$(echo -n $ORG | shasum -a 256 | head -c 7)
ORG_ENCODE=$(echo "$ORG_SHORT_NAME-$ENCODE")
NAMES=apigee-manager,apigee-cassandra-default,apigee-cassandra-backup-sa,apigee-cassandra-restore-sa,apigee-cassandra-schema-setup-${ORG_ENCODE},apigee-cassandra-schema-val-${ORG_ENCODE},apigee-cassandra-user-setup-${ORG_ENCODE},apigee-mart-${ORG_ENCODE},apigee-mint-task-scheduler-${ORG_ENCODE},apigee-connect-agent-${ORG_ENCODE},apigee-watcher-${ORG_ENCODE},apigee-udca-${ORG_ENCODE},apigee-metrics-apigee-telemetry,apigee-open-telemetry-collector-apigee-telemetry,apigee-logger-apigee-telemetry

for ENV in ${ENVS//,/ }
do
  ENV_SHORT_NAME=$(echo $ENV | head -c 15)
  ENCODE=$(echo -n $ORG:$ENV | shasum -a 256 | head -c 7)
  ENV_ENCODE=$(echo "$ORG_SHORT_NAME-$ENV_SHORT_NAME-$ENCODE")
  NAMES+=,apigee-synchronizer-${ENV_ENCODE},apigee-runtime-${ENV_ENCODE}
done

echo $NAMES

Run the script to generate the service account name list to bind the policy to:

./generate-encoded-sas.sh

Your output should be a list of Kubernetes service account names separated by commas, similar to the following example:

./generate-encoded-sas.sh
apigee-manager,apigee-cassandra-default,apigee-cassandra-backup-sa,
apigee-cassandra-restore-sa,apigee-cassandra-schema-setup-myhybrido
rg-5b044c1,apigee-cassandra-schema-val-myhybridorg-5b044c1,apigee-c
assandra-user-setup-myhybridorg-5b044c1,apigee-mart-myhybridorg-5b0
44c1,apigee-mint-task-scheduler-myhybridorg-5b044c1,apigee-connect-
agent-myhybridorg-5b044c1,apigee-watcher-myhybridorg-5b044c1,apigee
-udca-myhybridorg-5b044c1,apigee-metrics-apigee-telemetry,apigee-op
en-telemetry-collector-apigee-telemetry,apigee-logger-apigee-teleme
try,apigee-synchronizer-myhybridorg-dev-ee52aca,apigee-runtime-myhy
bridorg-dev-ee52aca,apigee-synchronizer-myhybridorg-prod-2d0221c,ap
igee-runtime-myhybridorg-prod-2d0221c

Copy the output text into and separate it into lists, one list for the org service account names and a separate list for the env service account name for each environment. The org service accounts are first in the output list up to apigee-logger-apigee-telemetry.

The list of org service names from the previous example:

apigee-manager,apigee-cassandra-default,apigee-cassandra-backup-sa,
apigee-cassandra-restore-sa,apigee-cassandra-schema-setup-myhybrido
rg-5b044c1,apigee-cassandra-schema-val-myhybridorg-5b044c1,apigee-c
assandra-user-setup-myhybridorg-5b044c1,apigee-mart-myhybridorg-5b0
44c1,apigee-mint-task-scheduler-myhybridorg-5b044c1,apigee-connect-
agent-myhybridorg-5b044c1,apigee-watcher-myhybridorg-5b044c1,apigee
-udca-myhybridorg-5b044c1,apigee-metrics-apigee-telemetry,apigee-op
en-telemetry-collector-apigee-telemetry,apigee-logger-apigee-teleme
try

The env service account names have the pattern apigee-synchronizer-ORG_NAME-ENV_NAME-HASH_TEXT and apigee-runtime-ORG_NAME-ENV_NAME-HASH_TEXT. Separate them into separated lists for each environment. For example, the output from the previous example can be separated into the following two lists:

dev environment:

apigee-synchronizer-myhybridorg-dev-ee52aca,apigee-runtime-myhybrid
org-dev-ee52aca

prod environment:

apigee-synchronizer-myhybridorg-prod-2d0221c,apigee-runtime-myhybri
dorg-prod-2d0221c

Using the policy, create a Vault role which binds the organization specific Apigee service accounts:

vault write auth/kubernetes/role/apigee-orgsakeys \
    bound_service_account_names=LIST_OF_ORG_SA_NAMES \
    bound_service_account_namespaces=apigee \
    policies=apigee-orgsakeys-auth \
    ttl=1m

For each environment, create a Vault role for its service account keys:

vault write auth/kubernetes/role/apigee-envsakeys-ENV_NAME \
    bound_service_account_names=LIST_OF_ENV_NAME_SA_NAMES \
    bound_service_account_namespaces=apigee \
    policies=apigee-envsakeys-ENV_NAME-auth \ 
    ttl=1m

Repeat this step for every environment.

Create `SecretProviderClass` objects

The SecretProviderClass resource tells the CSI driver what provider to communicate with when requesting secrets. The service account keys must be configured via this object. The following table shows the file names (objectNames) expected by Apigee Hybrid:

Service account	Expected secret file names
Cassandra backup	`cassandraBackup`
Cassandra restore	`cassandraRestore`
Connect agent	`connectAgent`
Logger	`logger`
MART	`mart`
Metrics	`metrics`
Monetization	`mint`
Runtime	`runtime`
Synchronizer	`synchronizer`
UDCA	`udca`
Watcher	`watcher`

Use the following SecretProviderClass template to configure this resource for the organization-specific secrets:

apiVersion: secrets-store.csi.x-k8s.io/v1
kind: SecretProviderClass
metadata:
  name: apigee-orgsakeys-spc
spec:
  provider: vault
  parameters:
    roleName: apigee-orgsakeys
    vaultAddress: VAULT_ADDRESS
    # "objectName" is an alias used within the SecretProviderClass to reference
    # that specific secret. This will also be the filename containing the secret.
    # Apigee Hybrid expects these exact values so they must not be changed.
    # "secretPath" is the path in Vault where the secret should be retrieved.
    # "secretKey" is the key within the Vault secret response to extract a value from.
    objects: |
      - objectName: "cassandraBackup"
        secretPath: ""
        secretKey: ""
      - objectName: "cassandraRestore"
        secretPath: ""
        secretKey: ""
      - objectName: "connectAgent"
        secretPath: ""
        secretKey: ""
      - objectName: "logger"
        secretPath: ""
        secretKey: ""
      - objectName: "mart"
        secretPath: ""
        secretKey: ""
      - objectName: "metrics"
        secretPath: ""
        secretKey: ""
      - objectName: "mint"
        secretPath: ""
        secretKey: ""
      - objectName: "udca"
        secretPath: ""
        secretKey: ""
      - objectName: "watcher"
        secretPath: ""
        secretKey: ""

VAULT_ADDRESS is the endpoint where your Vault server is running. If Vault is running in the same cluster as Apigee, the format will generally be http://vault.APIGEE_NAMESPACE.svc.cluster.local:VAULT_SERVICE_PORT.

Save the template to a file named spc-org.yaml.

Apply the org-specific SecretProviderClass to your apigee namespace:
```
kubectl -n APIGEE_NAMESPACE apply -f spc-org.yaml
```

For each environment, use the following SecretProviderClass template to configure this resource for the environment-specific secrets. Repeat this step for every environment:

apiVersion: secrets-store.csi.x-k8s.io/v1
kind: SecretProviderClass
metadata:
  name: apigee-envsakeys-ENV_NAME-spc
spec:
  provider: vault
  parameters:
    roleName: apigee-envsakeys-ENV_NAME
    vaultAddress: VAULT_ADDRESS
    # "objectName" is an alias used within the SecretProviderClass to reference
    # that specific secret. This will also be the filename containing the secret.
    # Apigee Hybrid expects these exact values so they must not be changed.
    # "secretPath" is the path in Vault where the secret should be retrieved.
    # "secretKey" is the key within the Vault secret response to extract a value from.
    objects: |
      - objectName: "runtime"
        secretPath: ""
        secretKey: ""
      - objectName: "synchronizer"
        secretPath: ""
        secretKey: ""
      - objectName: "udca"
        secretPath: ""
        secretKey: ""

VAULT_ADDRESS is the endpoint where your Vault server is running. If Vault is running in the same cluster and namespace as Apigee, the format will generally be http://vault.APIGEE_NAMESPACE.svc.cluster.local:VAULT_SERVICE_PORT.

Save the template to a file named spc-env-ENV_NAME.yaml.

For each environment, apply the environment-specific SecretProviderClass to your apigee namespace:
```
kubectl -n APIGEE_NAMESPACE apply -f spc-env-ENV_NAME.yaml
```
Repeat this step for every environment.

Enable external secrets for Google service accounts

Within your overrides file, add the serviceAccountSecretProviderClass and envs[].serviceAccountSecretProviderClass configuration properties to enable external secret usage for Google service accounts. You can remove or comment out the serviceAccountPath and serviceAccountPaths configuration properties:
```
serviceAccountSecretProviderClass: apigee-orgsakeys-spc

envs:
  - name: ENV_NAME
    serviceAccountSecretProviderClass: apigee-envsakeys-ENV_NAME-spc
```

Apply each Helm chart:

helm upgrade datastore apigee-datastore/ \
    --install \
    --namespace APIGEE_NAMESPACE \
    --atomic \
    -f overrides.yaml

helm upgrade telemetry apigee-telemetry/ \
    --install \
    --namespace APIGEE_NAMESPACE \
    --atomic \
    -f overrides.yaml

helm upgrade redis apigee-redis/ \
    --install \
    --namespace APIGEE_NAMESPACE \
    --atomic \
    -f overrides.yaml

helm upgrade ORG_NAME apigee-org/ \
    --install \
    --namespace APIGEE_NAMESPACE \
    --atomic \
    -f overrides.yaml

For each environment, apply the apigee-env chart:

helm upgrade ENV_NAME apigee-env/ \
    --install \
    --namespace APIGEE_NAMESPACE \
    --set env=ENV_NAME \
    --atomic \
    -f overrides.yaml

Rolling back to using service account cert files

If you need to roll back to using stored Google service account cert files, use the following procedure:

Update your overrides file:

Remove or comment out the serviceAccountSecretProviderClass and envs:serviceAccountSecretProviderClass lines.
Add the serviceAccountPath and serviceAccountPaths configuration properties with the paths to the appropriate service accounts.

For example:

# serviceAccountSecretProviderClass: apigee-orgsakeys-spc - (commented out)

cassandra:
  backup:
    serviceAccountPath: service-accounts/myhybridorg-apigee-cassandra.json
  restore:
    serviceAccountPath: service-accounts/myhybridorg-apigee-cassandra.json

connectAgent:
  serviceAccountPath: service-accounts/myhybridorg-apigee-mart.json

envs:
  - name: test
  # serviceAccountSecretProviderClass: apigee-envsakeys-spc - (commented out)
    serviceAccountPaths:
      runtime: service-accounts/myhybridorg-apigee-runtime.json
      synchronizer: service-accounts/myhybridorg-apigee-synchronizer.json

logger:
  serviceAccountPath: service-accounts/myhybridorg-apigee-logger.json

mart:
  serviceAccountPath: service-accounts/myhybridorg-apigee-mart.json

metrics:
  serviceAccountPath: service-accounts/myhybridorg-apigee-metrics.json

udca:
  serviceAccountPath: service-accounts/myhybridorg-apigee-udca.json

watcher:
  serviceAccountPath: service-accounts/myhybridorg-apigee-watcher.json

Note: If your hybrid installation uses kubernetes secrets instead of service account cert files, use the serviceAccountRef configuration property instead of serviceAccountPath.

Apply each Helm chart:

helm upgrade datastore apigee-datastore/ \
    --install \
    --namespace APIGEE_NAMESPACE \
    --atomic \
    -f overrides.yaml

helm upgrade telemetry apigee-telemetry/ \
    --install \
    --namespace APIGEE_NAMESPACE \
    --atomic \
    -f overrides.yaml

helm upgrade redis apigee-redis/ \
    --install \
    --namespace APIGEE_NAMESPACE \
    --atomic \
    -f overrides.yaml

helm upgrade ORG_NAME apigee-org/ \
    --install \
    --namespace APIGEE_NAMESPACE \
    --atomic \
    -f overrides.yaml

For each environment, apply the apigee-env chart:

helm upgrade ENV_NAME apigee-env/ \
    --install \
    --namespace APIGEE_NAMESPACE \
    --set env=ENV_NAME \
    --atomic \
    -f overrides.yaml

Repeat this step for every environment.

Storing service account keys in Hashicorp Vault

Storing service account secrets in Hashicorp Vault

Set up to store service account secrets in Vault

Install CSI driver and Vault provider

Create Vault secrets, policies, and roles

Create SecretProviderClass objects

Enable external secrets for Google service accounts

Rolling back to using service account cert files

Create `SecretProviderClass` objects