Storing data in an external secret manager
This guide tells you how to store and manage the following types of information in the external secret storage service, Hashicorp Vault, instead of in your overrides.yaml file.
- AX Hash Salt
- Redis password
- Encryption keys
To store other kinds of information in Vault, see:
Prerequisites
- A Kubernetes secrets store CSI driver. You can install it by following instructions in Secrets Store CSI Driver: Installation. For supported versions, see Apigee hybrid supported platforms and versions: Secret Store CSI driver.
- The Vault CSI provider. You can install it by following instructions in Installing the Vault CSI provider. For supported versions, see Apigee hybrid supported platforms and versions: Vault.
Procedure
-
Create the Vault secrets, policies, and roles.
- Use the Vault UI or APIs to create secrets and grant permissions for the Apigee Kubernetes service accounts to read those secrets as described here. The secrets need to consist of a key and one or more values as shown in the following table:
Secret Key Secret Data secret/data/apigee/axhashsalt
{ "ax-hash-salt": "AX_HASH_SALT_VALUE" }
secret/data/apigee/redis
{ "redis-password": "REDIS_PASSWORD_VALUE" }
secret/data/apigee/orgencryptionkeys
{ "kmsEncryptionKey": "KMS_ENCRYPTION_KEY_VALUE" "kvmEncryptionKey": "KVM_ENCRYPTION_KEY_VALUE" "contractEncryptionKey": "CONTRACT_ENCRYPTION_KEY_VALUE" }
secret/data/apigee/envencryptionkeys
{ "cacheEncryptionKey": "CACHE_ENCRYPTION_KEY_VALUE" "kvmEncryptionKey": "KVM_ENCRYPTION_KEY_VALUE" "envKvmEncryptionKey": "ENV_KVM_ENCRYPTION_KEY_VALUE" "kmsEncryptionKey": "KMS_ENCRYPTION_KEY_VALUE" }
- Within Vault, create policies which grant access to the secrets:
cat axhashsalt-auth-policy.txt path "secret/data/apigee/axhashsalt" { capabilities = ["read"] }
cat redis-auth-policy.txt path "secret/data/apigee/redis" { capabilities = ["read"] }
cat orgencryptionkeys-auth-policy.txt path "secret/data/apigee/orgencryptionkeys" { capabilities = ["read"] }
cat envencryptionkeys-auth-policy.txt path "secret/data/apigee/envencryptionkeys" { capabilities = ["read"] }
vault policy write apigee-axhashsalt-auth axhashsalt-auth-policy.txt
vault policy write apigee-redis-auth redis-auth-policy.txt
vault policy write apigee-orgencryptionkeys-auth orgencryptionkeys-auth-policy.txt
vault policy write apigee-envencryptionkeys-auth envencryptionkeys-auth-policy.txt
- Create a script called
generate-encoded-sas.sh
with the following contents:# generate-encoded-sas.sh ORG=$APIGEE_ORG # Apigee organization name ENVS=$APIGEE_ENV_LIST # comma separated env names, for example: dev,prod ORG_SHORT_NAME=$(echo $ORG | head -c 15) ENCODE=$(echo -n $ORG | shasum -a 256 | head -c 7) ORG_ENCODE=$(echo "$ORG_SHORT_NAME-$ENCODE") NAMES="apigee-manager,apigee-redis-default,apigee-redis-envoy-default,apigee-mart-${ORG_ENCODE},apigee-mint-task-scheduler-${ORG_ENCODE}" for ENV in ${ENVS//,/ } do ENV_SHORT_NAME=$(echo $ENV | head -c 15) ENCODE=$(echo -n $ORG:$ENV | shasum -a 256 | head -c 7) ENV_ENCODE=$(echo "$ORG_SHORT_NAME-$ENV_SHORT_NAME-$ENCODE") NAMES+=,apigee-runtime-${ENV_ENCODE},apigee-synchronizer-${ENV_ENCODE} done echo $NAMES
- Run the script to generate the service account name list to bind the policies to:
chmod +x ./generate-encoded-sas.sh
./generate-encoded-sas.sh
The output should list the encoded service account names.
- Using the policies, create Vault roles which bind the required Apigee service accounts.
vault write auth/kubernetes/role/apigee-axhashsalt \ bound_service_account_names=BOUND_SA_NAMES \ bound_service_account_namespaces=APIGEE_NAMESPACE \ policies=apigee-axhashsalt-auth \ ttl=1m
vault write auth/kubernetes/role/apigee-redis \ bound_service_account_names=BOUND_SA_NAMES \ bound_service_account_namespaces=APIGEE_NAMESPACE \ policies=apigee-redis-auth \ ttl=1m
vault write auth/kubernetes/role/apigee-orgencryptionkeys \ bound_service_account_names=BOUND_SA_NAMES \ bound_service_account_namespaces=APIGEE_NAMESPACE \ policies=apigee-orgencryptionkeys-auth \ ttl=1m
vault write auth/kubernetes/role/apigee-envencryptionkeys \ bound_service_account_names=BOUND_SA_NAMES \ bound_service_account_namespaces=APIGEE_NAMESPACE \ policies=apigee-envencryptionkeys-auth \ ttl=1m
- Use the Vault UI or APIs to create secrets and grant permissions for the Apigee Kubernetes service accounts to read those secrets as described here. The secrets need to consist of a key and one or more values as shown in the following table:
- Create
SecretProviderClass
objects.- Add the following secrets via the
SecretProviderClass
resources. These resources tell the CSI driver what provider to communicate with when requesting secrets. The following table shows the file names (objectNames
) expected by Apigee hybrid:Secret Expected secret file names AX Hash Salt ax-hash-salt
Redis redis-password
Org Encryption Keys kmsEncryptionKey
kvmEncryptionKey
contractEncryptionKey
Env Encryption Keys kmsEncryptionKey
kvmEncryptionKey
envKvmEncryptionKey
cacheEncryptionKey
- Use the following
SecretProviderClass
templates to configure these resources:# axhashsalt-spc.yaml apiVersion: secrets-store.csi.x-k8s.io/v1 kind: SecretProviderClass metadata: name: apigee-axhashsalt-spc spec: provider: vault parameters: roleName: apigee-axhashsalt vaultAddress: VAULT_ADDRESS # "objectName" is an alias used within the SecretProviderClass to reference # that specific secret. This will also be the filename containing the secret. # Apigee Hybrid expects these exact values so they must not be changed. # "secretPath" is the path in Vault where the secret should be retrieved. # "secretKey" is the key within the Vault secret response to extract a value from. objects: | - objectName: "ax-hash-salt" secretPath: "" secretKey: ""
# redis-spc.yaml apiVersion: secrets-store.csi.x-k8s.io/v1 kind: SecretProviderClass metadata: name: apigee-redis-spc spec: provider: vault parameters: roleName: apigee-redis vaultAddress: VAULT_ADDRESS # "objectName" is an alias used within the SecretProviderClass to reference # that specific secret. This will also be the filename containing the secret. # Apigee Hybrid expects these exact values so they must not be changed. # "secretPath" is the path in Vault where the secret should be retrieved. # "secretKey" is the key within the Vault secret response to extract a value from. objects: | - objectName: "redis-password" secretPath: "" secretKey: ""
# orgencryptionkeys-spc.yaml apiVersion: secrets-store.csi.x-k8s.io/v1 kind: SecretProviderClass metadata: name: apigee-orgencryptionkeys-spc spec: provider: vault parameters: roleName: apigee-orgencryptionkeys vaultAddress: VAULT_ADDRESS # "objectName" is an alias used within the SecretProviderClass to reference # that specific secret. This will also be the filename containing the secret. # Apigee Hybrid expects these exact values so they must not be changed. # "secretPath" is the path in Vault where the secret should be retrieved. # "secretKey" is the key within the Vault secret response to extract a value from. objects: | - objectName: "kmsEncryptionKey" secretPath: "" secretKey: "" - objectName: "kvmEncryptionKey" secretPath: "" secretKey: "" - objectName: "contractEncryptionKey" secretPath: "" secretKey: ""
# envencryptionkeys-spc.yaml apiVersion: secrets-store.csi.x-k8s.io/v1 kind: SecretProviderClass metadata: name: apigee-envencryptionkeys-spc spec: provider: vault parameters: roleName: apigee-envencryptionkeys vaultAddress: VAULT_ADDRESS # "objectName" is an alias used within the SecretProviderClass to reference # that specific secret. This will also be the filename containing the secret. # Apigee Hybrid expects these exact values so they must not be changed. # "secretPath" is the path in Vault where the secret should be retrieved. # "secretKey" is the key within the Vault secret response to extract a value from. objects: | - objectName: "cacheEncryptionKey" secretPath: "" secretKey: "" - objectName: "kvmEncryptionKey" secretPath: "" secretKey: "" - objectName: "envKvmEncryptionKey" secretPath: "" secretKey: "" - objectName: "kmsEncryptionKey" secretPath: "" secretKey: ""
VAULT_ADDRESS is the endpoint where your Vault server is running. If Vault is running in the same cluster and namespace as Apigee, the format will generally be
http://vault.APIGEE_NAMESPACE.svc.cluster.local:VAULT_SERVICE_PORT
. - Apply the above
SecretProviderClasses
to theAPIGEE_NAMESPACE
namespace:kubectl -n APIGEE_NAMESPACE apply -f axhashsalt-spc.yaml
kubectl -n APIGEE_NAMESPACE apply -f redis-spc.yaml
kubectl -n APIGEE_NAMESPACE apply -f orgencryptionkeys-spc.yaml
kubectl -n APIGEE_NAMESPACE apply -f envencryptionkeys-spc.yaml
- Add the following secrets via the
- Enable the external secret for the AX Hash Salt.
- Within your
overrides.yaml
file, add the following configuration to enable external secret usage for the AX Hash Salt:axHashSaltSecretProviderClass: apigee-axhashsalt-spc
- Apply the change by upgrading the
org
Helm chart:helm upgrade org apigee-org/ \ --namespace APIGEE_NAMESPACE \ -f overrides.yaml
- Within your
- Enable the external secret for the Redis password.
- Within your
overrides.yaml
file, add the following configuration to enable external secret usage for the Redis password:redis: auth: secretProviderClass: apigee-redis-spc
- Then apply the changes by upgrading the
operator
andredis
charts in the following order:helm upgrade operator apigee-operator/ \ --namespace APIGEE_NAMESPACE \ -f overrides.yaml
helm upgrade redis apigee-redis/ \ --namespace APIGEE_NAMESPACE \ -f overrides.yaml
- Within your
- Enable the external secret for the encryption keys
- Within your
overrides.yaml
file, add the following configuration to enable external secret usage for the org-level encryption keys:encryptionKeySecretProviderClass: apigee-orgencryptionkeys-spc
- Apply change by upgrading the
org
Helm chart:helm upgrade org apigee-org/ \ --namespace APIGEE_NAMESPACE \ -f overrides.yaml
- Within your
overrides.yaml
file for each environment, add the following configuration for the env-specific encryption keys:envs: - name: ENV_NAME encryptionKeySecretProviderClass: apigee-envencryptionkeys-spc
- Apply the change by upgrading the
env
Helm chart once for each environment:helm upgrade ENV_NAME apigee-env/ \ --namespace APIGEE_NAMESPACE \ --set env=ENV_NAME \ -f overrides.yaml
- Within your
Rollback
AX Hash Salt
- Within your
overrides.yaml
file, remove the configuration which enabled external secret usage for the AX Hash Salt:# Comment out or delete the following line: # axHashSaltSecretProviderClass: apigee-axhashsalt-spc
- Apply the change by upgrading the
org
Helm chart:helm upgrade org apigee-org/ \ --namespace APIGEE_NAMESPACE \ -f overrides.yaml
Redis password
- Within your
overrides.yaml
file, remove the configuration which enabled external secret usage for the Redis password:redis: auth: # Comment out or delete the following line: # secretProviderClass: apigee-redis-spc
- Then apply the changes by upgrading the
redis
andoperator
charts in the following order:helm upgrade redis apigee-redis/ \ --namespace APIGEE_NAMESPACE \ -f overrides.yaml
helm upgrade operator apigee-operator/ \ --namespace APIGEE_NAMESPACE \ -f overrides.yaml
Encryption keys
- Within your
overrides.yaml
file, remove the configuration which enabled external secret usage for the env encryption keys:envs: - name: ENV_NAME # Comment out or delete the following line: # encryptionKeySecretProviderClass: apigee-envencryptionkeys-spc
- Apply the change by upgrading the
env
Helm chart once for each environment:helm upgrade ENV_NAME apigee-env/ \ --namespace APIGEE_NAMESPACE \ --set env=ENV_NAME \ -f overrides.yaml
- Within your
overrides.yaml
file, remove the configuration which enabled external secret usage for the org encryption keys:# Comment out or delete the following line: # encryptionKeySecretProviderClass: apigee-orgencryptionkeys-spc
- And then Upgrade the
org
Helm chart:helm upgrade org apigee-org/ \ --namespace APIGEE_NAMESPACE \ -f overrides.yaml