Storing Cassandra secrets in Hashicorp Vault
This feature allows you to store Cassandra DB credentials for Apigee Hybrid in Hashicorp Vault, an external secret manager. External secret managers allow you to manage how secrets are stored in Kubernetes, including managing data residency and fine grained access controls.
Before Apigee hybrid version 1.10, the only way to supply passwords for Cassandra users was to specify the password in overrides.yaml. These passwords are stored in Kubernetes secrets. For example:
cassandra: auth: default: password: "********" admin: password: "********" ddl: password: "********" dml: password: "********" jmx: username: "jmxuser" password: "********" jolokia: username: "apigee" password: "********"
Using Hashicorp Vault, you can supply these passwords via the Kubernetes Secrets Store CSI Driver API
(SecretProviderClass
).
This allows Kubernetes to mount multiple secrets, keys, and certs stored in an external Vault.
Cassandra users and passwords
You will need to create secrets for the following Cassandra users. Change the default values to meet your organization's security policies.
Cassandra User | Default username | Default password |
---|---|---|
Admin | admin_user |
"********" |
DDL | ddl_user |
"********" |
Default | cassandra Note: The Default username must always be "cassandra" |
"********" |
DML | dml_user |
"********" |
JMX | "jmxuser" |
"********" |
Jolokia | "apigee" |
"********" |
See cassandra configuration property for more information.
Configure external secret integration
Setting up Vault integration for Apigee hybrid consists of the following procedures.
- In the first two procedures, you interact directly with Vault.
- In the third and fourth procedures, you apply the configurations to your Kubernetes cluster.
Use the following procedures to create the secrets in Vault and enable your hybrid installation to have access to them.
Create Vault secrets, policies, and roles
-
Verify that the current Kubernetes context is set to your cluster:
kubectl config current-context
-
Use the Vault API, CLI, or UI to create the cassandra secrets. The secret values you create
must match the Cassandra usernames and passwords currently used in your cluster.
-
Secret key: Any secret key (or combination of multiple keys) can be used, for example:
secret/data/apigee/cassandra
-
Secret data: Apigee Hybrid expects username and password pairs for the following Cassandra users:
Cassandra users Admin DDL Default DML JMX Jolokia -
Vault CLI: The following command shows how to create a single secret containing all the required usernames and passwords:
vault kv put secret/apigee/cassandra \ adminUsername="ADMIN_USERNAME" \ adminPassword="ADMIN_PASSWORD" \ ddlUsername="DDL_USERNAME" \ ddlPassword="DDL_PASSWORD" \ defaultUsername="cassandra" \ defaultPassword="DEFAULT_PASSWORD" \ dmlUsername="DML_USERNAME" \ dmlPassword="DML_PASSWORD" \ jmxUsername="JMX_USERNAME" \ jmxPassword="JMX_PASSWORD" \ jolokiaUsername="JOLOKIA_USERNAME" \ jolokiaPassword="JOLOKIA_PASSWORD"
The default usernames for each user are as follows:Cassandra user Default value Admin admin_user
DDL ddl_user
Default cassandra
DML dml_user
JMX jmxuser
Jolokia apigee
-
Secret key: Any secret key (or combination of multiple keys) can be used, for example:
-
Within Vault, create a policy to grant access to the secret you just created.
-
Create a policy file (suggested name:
apigee-cassandra-auth.txt
) with the following contents:path "secret/data/apigee/cassandra" { capabilities = ["read"] }
If you created multiple secrets, each secret must be added to the policy file:path "secret/data/apigee/cassandra/admin" { capabilities = ["read"] } path "secret/data/apigee/cassandra/ddl" { capabilities = ["read"] }
-
Apply the policy to Vault:
vault policy write apigee-cassandra-auth apigee-cassandra-auth.txt
It is possible to create the policy using standard input instead of reading from a file:
echo 'path "secret/data/apigee/cassandra" { capabilities = ["read"] }' | vault policy write apigee-cassandra-auth -
-
Create a policy file (suggested name:
-
Bind the policy to the Apigee Cassandra Kubernetes service accounts.
- Define the following environmental variables:
export ORG_NAME=APIGEE_ORG_NAME
export ENVS_LIST=LIST_OF_APIGEE-ENVS
export APIGEE_NAMESPACE=YOUR_APIGEE_NAMESPACE
Where:
- ORG_NAME is the name of your Apigee organization.
- ENVS_LIST Is a comma separated list of your Apigee environments, for
example
dev,prod
. - APIGEE_NAMESPACE is your Apigee namespace. The default is
apigee
.
- Create a script with the following contents. The script can have any name. In the
following example, the name of the script is
create-vault-cassandra-role.sh
:# create-vault-cassandra-role.sh ORG=ORG_NAME # ORG name ENVS=ENVS_LIST # comma separated env names, for example: dev,prod org_short_name=$(echo $ORG | head -c 15) encode=$(echo -n $ORG | shasum -a 256 | head -c 7) org_encode=$(echo "$org_short_name-$encode") names=apigee-manager,apigee-cassandra-default,apigee-cassandra-backup-sa,apigee-cassandra-restore-sa,apigee-cassandra-schema-setup-${org_encode},apigee-cassandra-schema-val-${org_encode},apigee-cassandra-user-setup-${org_encode},apigee-mart-${org_encode},apigee-mint-task-scheduler-${org_encode} for env in ${ENVS//,/ } do env_short_name=$(echo $env | head -c 15) encode=$(echo -n $ORG:$env | shasum -a 256 | head -c 7) env_encode=$(echo "$org_short_name-$env_short_name-$encode") names+=,apigee-synchronizer-${env_encode},apigee-runtime-${env_encode} done echo $names
- Run the script and assign the output to the SERVICE_ACCOUNT_NAMES variable.
This will create a comma-separated list of Kubernetes service account names.
export SERVICE_ACCOUNT_NAMES=$(./create-vault-cassandra-role)
Check that the variable was populated with the list:
echo $SERVICE_ACCOUNT_NAMES
-
Use the Vault CLI to create a role which binds the policy to Kubernetes service
accounts:
vault write auth/kubernetes/role/cassandra \ bound_service_account_names=${SERVICE_ACCOUNT_NAMES} \ bound_service_account_namespaces=${APIGEE_NAMESPACE} \ policies=apigee-cassandra-auth \ ttl=1m
- Define the following environmental variables:
Install CSI driver and Vault provider
Apigee hybrid v1.13.1 supports the following Helm chart versions:
Software | Version |
---|---|
Secrets Store CSI Driver | v1.3.4 |
Vault | v0.25.0 |
- Follow the Secrets Store CSI Driver installation instructions to Install the CSI driver on your cluster. The CSI driver has a Helm chart for installation.
- Follow the instructions in Installing the Vault CSI provider to install the Vault CSI provider if you have not installed it already.
Create SecretProviderClass object
The SecretProviderClass
resource tells the CSI driver what provider to
communicate with when requesting secrets. The Cassandra users' credentials must be configured
via this object. The following table shows the file names (objectName
s) expected
by Apigee Cassandra:
Cassandra User | Expected secret file names |
---|---|
Admin | adminUsername , adminPassword |
DDL | ddlUsername , ddlPassword |
Default | cassandra , defaultPassword |
DML | dmlUsername , dmlPassword |
JMX | jmxUsername , jmxPassword |
Jolokia | jolokiaUsername , jolokiaPassword |
- Create a YAML file for your
SecretProviderClass
. The file name can be anything, for example:spc.yaml
. Use the followingSecretProviderClass
template to configure this resource:apiVersion: secrets-store.csi.x-k8s.io/v1 kind: SecretProviderClass metadata: name: apigee-cassandra-auth-spc spec: provider: vault parameters: roleName: apigee-cassandra-auth # the roleName should match the vault role you created earlier in this procedure # vaultAddress is the endpoint your Vault server is running at. # If Vault is running in the same cluster as Apigee, the format will generally be: # http://vault.<namespace>.svc.cluster.local:<vaultServicePort> vaultAddress: VAULT_ADDRESS # "objectName" is an alias used within the SecretProviderClass to reference # that specific secret. This will also be the filename containing the secret. # Apigee Cassandra expects these exact values so they must not be changed. # "secretPath" is the path in Vault where the secret should be retrieved. # "secretKey" is the key within the Vault secret response to extract a value from. # For example, if the Vault secret is located at `secret/data/apigee/cassandra` # and you want to specify the admin password, you would use the following: # - objectName: "adminPassword" # secretPath: "secret/data/apigee/cassandra" # secretKey: "key within Vault secret specifying the admin password" objects: | - objectName: "adminUsername" secretPath: "" secretKey: "" - objectName: "adminPassword" secretPath: "" secretKey: "" - objectName: "defaultUsername" secretPath: "" secretKey: "" - objectName: "defaultPassword" secretPath: "" secretKey: "" - objectName: "ddlUsername" secretPath: "" secretKey: "" - objectName: "ddlPassword" secretPath: "" secretKey: "" - objectName: "dmlUsername" secretPath: "" secretKey: "" - objectName: "dmlPassword" secretPath: "" secretKey: "" - objectName: "jolokiaUsername" secretPath: "" secretKey: "" - objectName: "jolokiaPassword" secretPath: "" secretKey: "" - objectName: "jmxUsername" secretPath: "" secretKey: "" - objectName: "jmxPassword" secretPath: "" secretKey: ""
- Apply the
SecretProviderClass
to yourapigee
namespace:kubectl -n $APIGEE_NAMESPACE apply -f spc.yaml
Enable external secret for Cassandra
- Within your
overrides.yaml
, add the following configuration to enable external secret usage for Cassandra:cassandra: auth: secretProviderClass: apigee-cassandra-auth-spc # The name of the SecretProviderClass created in spc.yaml.
- Use
helm upgrade
to apply the change to theapigee-operator
andapigee-datastore
components:- The datastore controller in
apigee-operator
takes part in Cassandra decommissioning and data replication during region expansion. These tasks require the JMX and Jolokia credentials.helm upgrade operator apigee-operator/ \ --namespace $APIGEE_NAMESPACE> \ --atomic \ -f overrides.yaml
apigee-datastore
provides credentials that downstream components likeapigee-runtime
, Synchronizer & MART use when connecting to Cassandra.helm upgrade datastore apigee-datastore/ \ --namespace $APIGEE_NAMESPACE \ --atomic \ -f overrides.yaml
- The datastore controller in
-
Verify external secrets are being used. When external secrets are enabled, new
Volume
s,Volume Mount
s , andEnvironment Variable
s, are added referencing the secrets.- Verify the
apigee-controller-manager
deployment.Check that a
Volume
namedapigee-external-secrets
exists and references theSecretProviderClass
created above:kubectl -n $APIGEE_NAMESPACE get deployment apigee-controller-manager -o jsonpath='{.spec.template.spec.volumes[?(@.name=="apigee-external-secrets")]}' { "csi": { "driver": "secrets-store.csi.k8s.io", "readOnly": true, "volumeAttributes": { "secretProviderClass": "apigee-cassandra-auth-spc" } }, "name": "apigee-external-secrets" }
Check that a
VolumeMount
namedapigee-external-secrets
exists:kubectl -n $APIGEE_NAMESPACE get deployment apigee-controller-manager -o jsonpath='{.spec.template.spec.containers[?(@.name=="manager")].volumeMounts[?(@.name=="apigee-external-secrets")]}' { "mountPath": "/opt/apigee/externalsecrets", "name": "apigee-external-secrets", "readOnly": true }
Check that
Environment Variable
s exist that reference external secrets:kubectl -n $APIGEE_NAMESPACE get deployment apigee-controller-manager -o jsonpath='{.spec.template.spec.containers[?(@.name=="manager")].env}' [ ... { "name": "CASSANDRA_JOLOKIA_USERNAME_PATH", "value": "/opt/apigee/externalsecrets/jolokiaUsername" }, { "name": "CASSANDRA_JOLOKIA_PASSWORD_PATH", "value": "/opt/apigee/externalsecrets/jolokiaPassword" } ]
- Verify the
Rollback to K8s Secret
- To revert back to non-external secrets, remove the
secretProviderClass
configuration inoverrides.yaml
and use the previous configuration:cassandra: auth: secretProviderClass: apigee-cassandra-auth-spc # remove this line
- Use
helm upgrade
to apply the change to theapigee-operator
andapigee-datastore
components:helm upgrade operator apigee-operator/ \ --namespace $APIGEE_NAMESPACE \ --atomic \ -f overrides.yaml
helm upgrade datastore apigee-datastore/ \ --namespace $APIGEE_NAMESPACE \ --atomic \ -f overrides.yaml
Troubleshooting: Create a client container for debugging
If you are using Vault, this section replaces instructions in the troubleshooting section, Create a client container for debugging.
This section explains how to create a client container from which you can access
Cassandra debugging utilities
such as cqlsh
. These utilities allow you to query Cassandra tables and
can be useful for debugging purposes.
Create the client container
To create the client container, follow these steps:
- The container uses the TLS certificate from the
apigee-cassandra-user-setup
pod. The first step is to fetch this certificate name:kubectl get secrets -n APIGEE_NAMESPACE --field-selector type=kubernetes.io/tls | grep apigee-cassandra-user-setup | awk '{print $1}'
This command returns the certificate name. For example:
apigee-cassandra-user-setup-rg-hybrid-b7d3b9c-tls
. - Open a new file and paste the following pod spec into it:
apiVersion: v1 kind: Pod metadata: labels: name: CASSANDRA_CLIENT_NAME # For example: my-cassandra-client namespace: $APIGEE_NAMESPACE spec: containers: - name: CASSANDRA_CLIENT_NAME image: "gcr.io/apigee-release/hybrid/apigee-hybrid-cassandra-client:1.13.1" imagePullPolicy: Always command: - sleep - "3600" env: - name: CASSANDRA_SEEDS value: apigee-cassandra-default.apigee.svc.cluster.local - name: APIGEE_DML_USERNAME_PATH value: /opt/apigee/externalsecrets/dmlUsername - name: APIGEE_DML_PASSWORD_PATH value: /opt/apigee/externalsecrets/dmlPassword volumeMounts: - mountPath: /opt/apigee/ssl name: tls-volume readOnly: true - name: apigee-external-secrets mountPath: /opt/apigee/externalsecrets readOnly: true volumes: - name: tls-volume secret: defaultMode: 420 secretName: apigee-cassandra-user-setup-vaibhavhybridor-8b3e61d-tls - name: apigee-external-secrets csi: driver: secrets-store.csi.k8s.io readOnly: true volumeAttributes: secretProviderClass: apigee-cass-password serviceAccount: apigee-cassandra-default serviceAccountName: apigee-cassandra-default restartPolicy: Never
- Save the file with a
.yaml
extension. For example:my-spec.yaml
. - Apply the spec to your cluster:
kubectl apply -f my-spec.yaml -n $APIGEE_NAMESPACE
- Log in to the container:
kubectl exec -n CASSANDRA_CLIENT_NAME -it -- bash
- Connect to the Cassandra
cqlsh
interface with the following commands. Enter the commands exactly as shown:APIGEE_DML_USER=$(cat "$APIGEE_DML_USERNAME_PATH")
export APIGEE_DML_USER
APIGEE_DML_PASSWORD=$(cat "$APIGEE_DML_PASSNAME_PATH")
export APIGEE_DML_PASSWORD
cqlsh ${CASSANDRA_SEEDS} -u ${APIGEE_DML_USER} -p ${APIGEE_DML_PASSWORD} --ssl
Deleting the client pod
Use this command to delete the Cassandra client pod:
kubectl delete pods -n $APIGEE_NAMESPACE cassandra-client