Troubleshooting Cassandra credential rotation

You're viewing Apigee and Apigee hybrid documentation.
There is no equivalent Apigee Edge documentation for this topic.

Symptom

During multi-region Cassandra credential rotation in Apigee hybrid, after the first region is rotated, rotations in subsequent regions fail and errors are logged in the secret rotation pod logs.

Error message

You see the following in the logs:

failed to run secret rotation: failed to create new users: failed to create new users

Workaround for Known Issue 397693324

Diagnosis

The SecretRotation resource indicates a failure in the Status field: 

Status:
  Message:  initiated automated rollback
  State:    error

The secret rotation job pod logs contain the following error: 

failed to run secret rotation: failed to create new users: failed to create new users

The create-new-users-job pod logs contain the following error: 

Error creating clients with updated password: gocql: unable to create session: unable to discover protocol version: Provided username cassandra and/or password are incorrect

Resolution

Perform the following steps.

  1. In every region except the first, update the default Cassandra user (cassandra) password to the new rotated value in the old Secret. 

    apiVersion: v1
    kind: Secret
    metadata:
      name: OLD_SECRET_NAME   # oldSecretRef
      namespace: APIGEE_NAMESPACE
    type: Opaque
    data:
      default.password: NEW_DEFAULT_PASSWORD   #base64-encoded string
      admin.user: OLD_ADMIN_USERNAME   #base64-encoded string
      admin.password: OLD_ADMIN_PASSWORD   #base64-encoded string
      dml.user: OLD_DML_USERNAME   #base64-encoded string
      dml.password: OLD_DML_PASSWORD   #base64-encoded string
      ddl.user: OLD_DDL_USERNAME   #base64-encoded string
      ddl.password: OLD_DDL_PASSWORD   #base64-encoded string
      jmx.user: OLD_JMX_USERNAME   #base64-encoded string
      jmx.password: OLD_JMX_PASSWORD   #base64-encoded string
      jolokia.user: OLD_JOLOKIA_USERNAME   #base64-encoded string
      jolokia.password: OLD_JOLOKIA_PASSWORD   #base64-encoded string

  2. Apply the updated Secret: 

    kubectl apply -f OLD_SECRET_FILE

  3. Continue with the normal rotation process and it should succeed.

Must gather diagnostic information

If the problem persists even after following the above instructions, gather the following diagnostic information and then contact Google Cloud Customer Care:

  • In addition to the usual data you might be asked to provide, collect the logs from all the secret rotation pods.