Stay organized with collections
Save and categorize content based on your preferences.
You're viewing Apigee and Apigee hybrid documentation.
There is no equivalent
Apigee Edge documentation for this topic.
Symptom
During multi-region Cassandra credential rotation in Apigee hybrid, after the first region is
rotated, rotations in subsequent regions fail and errors are logged in the secret rotation pod logs.
Error message
You see the following in the logs:
failed to run secret rotation: failed to create new users: failed to create new users
Workaround for Known Issue 397693324
Diagnosis
The SecretRotation resource indicates a failure in the Status field:
Continue with the normal rotation process and it should succeed.
Must gather diagnostic information
If the problem persists even after following the above instructions,
gather the following diagnostic information and then contact Google Cloud Customer Care:
In addition to the usual data you might be asked to provide, collect the
logs from all the secret rotation pods.
[[["Easy to understand","easyToUnderstand","thumb-up"],["Solved my problem","solvedMyProblem","thumb-up"],["Other","otherUp","thumb-up"]],[["Hard to understand","hardToUnderstand","thumb-down"],["Incorrect information or sample code","incorrectInformationOrSampleCode","thumb-down"],["Missing the information/samples I need","missingTheInformationSamplesINeed","thumb-down"],["Other","otherDown","thumb-down"]],["Last updated 2025-09-04 UTC."],[[["\u003cp\u003eThis document addresses a specific issue encountered during multi-region Cassandra credential rotation in Apigee hybrid, exclusively.\u003c/p\u003e\n"],["\u003cp\u003eThe problem occurs after the first region's rotation, causing subsequent region rotations to fail, indicated by an error message: \u003ccode\u003efailed to run secret rotation: failed to create new users: failed to create new users\u003c/code\u003e.\u003c/p\u003e\n"],["\u003cp\u003eThe root cause is identified by error messages in both \u003ccode\u003eSecretRotation\u003c/code\u003e resource status and \u003ccode\u003ecreate-new-users-job\u003c/code\u003e pod logs, signaling incorrect username and/or password issues.\u003c/p\u003e\n"],["\u003cp\u003eThe resolution involves updating the default Cassandra user's password in the old \u003ccode\u003eSecret\u003c/code\u003e to match the new rotated value in every region except the first, and then reapplying that updated secret.\u003c/p\u003e\n"],["\u003cp\u003eIf the problem persists after the resolution steps, users should gather logs from all secret rotation pods and contact Google Cloud Customer Care for further assistance.\u003c/p\u003e\n"]]],[],null,["# Troubleshooting Cassandra credential rotation\n\n*You're viewing **Apigee** and **Apigee hybrid** documentation.\nThere is no equivalent\n[Apigee Edge](https://docs.apigee.com/api-platform/get-started/get-started) documentation for this topic.*\n| **Note:** This document is applicable for Apigee hybrid users only.\n\nSymptom\n-------\n\n\nDuring multi-region Cassandra credential rotation in Apigee hybrid, after the first region is\nrotated, rotations in subsequent regions fail and errors are logged in the secret rotation pod logs.\n\nError message\n-------------\n\nYou see the following in the logs: \n\n```text\nfailed to run secret rotation: failed to create new users: failed to create new users\n```\n\nWorkaround for Known Issue 397693324\n------------------------------------\n\n### Diagnosis\n\n\nThe `SecretRotation` resource indicates a failure in the `Status` field: \n\n```actionscript-3\nStatus:\n Message: initiated automated rollback\n State: error\n```\n\n\nThe secret rotation job pod logs contain the following error: \n\n```text\nfailed to run secret rotation: failed to create new users: failed to create new users\n```\n\n\nThe `create-new-users-job` pod logs contain the following error: \n\n```text\nError creating clients with updated password: gocql: unable to create session: unable to discover protocol version: Provided username cassandra and/or password are incorrect\n```\n\n### Resolution\n\n\nPerform the following steps.\n\n1.\n In every region except the first, update the default Cassandra user (`cassandra`)\n password to the **new** rotated value in the **old** `Secret`.\n\n ```actionscript-3\n apiVersion: v1\n kind: Secret\n metadata:\n name: OLD_SECRET_NAME # oldSecretRef\n namespace: APIGEE_NAMESPACE\n type: Opaque\n data:\n default.password: NEW_DEFAULT_PASSWORD #base64-encoded string\n admin.user: OLD_ADMIN_USERNAME #base64-encoded string\n admin.password: OLD_ADMIN_PASSWORD #base64-encoded string\n dml.user: OLD_DML_USERNAME #base64-encoded string\n dml.password: OLD_DML_PASSWORD #base64-encoded string\n ddl.user: OLD_DDL_USERNAME #base64-encoded string\n ddl.password: OLD_DDL_PASSWORD #base64-encoded string\n jmx.user: OLD_JMX_USERNAME #base64-encoded string\n jmx.password: OLD_JMX_PASSWORD #base64-encoded string\n jolokia.user: OLD_JOLOKIA_USERNAME #base64-encoded string\n jolokia.password: OLD_JOLOKIA_PASSWORD #base64-encoded string\n \n ```\n2.\n Apply the updated `Secret`:\n\n ```scdoc\n kubectl apply -f OLD_SECRET_FILE\n ```\n3.\n Continue with the normal rotation process and it should succeed.\n\nMust gather diagnostic information\n----------------------------------\n\n\nIf the problem persists even after following the above instructions,\ngather the following diagnostic information and then contact [Google Cloud Customer Care](https://cloud.google.com/support-hub/):\n\n-\n In addition to the usual data you might be asked to provide, collect the\n logs from all the secret rotation pods."]]
