Rotating service account keys

This page describes how to rotate keys for the following service accounts:

To rotate your service account keys:

  1. Create a directory to store a backup of your current secrets:

    mkdir backup

  2. Note the following information for the relevant service account:

    Component access

    Cluster Secret Namespace
    Admin admin-cluster-creds kube-system
    Admin user-cluster-creds CLUSTER_NAME-gke-onprem-mgmt
    Admin private-registry-creds kube-system
    User private-registry-creds kube-system
    • If you aren't using a private registry, the private-registry-creds Secret holds the key for your component access service account.
    • If you are using a private registry, the private-registry-creds Secret holds the credentials for your private registry, not the component access service account key.

    Connect-register

    Cluster Secret Namespace
    Admin admin-cluster-creds kube-system
    Admin user-cluster-creds CLUSTER_NAME-gke-onprem-mgmt

    Logging-monitoring

    Cluster Secret Namespace
    Admin admin-cluster-creds kube-system
    Admin user-cluster-creds CLUSTER_NAME-gke-onprem-mgmt
    User google-cloud-credentials kube-system
    User stackdriver-service-account-key knative-serving

    Audit logging

    Cluster Secret Namespace
    Admin admin-cluster-creds kube-system
    Admin user-cluster-creds CLUSTER_NAME-gke-onprem-mgmt
    Admin kube-apiserver CLUSTER_NAME

    Usage Metering

    Cluster Secret Namespace
    Admin user-cluster-creds CLUSTER_NAME-gke-onprem-mgmt
    User usage-metering-bigquery-service-account-key kube-system

    Stackdriver

    Cluster Secret Namespace
    Admin admin-cluster-creds kube-system
    Admin user-cluster-creds CLUSTER_NAME-gke-onprem-mgmt
    User google-cloud-credentials kube-system
    User stackdriver-service-account-key knative-serving

  3. Create a backup of each secret using the following command:

    kubectl get secret SECRET --namespace NAMESPACE \
    --kubeconfig KUBECONFIG -o json > backup/SECRET-NAMESPACE.json

    Replace the following:

    • NAMESPACE: the namespace where the secret is located. For example, kube-system.
    • KUBECONFIG: the path to the kubeconfig file for the admin or user cluster.
    • SECRET: the name of the secret. For example, admin-cluster-creds.

    For example, run the following commands for the audit logging service account:

    kubectl get secret admin-cluster-creds --namespace kube-system \
        --kubeconfig KUBECONFIG -o json > backup/admin-cluster-creds-kube-system.json

kubectl get secret user-cluster-creds --namespace NAMESPACE \
        --kubeconfig KUBECONFIG -o json > backup/user-cluster-creds-NAMESPACE.json

kubectl get secret kube-apiserver --namespace NAMESPACE \
        --kubeconfig KUBECONFIG -o json > backup/kube-apiserver-NAMESPACE.json

  4. To create a new service account key file, run the following command:

    gcloud iam service-accounts keys create NEW_KEY_FILE --iam-account IAM_ACCOUNT

    Replace the following:

    • NEW_KEY_FILE: the name for your new service account key file
    • IAM_ACCOUNT: the email address of the service account

  5. In the admin cluster configuration file, find the componentAccessServiceAccountKeyPath field, the gkeConnect section, the stackdriver section, and the cloudAuditLogging section. In those places, replace the paths to the service account key files.

  6. In the user cluster configuration file, find the componentAccessServiceAccountKeyPath field, the gkeConnect section, the stackdriver section, the cloudAudigLogging section, and the usageMetering section. In those places, replace the paths to the service account key files.

  7. Save the changes you made by running the following commands. You can rotate the keys for one component at a time, or you can rotate all keys at once by setting the component to sakeys:

    gkectl update credentials COMPONENT \
    --kubeconfig ADMIN_CLUSTER_KUBECONFIG \
    --config ADMIN_CLUSTER_CONFIG \
    --admin-cluster

gkectl update credentials COMPONENT \
    --kubeconfig ADMIN_CLUSTER_KUBECONFIG \
    --config USER_CLUSTER_CONFIG

    Replace the following;

    • COMPONENT: one of:

      • componentaccess
      • register
      • cloudauditlogging
      • usagemetering
      • stackdriver
      • sakeys (Rotate keys for multiple components.)

    • ADMIN_CLUSTER_KUBECONFIG: the path to the kubeconfig file for the admin cluster.

    • ADMIN_CLUSTER_CONFIG: the path to the admin cluster configuration file.

    • USER_CLUSTER_CONFIG: the path to the user cluster configuration file.

Node re-creation

Some service account key rotations may take longer time because node re-creation is required:

Service account Nodes re-creation required
Component access If using Container Registry: Yes
If using a private registry: No
Audit logging Admin cluster: Yes but only control-plane nodes
User cluster using kubeception: No
User cluster using Controlplane V2: Yes but only control plane nodes
Logging-monitoring No
Connect-register No
Usage metering No

For a key rotation that requires nodes to be re-created, the nodes are replaced in a rolling update process; that is, the nodes are re-created one by one.

The possible downtime during a key rotation is similar to the downtime for a cluster upgrade. For details, see Downtime during upgrades.

Restoring backups

If you need to restore the backups of the secrets you made earlier, run the following command:

kubectl apply -f backup/