Configure prepared credentials for user clusters

This document shows how to configure prepared credentials for user clusters in Google Distributed Cloud.

With prepared credentials, you can store credentials for your user clusters in Secrets in your admin cluster. This provides an element of security, because you don't have to keep passwords and service account keys on your admin workstation as you create your user clusters.

You prepare Secrets in the admin cluster ahead of time. Then as you create user clusters, you can specify that certain credentials should be taken from the prepared Secrets in the admin cluster. You can also use the prepared Secrets when you rotate credentials in a user cluster.

Before you begin

Create an admin cluster if you don't already have one.

Overview of the procedure

  1. Fill in a Secrets configuration file.

  2. In your admin cluster, create groups of Secrets. Each Secret group is in its own Kubernetes namespace.

  3. Create a user cluster. In the user cluster configuration file, indicate that you want credentials to be taken from Secrets in a particular namespace in the admin cluster.

  4. Create additional Secret groups and additional versions of your Secrets as needed.

  5. Update credentials for an existing user cluster as needed.

  6. Create additional user clusters as desired. In each user cluster configuration file, specify a Secrets namespace. You can also specify which version of a Secret you want to use for a particular credential.

Fill in your secrets configuration file

Generate a template for a Secrets configuration file:

gkectl create-config secrets

The preceding command generates a file named secrets.yaml. You can change the name and location of this file if you like.

Familiarize yourself with the configuration file by reading the Secrets configuration file document. You might want to keep this document open in a separate tab or window.

In your Secrets configuration file, fill in the values that are relevant for your situation. You must fill in a value for namespace that begins with gke-onprem-secrets-.

Here's an example of a Secrets configuration file that has one Secret group. The group has values for vCenter credentials and three service account keys:

apiVersion: v1
kind: ClusterSecrets
secretGroups:
- namespace: "gke-onprem-secrets-user-cluster-1"
  secrets
    vCenter:
      username: "my-vcenter-account"
      password: "U$icUKEW#INE"
    registerServiceAccount:
      serviceAccountKeyPath: "my-key-folder/connect-register-key.json"
    stackdriverServiceAccount:
      serviceAccountKeyPath: "my-key-folder/log-mon-key.json"
    cloudAuditLoggingServiceAccount:
      serviceAccountKeyPath: "my-key-folder/audit-log-key.json"

Create prepared Secrets

Create prepared Secrets in your admin cluster:

gkectl prepare secrets --kubeconfig ADMIN_CLUSTER_KUBECONFIG --secret-config SECRETS_CONFIG

Replace the following:

  • ADMIN_CLUSTER_KUBECONFIG: the path of the admin cluster kubeconfig file

  • SECRETS_CONFIG: the path of your Secrets configuration file

View your prepared Secrets

List the prepared Secrets in the admin cluster:

gkectl list secrets --kubeconfig ADMIN_CLUSTER_KUBECONFIG

Example output:

The following secrets have been found:
- namespace: gke-onprem-secrets-user-cluster-1
  - secrets with name prefix: cloud-audit-logging-service-account-creds
    name: cloud-audit-logging-service-account-creds.1, version: 1, age: 58s
  - secrets with name prefix: register-service-account-creds
    name: register-service-account-creds.1, version: 1, age: 58s
  - secrets with name prefix: stackdriver-service-account-creds
    name: stackdriver-service-account-creds.1, version: 1, age: 58s
  - secrets with name prefix: vsphere-creds
    name: vsphere-creds.1, version: 1, age: 58s

You can also run kubectl get secrets to list the Secrets in a namespace. For example:

kubectl --kubeconfig ADMIN_CLUSTER_KUBECONFIG get secrets --namespace gke-onprem-secrets-user-cluster-1

Example output:

cloud-audit-logging-service-account-creds ...
register-service-account-creds.1 ...
stackdriver-service-account-creds.1 ...
vsphere-creds.1 ...

In the preceding output, you can see that each Secret name has an extension that indicates the version of the Secret. In this example, all the Secrets have a version of 1.

Create a user cluster

Follow the instructions in Create a user cluster.

As you fill in your user cluster configuration file, enter a value for preparedSecrets.namespace. This value must match a namespace that you specified previously in a Secrets configuration file.

Example:

preparedSecrets:
  namespace: "gke-onprem-secrets-user-cluster-1"

In your user cluster configuration file, do not specify values for the following fields. These fields are not needed, because Google Distributed Cloud gets the credentials and keys from your prepared Secrets.

  • vCenter.credentials.fileRef.path
  • loadBalancer.f5BigIP.credentials.fileRef.path
  • gkeConnect.registerServiceAccountKeyPath
  • stackdriver.serviceAccountKeyPath
  • usageMetering.bigQueryServiceAccountKeyPath
  • cloudAuditLogging.serviceAccountKeyPath
  • authentication.sni.keyPath

Finish creating your user cluster as described in Create a user cluster.

Create additional prepared Secrets

This section shows how to create version 2 of some of the Secrets in an existing namespace.

Create a new Secrets configuration file named secrets-2.yaml. Specify an existing namespace, and provide credentials for selected Secrets.

Example:

apiVersion: v1
kind: ClusterSecrets
secretGroups:
- namespace: "gke-onprem-secrets-user-cluster-1"
  secrets:
    stackdriverServiceAccount:
      serviceAccountKeyPath: "log-mon-sa-2.json"
    cloudAuditLoggingServiceAccount:
      serviceAccountKeyPath: "audit-log-sa-2.json"

The preceding example provides key paths for the following Secrets in the gke-onprem-secrets-user-cluster-1 namespace.

  • Version 2 of the stackdriver-service-account-creds Secret
  • Version 2 of the cloud-audit-logging-service-account-creds Secret

Create the new Secrets:

gkectl prepare secrets --kubeconfig ADMIN_CLUSTER_KUBECONFIG --secret-config secrets-2.yaml

List the prepared Secrets in the admin cluster:

gkectl --kubeconfig ADMIN_CLUSTER_KUBECONFIG list secrets

Example output:

The following secrets have been found:
- namespace: gke-onprem-secrets-user-cluster-1
  - secrets with name prefix: cloud-audit-logging-service-account-creds
    name: cloud-audit-logging-service-account-creds.1, version: 1, age: 11h
    name: cloud-audit-logging-service-account-creds.2, version: 2, age: 33m
  - secrets with name prefix: register-service-account-creds
    name: register-service-account-creds.1, version: 1, age: 11h
  - secrets with name prefix: stackdriver-service-account-creds
    name: stackdriver-service-account-creds.1, version: 1, age: 11h
    name: stackdriver-service-account-creds.2, version: 2, age: 33m
  - secrets with name prefix: vsphere-creds
    name: vsphere-creds.1, version: 1, age: 11h

In the preceding output, you can see that there are two versions of the stackdriver-service-account-creds Secret and two versions of the cloud-audit-logging-service-account-creds Secret.

Rotate credentials for a user cluster

This section shows how to rotate selected credentials for an existing user cluster.

Before rotating credentials, check the current Secret versions used in the cluster:

gkectl list secrets cluster --cluster-name USER_CLUSTER_NAME  kubeconfig ADMIN_CLUSTER_KUBECONFIG

Example output:

The following prepared secrets have been used for cluster "user-cluster-1":
- namespace: gke-onprem-secrets-user-cluster-1
  secret: vsphere-creds.1, version: 1
  secret: f5-creds.1, version: 1
  secret: register-service-account-creds.1, version: 1
  secret: stackdriver-service-account-creds.1, version: 1
  secret: cloud-audit-logging-service-account-creds.1, version: 1

Copy your user cluster configuration file to a file named user-cluster-update.yaml.

In user-cluster-update.yaml, add serviceAccountKey sections. For example, the following example has serviceAccountKey sections under stackdriver and cloudAuditLogging:

stackdriver:
  projectID: "my-project-123"
  clusterLocation: "us-central1"
  serviceAccountKey:
    secretRef:
      version: "2"
cloudAuditLogging:
  projectID: "my-project-123"
  clusterLocation: "us-central-1"
  serviceAccountKey:
    secretRef:
      version: "latest"

The preceding example specifies that when the user cluster is updated, it will use:

  • Version 2 of the stackdriver-service-account-creds Secret

  • The latest version of the cloud-audit-logging-service-account-creds Secret. In this example, that is version 2.

When you update a Secret, the value that you provide for secretRef.version must be "latest" or an integer string. You cannot omit secretRef.version, and you cannot set it to the empty string.

Update the credentials for the user cluster:

gkectl update credentials stackdriver --kubeconfig ADMIN_CLUSTER_KUBECONFIG --config user-cluster-2.yaml

gkectl update credentials cloudauditlogging --kubeconfig ADMIN_CLUSTER_KUBECONFIG --config user-cluster-2.yaml

Now the user cluster uses the following prepared Secrets:

  • Version 1 of vsphere-creds
  • Version 1 of register-service-account-creds
  • Version 2 of stackdriver-service-account-creds
  • Version 2 of cloud-audit-logging-service-account-creds

Create additional Secrets and user clusters

If you plan to create additional user clusters, think about how you want to organize your prepared Secrets. You might want to create a separate namespace in your admin cluster for each user cluster. Or you might want to share the same prepared Secret namespace for several or all user clusters.

For example, suppose Alice, Bob, and Carol will each have a user cluster. You could create three Secret groups as shown in this example:

apiVersion: v1
kind: ClusterSecrets
secretGroups:
- namespace: "gke-onprem-secrets-alice"
  secrets:
    vCenter:
      username: "alice"
      password: "zC7r^URDPq2t"
    registerServiceAccount:
      serviceAccountKeyPath: "register-sa-a.json"
    stackdriverServiceAccount:
      serviceAccountKeyPath: "log-mon-sa-a.json"
    cloudAuditLoggingServiceAccount:
      serviceAccountKeyPath: "audit-log-sa-a.json"
- namespace: "gke-onprem-secrets-bob"
  secrets:
    registerServiceAccount:
      serviceAccountKeyPath: "register-sa-b.json"
    stackdriverServiceAccount:
      serviceAccountKeyPath: "log-mon-sa-b.json"
    cloudAuditLoggingServiceAccount:
      serviceAccountKeyPath: "audit-log-sa-b.json"
- namespace: "gke-onprem-secrets-carol"
  secrets:
    registerServiceAccount:
      serviceAccountKeyPath: "register-sa-c.json"
    stackdriverServiceAccount:
      serviceAccountKeyPath: "log-mon-sa-c.json"
    cloudAuditLoggingServiceAccount:
      serviceAccountKeyPath: "audit-log-sa-c.json"

Over time you could create additional versions of the Secrets in each Secret group.

In your user cluster configuration files, provide values for serviceAccountKey.secretRef.version to specify which versions of your Secrets you want to use. You can set the value to "latest", the empty string, or an integer string.

For example, suppose all your Secrets have versions 1, 2, and 3. And suppose this is a portion of the user cluster configuration file for Alice.

apiVersion: v1
kind: UserCluster
name: "user-cluster-alice"
preparedSecrets:
  namespace: "gke-onprem-secrets-alice"
...
vCenter:
  credentials:
gkeConnect:
  projectID: "project-a"
  serviceAccountKey:
    secretRef:
      version: "2"
stackdriver:
  projectID: "project-a"
  clusterLocation: "us-central1"
  serviceAccountKey:
    secretRef:
      version: "latest"
cloudAuditLogging:
  projectID: "project-a"
  clusterLocation: "us-central-1"
  serviceAccountKey:
    secretRef:
      version: ""

In the preceding example, we can see:

  • There is no secretRef specified for vCenter, so the cluster will use the latest version of the vsphere-creds Secret in the gke-onprem-secrets-alice namespace.

  • The cluster will use version 2 of the register-service-account-creds Secret in the gke-onprem-secrets-alice namespace.

  • The cluster will use the latest version of the stackdriver-service-account-creds Secret in the gke-onprem-secrets-alice namespace. In this example, that is version 3.

  • The version for cloudAuditLogging is the empty string, so the cluster will use the latest version of the cloud-audit-logging-service-account-creds Secret in the gke-onprem-secrets-alice namespace. In this example, that is version 3.