Run your own certificate authorities and keys in GKE

Autopilot Standard

---

Note: This document uses a general availability feature named GKE control plane authority that's only available for eligible Google Cloud customers. To request access to this feature, contact your Google Cloud account team.

This page shows you how to configure your Google Kubernetes Engine (GKE) cluster control plane with certificate authorities (CAs) and keys that you manage. This guidance is intended for security administrators who have specific organizational compliance or policy requirements for control over credential issuance and signing.

This page describes one part of a set of optional control plane features in GKE that lets you perform tasks like verifying your control plane security posture or configuring encryption and credential signing in the control plane using keys that you manage. For details, see About GKE control plane authority.

By default, Google Cloud applies various security measures to the managed control plane. This page describes optional capabilities that give you more visibility or control over the GKE control plane.

You should already be familiar with the following concepts:

• Certificate Signing Requests (CSRs)
• Certificate authorities (CAs)
• etcd
• GKE cluster architecture

Control plane credential components

GKE clusters use specific CAs and keys to issue credentials in the cluster, like X.509 certificates or ServiceAccount tokens. You can create keys in Cloud Key Management Service (Cloud KMS) and CAs in Certificate Authority Service (CA Service) and configure your clusters to use these resources instead of Google Cloud-managed CAs and keys.

To learn more about the specific components that you create, see Self-managed CAs and keys.

Usage with other GKE control plane authority features

GKE control plane authority provides the following features related to self-managed keys:

• Run your own certificate authorities (CAs) and keys (this page)
• Encrypt control plane components

Note: You can only enable these features when you create a new GKE cluster. You can't update existing clusters to use these features. To use both of these features in the same cluster, perform all of the key and CA configuration procedures in both guides, and then run the cluster creation command that enables both sets of features as described in the Create a cluster section.

Objectives

• Create keys in Cloud KMS
• Create CAs in CA Service
• Grant Identity and Access Management (IAM) roles to the GKE service agent
• Create a GKE cluster that uses your CAs and keys
• Verify that the cluster uses your CAs and keys

Costs

In this document, you use the following billable components of Google Cloud:

• GKE
• Cloud KMS
• CA Service
• Cloud Logging

To generate a cost estimate based on your projected usage, use the pricing calculator. New Google Cloud users might be eligible for a free trial.

When you finish the tasks that are described in this document, you can avoid continued billing by deleting the resources that you created. For more information, see Clean up.

Before you begin

1. Sign in to your Google Cloud account. If you're new to Google Cloud, create an account to evaluate how our products perform in real-world scenarios. New customers also get $300 in free credits to run, test, and deploy workloads.
2. Install the Google Cloud CLI.

3. To initialize the gcloud CLI, run the following command:

   gcloud init

   Note: If you installed the gcloud CLI previously, make sure you have the latest version by running gcloud components update.

4. Make sure that billing is enabled for your Google Cloud project.

5. Enable the Kubernetes Engine, Certificate Authority Service, and Cloud Key Management Service APIs:

   gcloud services enable container.googleapis.com privateca.googleapis.com cloudkms.googleapis.com

6. Install the Google Cloud CLI.

7. To initialize the gcloud CLI, run the following command:

   gcloud init

   Note: If you installed the gcloud CLI previously, make sure you have the latest version by running gcloud components update.

8. Make sure that billing is enabled for your Google Cloud project.

9. Enable the Kubernetes Engine, Certificate Authority Service, and Cloud Key Management Service APIs:

   gcloud services enable container.googleapis.com privateca.googleapis.com cloudkms.googleapis.com

10. Ensure that your environment is eligible to use GKE control plane authority features. To opt-in to these features, contact your Google Cloud sales team.
11. To reliably track credential issuance and usage, ensure that the following Data Access audit logs are enabled:
    • Cloud KMS: DATA_READ
    • CA Service: ADMIN_READ and ADMIN_WRITE

    To enable these log types, see Enable Data Access audit logs.

Required roles and permissions

To get the permissions that you need to run your own CAs and keys, ask your administrator to grant you the following IAM roles:

• Create Cloud KMS keys: Cloud KMS Admin (roles/cloudkms.admin) on your key project
• Create CA pools and root CAs: CA Service Admin (roles/privateca.admin) on your key project
• Create GKE clusters: Kubernetes Engine Cluster Admin (roles/container.clusterAdmin) on your cluster project

For more information about granting roles, see Manage access to projects, folders, and organizations.

You might also be able to get the required permissions through custom roles or other predefined roles.

Requirements

You must use GKE version 1.31.1-gke.1846000 or later.

Limitations

The following limitations apply:

• You can only use keys from Cloud KMS. You can't use another KMS provider or another encryption provider.
• Cloud External Key Manager (Cloud EKM) keys aren't supported.
• You can only use CAs from CA Service.

Prepare the environment

In this section, you identify Google Cloud projects that you'll use in this tutorial, and you create a key ring in Cloud KMS to hold your keys.

Identify projects

We recommend that you use separate Google Cloud projects as follows:

• Key project: contains all keys and CAs.
• Cluster project: contains your GKE clusters.

You can optionally use the same project for your keys, CAs, and GKE clusters, but we recommend that you use separate projects so that the teams that manage cryptographic operations in your organization are separated from the teams that manage cluster operations.

Create a key ring

Create a key ring in the key project to hold all of the keys for a specific cluster. You must create the key ring in the same location as your GKE cluster.

Note: You can't delete a key ring after creation. Key rings don't incur additional costs. However, having too many key rings in a project might make managing your resources in Cloud KMS more tedious.

Run the following command:

gcloud kms keyrings create KEY_RING_NAME \
    --location=us-central1 \
    --project=KEY_PROJECT_ID

Replace the following:

• KEY_RING_NAME: a name for your key ring.
• KEY_PROJECT_ID: the project ID of your key project.

Create keys

For each of the credential authorities like service account keys and CAs, you create a key using Cloud KMS. This section shows you how to create the keys that GKE uses to sign and verify credentials in the cluster. You can specify your own properties for these keys depending on your organization's needs. For details, see the Create a key page and the projects.locations.keyRings.cryptoKeys API reference.

Caution: Don't enable automatic key rotation with Cloud KMS. Rotating your encryption keys requires additional steps to ensure that your control plane resources are re-encrypted with new key versions before the original key versions are destroyed.

Consider the following when you create these resources in Cloud KMS:

• If you have a existing key ring in your key project, you can use that key ring to store all of the keys that you create for use with your cluster.
• Your key ring must be in the same Google Cloud location as your cluster to minimize latency.
• Keys must specify asymmetric-signing as the key purpose.
• Use the following algorithms based on the type of key:
  • ServiceAccount signing keys: a strong RSA signing PKCS1 algorithm, like rsa-sign-pkcs1-4096-sha256 or rsa-sign-pkcs1-3072-sha256.
  • Certificate authority keys: a strong algorithm like ec-sign-p256-sha256.
• Cloud HSM hardware keys are supported, but the software protection level is sufficient for most use cases. For details about hardware keys, see Cloud HSM.
• Don't modify the default duration for key destruction.
• GKE doesn't prevent you from deleting Cloud KMS keys, including CA Service keys, that are in use by the cluster. Before you delete keys or CAs, ensure that the resources are unused.

To create your keys, run the following commands:

1. Create the Kubernetes ServiceAccount signing key, which you also specify as the service account verification key during cluster creation:

   gcloud kms keys create sa-signing-key \
       --keyring=KEY_RING_NAME \
       --location=us-central1\
       --purpose="asymmetric-signing" \
       --protection-level=hsm \
       --default-algorithm=rsa-sign-pkcs1-4096-sha256 \
       --project=KEY_PROJECT_ID
   

   Replace KEY_PROJECT_ID with the project ID for your dedicated key project.

2. Create the cluster root CA key:

   gcloud kms keys create cluster-ca-key \
       --keyring=KEY_RING_NAME \
       --location=us-central1\
       --purpose="asymmetric-signing" \
       --protection-level=hsm \
       --default-algorithm=ec-sign-p256-sha256 \
       --project=KEY_PROJECT_ID
   

3. Create the etcd peer root CA key:

   gcloud kms keys create etcd-peer-ca-key \
       --keyring=KEY_RING_NAME \
       --location=us-central1\
       --purpose="asymmetric-signing" \
       --protection-level=hsm \
       --default-algorithm=ec-sign-p256-sha256 \
       --project=KEY_PROJECT_ID
   

4. Create the etcd API root CA key:

   gcloud kms keys create etcd-api-ca-key \
       --keyring=KEY_RING_NAME \
       --location=us-central1\
       --purpose="asymmetric-signing" \
       --protection-level=hsm \
       --default-algorithm=ec-sign-p256-sha256 \
       --project=KEY_PROJECT_ID
   

5. Create the aggregation root CA key:

   gcloud kms keys create aggregation-ca-key \
       --keyring=KEY_RING_NAME \
       --location=us-central1\
       --purpose="asymmetric-signing" \
       --protection-level=hsm \
       --default-algorithm=ec-sign-p256-sha256 \
       --project=KEY_PROJECT_ID
   

Create the CAs

After creating the keys for each of the control plane functions, use each key to create the CA pools and corresponding root CAs using CA Service:

1. Create the cluster CA pool:

   gcloud privateca pools create cluster-ca-pool \
       --location=us-central1 \
       --tier=enterprise \
       --project=KEY_PROJECT_ID \
       --no-publish-crl --no-publish-ca-cert
   

   The --no-publish-crl flag and the --no-publish-ca-cert flag are optional. Omitting these flags publishes certificates to a Cloud Storage bucket. For details, see Enable CA certificate and CRL publication for CAs in a CA pool.

2. Create the cluster root CA:

   gcloud privateca roots create cluster-root-ca \
       --pool=cluster-ca-pool \
       --location=us-central1 \
       --kms-key-version=projects/KEY_PROJECT_ID/locations/us-central1/keyRings/KEY_RING_NAME/cryptoKeys/cluster-ca-key/cryptoKeyVersions/1 \
       --subject="CN=cluster-ca, O=ORGANIZATION" \
       --project=KEY_PROJECT_ID \
       --auto-enable
   

   Replace ORGANIZATION with your organization name.

3. Create the etcd peer CA pool:

   gcloud privateca pools create etcd-peer-ca-pool \
       --location=us-central1 \
       --tier=enterprise \
       --project=KEY_PROJECT_ID \
       --no-publish-crl --no-publish-ca-cert
   

4. Create the etcd peer root CA:

   gcloud privateca roots create etcd-peer-root-ca \
       --pool=etcd-peer-ca-pool \
       --location=us-central1 \
       --kms-key-version=projects/KEY_PROJECT_ID/locations/us-central1/keyRings/KEY_RING_NAME/cryptoKeys/etcd-peer-ca-key/cryptoKeyVersions/1 \
       --subject="CN=etcd-peer-ca, O=ORGANIZATION" \
       --project=KEY_PROJECT_ID \
       --auto-enable
   

5. Create the etcd API CA pool:

   gcloud privateca pools create etcd-api-ca-pool \
       --location=us-central1 \
       --tier=enterprise \
       --project=KEY_PROJECT_ID \
       --no-publish-crl --no-publish-ca-cert
   

6. Create the etcd API root CA:

   gcloud privateca roots create etcd-api-root-ca \
       --pool=etcd-api-ca-pool \
       --location=us-central1 \
       --kms-key-version=projects/KEY_PROJECT_ID/locations/us-central1/keyRings/KEY_RING_NAME/cryptoKeys/etcd-api-ca-key/cryptoKeyVersions/1 \
       --subject="CN=etcd-api-ca, O=ORGANIZATION" \
       --project=KEY_PROJECT_ID \
       --auto-enable
   

7. Create the aggregation CA pool:

   gcloud privateca pools create aggregation-ca-pool \
       --location=us-central1 \
       --tier=enterprise \
       --project=KEY_PROJECT_ID \
       --no-publish-crl --no-publish-ca-cert
   

8. Create the aggregation root CA:

   gcloud privateca roots create aggregation-root-ca \
       --pool=aggregation-ca-pool \
       --location=us-central1 \
       --kms-key-version=projects/KEY_PROJECT_ID/locations/us-central1/keyRings/KEY_RING_NAME/cryptoKeys/aggregation-ca-key/cryptoKeyVersions/1 \
       --subject="CN=aggregation-ca, O=ORGANIZATION" \
       --project=KEY_PROJECT_ID \
       --auto-enable
   

Grant IAM roles to the GKE service agent

The GKE service agent requires access to the resources that you created in Cloud KMS and in CA Service. The service agent uses these resources to sign, verify, and issue credentials in the cluster. You can use the following predefined IAM roles:

• Kubernetes Engine KMS Crypto Key User (roles/container.cloudKmsKeyUser)
• CA Service Certificate Manager (roles/privateca.certificateManager)

To grant these roles to the GKE service agent, do the following:

1. Find the project number of your cluster project:

   gcloud projects describe CLUSTER_PROJECT_ID \
       --format='value(projectNumber)'
   

   Replace CLUSTER_PROJECT_ID with the project ID of your cluster project.

2. Grant the Kubernetes Engine KMS Crypto Key User role on the service account signing key that you created in Create keys:

   gcloud kms keys add-iam-policy-binding sa-signing-key \
     --location=us-central1 \
     --keyring=KEY_RING_NAME \
     --member="serviceAccount:[e-mail]" \
     --role=roles/container.cloudKmsKeyUser \
     --project=KEY_PROJECT_ID
   

   Replace CLUSTER_PROJECT_NUMBER with the project number of the cluster project.

3. Grant the CA Service Certificate Manager role on the CA pools that you created in Create the CAs:

   gcloud privateca pools add-iam-policy-binding cluster-ca-pool \
       --location=us-central1 \
       --member="serviceAccount:[e-mail]" \
       --role=roles/privateca.certificateManager \
       --project=KEY_PROJECT_ID
   
   gcloud privateca pools add-iam-policy-binding etcd-peer-ca-pool \
       --location=us-central1 \
       --member="serviceAccount:[e-mail]" \
       --role=roles/privateca.certificateManager \
       --project=KEY_PROJECT_ID
   
   gcloud privateca pools add-iam-policy-binding etcd-api-ca-pool \
       --location=us-central1 \
       --member="serviceAccount:[e-mail]" \
       --role=roles/privateca.certificateManager \
       --project=KEY_PROJECT_ID
   
   gcloud privateca pools add-iam-policy-binding aggregation-ca-pool \
       --location=us-central1 \
       --member="serviceAccount:[e-mail]" \
       --role=roles/privateca.certificateManager \
       --project=KEY_PROJECT_ID
   

Set up CAs and keys on a new cluster

Warning: Deleting keys or CAs that are in use by the cluster might lead to irrecoverable clusters. Before you delete keys or CAs, ensure that your cluster no longer uses the resources.

After you create keys, CA pools, root CAs, and grant IAM roles to the GKE service agent, create a new cluster that uses these resources.

The flags that you specify in the cluster creation command require the following resource paths as values:

• Path to a key version in Cloud KMS for the service account signing key that you created in Create keys. You specify this path for the service-account-signing-keys flag and for the service-account-verification-keys flag.
• Path to each of the CA pools that you created in Create the CAs.

Note: Before you perform these steps, decide on whether you also want to encrypt your control plane boot disks and etcd disks with keys that you manage. If you do, perform all of the key configuration steps in Encrypt etcd and control plane boot disks and specify the optional flags described in the cluster creation command in this section.

To configure a new cluster to use your keys and CAs, do the following steps:

1. Find the path to the latest enabled service account signing key version:

   gcloud kms keys versions list \
       --key=sa-signing-key \
       --keyring=KEY_RING_NAME \
       --location=us-central1 \
       --project=KEY_PROJECT_ID \
       --filter="STATE=ENABLED" --sort-by=~ --format="value(name)" | sed 1q
   

   Replace KEY_PROJECT_ID with the project ID of the key project.

   The output is similar to the following:

   projects/KEY_PROJECT_ID/locations/us-central1/keyRings/KEY_RING_NAME/cryptoKeys/sa-signing-key/cryptoKeyVersions/1
   

2. Find the paths to each of the CA pools that you created:

   gcloud privateca pools list --format="get(name)" \
       --project=KEY_PROJECT_ID
   

   The output is similar to the following:

   projects/KEY_PROJECT_ID/locations/us-central1/caPools/cluster-ca-pool
   projects/KEY_PROJECT_ID/locations/us-central1/caPools/etcd-peer-ca-pool
   projects/KEY_PROJECT_ID/locations/us-central1/caPools/etcd-api-ca-pool
   projects/KEY_PROJECT_ID/locations/us-central1/caPools/aggregation-ca-pool
   

   Ensure that the output contains all of the CA pools that you created for GKE.

Create a cluster

In this section, you create a cluster with different options specified depending on which GKE control plane authority features you want to configure. You can only configure these features on a cluster during cluster creation. The following commands create Autopilot mode clusters. To create Standard mode clusters instead, use the same flags with the gcloud container clusters create command.

• To only configure the CAs and keys that you created in this tutorial, run the following command:

  gcloud container clusters create-auto example-cluster \
      --location=us-central1 \
      --project=CLUSTER_PROJECT_ID \
      --cluster-version=VERSION \
      --service-account-signing-keys=projects/KEY_PROJECT_ID/locations/us-central1/keyRings/KEY_RING_NAME/cryptoKeys/sa-signing-key/cryptoKeyVersions/1 \
      --service-account-verification-keys=projects/KEY_PROJECT_ID/locations/us-central1/keyRings/KEY_RING_NAME/cryptoKeys/sa-signing-key/cryptoKeyVersions/1 \
      --cluster-ca=projects/KEY_PROJECT_ID/locations/us-central1/caPools/cluster-ca-pool \
      --etcd-peer-ca=projects/KEY_PROJECT_ID/locations/us-central1/caPools/etcd-peer-ca-pool \
      --etcd-api-ca=projects/KEY_PROJECT_ID/locations/us-central1/caPools/etcd-api-ca-pool \
      --aggregation-ca=projects/KEY_PROJECT_ID/locations/us-central1/caPools/aggregation-ca-pool
  

  Replace the following:

  • CLUSTER_PROJECT_ID: the project ID of the cluster project.
  • VERSION: the GKE version of the cluster. Must be 1.31.1-gke.1846000 or later.

• To configure the CAs and keys as well as control plane boot disk encryption and etcd encryption, do the following:

  1. Perform all of the key configuration steps in Encrypt etcd and control plane boot disks.
  2. Find the paths to each of the keys by using the instructions in Use encryption keys in a cluster.

  3. Create a cluster:

     gcloud container clusters create-auto example-cluster \
         --location=us-central1 \
         --project=CLUSTER_PROJECT_ID \
         --cluster-version=VERSION \
         --service-account-signing-keys=projects/KEY_PROJECT_ID/locations/us-central1/keyRings/KEY_RING_NAME/cryptoKeys/sa-signing-key/cryptoKeyVersions/1 \
         --service-account-verification-keys=projects/KEY_PROJECT_ID/locations/us-central1/keyRings/KEY_RING_NAME/cryptoKeys/sa-signing-key/cryptoKeyVersions/1 \
         --cluster-ca=projects/KEY_PROJECT_ID/locations/us-central1/caPools/cluster-ca-pool \
         --etcd-peer-ca=projects/KEY_PROJECT_ID/locations/us-central1/caPools/etcd-peer-ca-pool \
         --etcd-api-ca=projects/KEY_PROJECT_ID/locations/us-central1/caPools/etcd-api-ca-pool \
         --aggregation-ca=projects/KEY_PROJECT_ID/locations/us-central1/caPools/aggregation-ca-pool \
         --control-plane-disk-encryption-key=PATH_TO_DISK_KEY \
         --gkeops-etcd-backup-encryption-key=PATH_TO_ETCD_BACKUP_KEY
     

     Replace the following:

     • CLUSTER_PROJECT_ID: the project ID of the cluster project.
     • VERSION: the GKE version of the cluster. Must be 1.31.1-gke.1846000 or later.
     • PATH_TO_DISK_KEY: the path to your disk encryption key.
     • PATH_TO_ETCD_BACKUP_KEY: the path to your etcd internal backup encryption key.

  You can also use these flags when you create a new Standard mode cluster.

Verify that the cluster uses your specified keys and CAs

This section shows you how to verify the keys and CAs that were used during cluster creation. You can perform this verification using Cloud Logging or by using the Google Cloud CLI.

Use Logging to verify keys and CAs

To verify the keys and CAs using Logging, do the following:

1. In the Google Cloud console, go to the Logs Explorer page:

   Go to Logs Explorer

2. Specify the following query:

   resource.type="gke_cluster"
   resource.labels.cluster_name="CLUSTER_NAME"
   resource.labels.location="CLUSTER_LOCATION"
   protoPayload.serviceName="container.googleapis.com"
   protoPayload.methodName=~"google.container.v(1|1alpha1|1beta1).ClusterManager.CreateCluster"
   protoPayload.request.cluster.userManagedKeysConfig:*
   

   protoPayload.request.cluster.userManagedKeysConfig:* filters the results for cluster creation logs that include keys and CAs that you manage.

3. Click Run query.

In the results, expand your cluster creation log. Verify that the paths to keys and CAs are the same as those that you created for that cluster, like in the following example:

# lines omitted for clarity
userManagedKeysConfig: {
  aggregationCa: "projects/KEY_PROJECT_ID/locations/us-central1/caPools/aggregation-ca-pool"
  clusterCa: "projects/KEY_PROJECT_ID/locations/us-central1/caPools/cluster-ca-pool"
  etcdApiCa: "projects/KEY_PROJECT_ID/locations/us-central1/caPools/etcd-api-ca-pool"
  etcdPeerCa: "projects/KEY_PROJECT_ID/locations/us-central1/caPools/etcd-peer-ca-pool"
  serviceAccountSigningKeys: [
    0: "projects/KEY_PROJECT_ID/locations/us-central1/keyRings/KEY_RING_NAME/cryptoKeys/sa-signing-key/cryptoKeyVersions/1"
  ]
  serviceAccountVerificationKeys: [
    0: "projects/KEY_PROJECT_ID/locations/us-central1/keyRings/KEY_RING_NAME/cryptoKeys/sa-signing-key/cryptoKeyVersions/1"
  ]
}

Use the gcloud CLI to verify keys and CAs

To verify that the cluster uses the CAs and keys that you created, run the following command:

gcloud container clusters describe example-cluster \
    --location=us-central1 \
    --project=CLUSTER_PROJECT_ID

The output should include the userManagedKeysConfig field like in the following example:

# lines omitted for clarity
userManagedKeysConfig:
  sa-signing-key: projects/KEY_PROJECT_ID/locations/us-central1/keyRings/KEY_RING_NAME/cryptoKeys/sa-signing-key/cryptoKeyVersions/1
  sa-verification-key: projects/KEY_PROJECT_ID/locations/us-central1/keyRings/KEY_RING_NAME/cryptoKeys/sa-signing-key/cryptoKeyVersions/1
  cluster-ca: projects/KEY_PROJECT_ID/locations/us-central1/caPools/cluster-ca-pool
  etcd-peer-ca: projects/KEY_PROJECT_ID/locations/us-central1/caPools/etcd-peer-ca-pool
  etcd-api-ca: projects/KEY_PROJECT_ID/locations/us-central1/caPools/etcd-api-ca-pool
  aggregation-ca: projects/KEY_PROJECT_ID/locations/us-central1/caPools/aggregation-ca-pool

Clean up

To avoid incurring charges to your Google Cloud account for the resources used in this tutorial, either delete the project that contains the resources, or keep the project and delete the individual resources.

Delete the projects

Caution: Deleting a project has the following effects:

• Everything in the project is deleted. If you used an existing project for the tasks in this document, when you delete it, you also delete any other work you've done in the project.
• Custom project IDs are lost. When you created this project, you might have created a custom project ID that you want to use in the future. To preserve the URLs that use the project ID, such as an appspot.com URL, delete selected resources inside the project instead of deleting the whole project.

If you plan to explore multiple architectures, tutorials, or quickstarts, reusing projects can help you avoid exceeding project quota limits.

Delete a Google Cloud project:

gcloud projects delete PROJECT_ID

Delete individual resources

1. Delete the cluster:

   gcloud container clusters delete example-cluster \
       --location=us-central1 \
       --project=CLUSTER_PROJECT_ID
   

2. Disable the root CAs:

   gcloud privateca roots disable cluster-root-ca \
       --location=us-central1 \
       --pool=projects/KEY_PROJECT_ID/locations/us-central1/caPools/cluster-ca-pool \
       --project=KEY_PROJECT_ID
   
   gcloud privateca roots disable etcd-peer-root-ca \
       --location=us-central1 \
       --pool=projects/KEY_PROJECT_ID/locations/us-central1/caPools/etcd-peer-ca-pool \
       --project=KEY_PROJECT_ID
   
   gcloud privateca roots disable etcd-api-root-ca \
       --location=us-central1 \
       --pool=projects/KEY_PROJECT_ID/locations/us-central1/caPools/etcd-api-ca-pool \
       --project=KEY_PROJECT_ID
   
   gcloud privateca roots disable aggregation-root-ca \
       --location=us-central1 \
       --pool=projects/KEY_PROJECT_ID/locations/us-central1/caPools/aggregation-ca-pool \
       --project=KEY_PROJECT_ID
   

3. Delete the root CAs:

   gcloud privateca roots delete cluster-root-ca \
       --location=us-central1 \
       --pool=projects/KEY_PROJECT_ID/locations/us-central1/caPools/cluster-ca-pool \
       --project=KEY_PROJECT_ID
   
   gcloud privateca roots delete etcd-peer-root-ca \
       --location=us-central1 \
       --pool=projects/KEY_PROJECT_ID/locations/us-central1/caPools/etcd-peer-ca-pool \
       --project=KEY_PROJECT_ID
   
   gcloud privateca roots delete etcd-api-root-ca \
       --location=us-central1 \
       --pool=projects/KEY_PROJECT_ID/locations/us-central1/caPools/etcd-api-ca-pool \
       --project=KEY_PROJECT_ID
   
   gcloud privateca roots delete aggregation-root-ca \
       --location=us-central1 \
       --pool=projects/KEY_PROJECT_ID/locations/us-central1/caPools/aggregation-ca-pool \
       --project=KEY_PROJECT_ID
   

4. Delete the CA pools:

   gcloud privateca pools delete cluster-ca-pool --location=us-central1 \
       --project=KEY_PROJECT_ID
   
   gcloud privateca pools delete etcd-peer-ca-pool --location=us-central1 \
       --project=KEY_PROJECT_ID
   
   gcloud privateca pools delete etcd-api-ca-pool --location=us-central1 \
       --project=KEY_PROJECT_ID
   
   gcloud privateca pools delete aggregation-ca-pool --location=us-central1 \
       --project=KEY_PROJECT_ID
   

5. Delete the keys:

   gcloud kms keys versions destroy 1 \
       --location=us-central1 \
       --keyring=KEY_RING_NAME \
       --key=sa-signing-key \
       --project=KEY_PROJECT_ID
   
   gcloud kms keys versions destroy 1 \
       --location=us-central1 \
       --keyring=KEY_RING_NAME \
       --key=cluster-ca-key \
       --project=KEY_PROJECT_ID
   
   gcloud kms keys versions destroy 1 \
       --location=us-central1 \
       --keyring=KEY_RING_NAME \
       --key=etcd-peer-ca-key \
       --project=KEY_PROJECT_ID
   
   gcloud kms keys versions destroy 1 \
       --location=us-central1 \
       --keyring=KEY_RING_NAME \
       --key=etcd-api-ca-key \
       --project=KEY_PROJECT_ID
   
   gcloud kms keys versions destroy 1 \
       --location=us-central1 \
       --keyring=KEY_RING_NAME \
       --key=aggregation-ca-key \
       --project=KEY_PROJECT_ID
   

You can't delete key rings from Cloud KMS. However, key rings don't incur additional costs.

What's next

• Track identity usage from the moment of issuance
• Explore reference architectures, diagrams, and best practices about Google Cloud. Take a look at our Cloud Architecture Center.