This page shows you how to configure your Google Kubernetes Engine (GKE) cluster control plane with certificate authorities (CAs) and keys that you manage. This guidance is intended for security administrators who have specific organizational compliance or policy requirements for control over credential issuance and signing.
This page describes one part of a set of optional control plane features in GKE that lets you perform tasks like verifying your control plane security posture or configuring encryption and credential signing in the control plane using keys that you manage. For details, see About GKE control plane authority.
By default, Google Cloud applies various security measures to the managed control plane. This page describes optional capabilities that give you more visibility or control over the control plane.
You should already be familiar with the following concepts:
Control plane credential components
GKE clusters use specific CAs and keys to issue credentials in the cluster, like X.509 certificates or ServiceAccount tokens. You can create keys in Cloud Key Management Service (Cloud KMS) and CAs in Certificate Authority Service (CA Service) and configure your clusters to use these resources instead of Google Cloud-managed CAs and keys.
To learn more about the specific components that you create, see Self-managed CAs and keys.
Usage with other GKE control plane authority features
GKE control plane authority provides the following features related to self-managed keys:
- Run your own certificate authorities (CAs) and keys (this page)
- Encrypt control plane components
Objectives
- Create keys in Cloud KMS
- Create CAs in CA Service
- Grant Identity and Access Management (IAM) roles to the GKE service agent
- Create a GKE cluster that uses your CAs and keys
- Verify that the cluster uses your CAs and keys
Costs
In this document, you use the following billable components of Google Cloud:
To generate a cost estimate based on your projected usage,
use the pricing calculator.
When you finish the tasks that are described in this document, you can avoid continued billing by deleting the resources that you created. For more information, see Clean up.
Before you begin
- Sign in to your Google Cloud account. If you're new to Google Cloud, create an account to evaluate how our products perform in real-world scenarios. New customers also get $300 in free credits to run, test, and deploy workloads.
- Install the Google Cloud CLI.
-
To initialize the gcloud CLI, run the following command:
gcloud init
-
Make sure that billing is enabled for your Google Cloud project.
-
Enable the Kubernetes Engine, Certificate Authority Service, and Cloud Key Management Service APIs:
gcloud services enable container.googleapis.com
privateca.googleapis.com cloudkms.googleapis.com - Install the Google Cloud CLI.
-
To initialize the gcloud CLI, run the following command:
gcloud init
-
Make sure that billing is enabled for your Google Cloud project.
-
Enable the Kubernetes Engine, Certificate Authority Service, and Cloud Key Management Service APIs:
gcloud services enable container.googleapis.com
privateca.googleapis.com cloudkms.googleapis.com - Ensure that your environment is eligible to use GKE control plane authority features. To opt-in to these features, contact your Google Cloud sales team.
- To reliably track credential issuance and usage, ensure that the following
Data Access audit logs are enabled:
- Cloud KMS:
DATA_READ
- CA Service:
ADMIN_READ
andADMIN_WRITE
To enable these log types, see Enable Data Access audit logs.
- Cloud KMS:
Required roles and permissions
To get the permissions that you need to run your own CAs and keys, ask your administrator to grant you the following IAM roles:
-
Create Cloud KMS keys:
Cloud KMS Admin (
roles/cloudkms.admin
) on your key project -
Create CA pools and root CAs:
CA Service Admin (
roles/privateca.admin
) on your key project -
Create GKE clusters:
Kubernetes Engine Cluster Admin (
roles/container.clusterAdmin
) on your cluster project
For more information about granting roles, see Manage access to projects, folders, and organizations.
You might also be able to get the required permissions through custom roles or other predefined roles.
Requirements
You must use GKE version 1.31.1-gke.1846000 or later.
Limitations
The following limitations apply:
- You can only use keys from Cloud KMS. You can't use another KMS provider or another encryption provider.
- Cloud External Key Manager (Cloud EKM) keys aren't supported.
- You can only use CAs from CA Service.
Prepare the environment
In this section, you identify Google Cloud projects that you'll use in this tutorial, and you create a key ring in Cloud KMS to hold your keys.
Identify projects
We recommend that you use separate Google Cloud projects as follows:
- Key project: contains all keys and CAs.
- Cluster project: contains your GKE clusters.
You can optionally use the same project for your keys, CAs, and GKE clusters, but we recommend that you use separate projects so that the teams that manage cryptographic operations in your organization are separated from the teams that manage cluster operations.
Create a key ring
Create a key ring in the key project to hold all of the keys for a specific cluster. You must create the key ring in the same location as your GKE cluster.
Run the following command:
gcloud kms keyrings create KEY_RING_NAME \
--location=us-central1 \
--project=KEY_PROJECT_ID
Replace the following:
KEY_RING_NAME
: a name for your key ring.KEY_PROJECT_ID
: the project ID of your key project.
Create keys
For each of the credential authorities like service account keys and CAs, you
create a key using Cloud KMS. This section shows you how to create the
keys that GKE uses to sign and verify credentials in the cluster.
You can specify your own properties for these keys depending on your
organization's needs. For details, see the Create a key
page and the
projects.locations.keyRings.cryptoKeys
API reference.
Consider the following when you create these resources in Cloud KMS:
- If you have a existing key ring in your key project, you can use that key ring to store all of the keys that you create for use with your cluster.
- Your key ring must be in the same Google Cloud location as your cluster to minimize latency.
- Keys must specify
asymmetric-signing
as the key purpose. - Use the following algorithms based on the type of key:
- ServiceAccount signing keys: a strong RSA signing PKCS1 algorithm, like
rsa-sign-pkcs1-4096-sha256
orrsa-sign-pkcs1-3072-sha256
. - Certificate authority keys: a strong algorithm like
ec-sign-p256-sha256
.
- ServiceAccount signing keys: a strong RSA signing PKCS1 algorithm, like
- Cloud HSM hardware keys are supported, but the
software
protection level is sufficient for most use cases. For details about hardware keys, see Cloud HSM. - Don't modify the default duration for key destruction.
- GKE doesn't prevent you from deleting Cloud KMS keys, including CA Service keys, that are in use by the cluster. Before you delete keys or CAs, ensure that the resources are unused.
To create your keys, run the following commands:
Create the Kubernetes ServiceAccount signing key, which you also specify as the service account verification key during cluster creation:
gcloud kms keys create sa-signing-key \ --keyring=KEY_RING_NAME \ --location=us-central1\ --purpose="asymmetric-signing" \ --protection-level=hsm \ --default-algorithm=rsa-sign-pkcs1-4096-sha256 \ --project=KEY_PROJECT_ID
Replace
KEY_PROJECT_ID
with the project ID for your dedicated key project.Create the cluster root CA key:
gcloud kms keys create cluster-ca-key \ --keyring=KEY_RING_NAME \ --location=us-central1\ --purpose="asymmetric-signing" \ --protection-level=hsm \ --default-algorithm=ec-sign-p256-sha256 \ --project=KEY_PROJECT_ID
Create the etcd peer root CA key:
gcloud kms keys create etcd-peer-ca-key \ --keyring=KEY_RING_NAME \ --location=us-central1\ --purpose="asymmetric-signing" \ --protection-level=hsm \ --default-algorithm=ec-sign-p256-sha256 \ --project=KEY_PROJECT_ID
Create the etcd API root CA key:
gcloud kms keys create etcd-api-ca-key \ --keyring=KEY_RING_NAME \ --location=us-central1\ --purpose="asymmetric-signing" \ --protection-level=hsm \ --default-algorithm=ec-sign-p256-sha256 \ --project=KEY_PROJECT_ID
Create the aggregation root CA key:
gcloud kms keys create aggregation-ca-key \ --keyring=KEY_RING_NAME \ --location=us-central1\ --purpose="asymmetric-signing" \ --protection-level=hsm \ --default-algorithm=ec-sign-p256-sha256 \ --project=KEY_PROJECT_ID
Create the CAs
After creating the keys for each of the control plane functions, use each key to create the CA pools and corresponding root CAs using CA Service:
Create the cluster CA pool:
gcloud privateca pools create cluster-ca-pool \ --location=us-central1 \ --tier=enterprise \ --project=KEY_PROJECT_ID \ --no-publish-crl --no-publish-ca-cert
The
--no-publish-crl
flag and the--no-publish-ca-cert
flag are optional. Omitting these flags publishes certificates to a Cloud Storage bucket. For details, see Enable CA certificate and CRL publication for CAs in a CA pool.Create the cluster root CA:
gcloud privateca roots create cluster-root-ca \ --pool=cluster-ca-pool \ --location=us-central1 \ --kms-key-version=projects/KEY_PROJECT_ID/locations/us-central1/keyRings/KEY_RING_NAME/cryptoKeys/cluster-ca-key/cryptoKeyVersions/1 \ --subject="CN=cluster-ca, O=ORGANIZATION" \ --project=KEY_PROJECT_ID \ --auto-enable
Replace
ORGANIZATION
with your organization name.Create the etcd peer CA pool:
gcloud privateca pools create etcd-peer-ca-pool \ --location=us-central1 \ --tier=enterprise \ --project=KEY_PROJECT_ID \ --no-publish-crl --no-publish-ca-cert
Create the etcd peer root CA:
gcloud privateca roots create etcd-peer-root-ca \ --pool=etcd-peer-ca-pool \ --location=us-central1 \ --kms-key-version=projects/KEY_PROJECT_ID/locations/us-central1/keyRings/KEY_RING_NAME/cryptoKeys/etcd-peer-ca-key/cryptoKeyVersions/1 \ --subject="CN=etcd-peer-ca, O=ORGANIZATION" \ --project=KEY_PROJECT_ID \ --auto-enable
Create the etcd API CA pool:
gcloud privateca pools create etcd-api-ca-pool \ --location=us-central1 \ --tier=enterprise \ --project=KEY_PROJECT_ID \ --no-publish-crl --no-publish-ca-cert
Create the etcd API root CA:
gcloud privateca roots create etcd-api-root-ca \ --pool=etcd-api-ca-pool \ --location=us-central1 \ --kms-key-version=projects/KEY_PROJECT_ID/locations/us-central1/keyRings/KEY_RING_NAME/cryptoKeys/etcd-api-ca-key/cryptoKeyVersions/1 \ --subject="CN=etcd-api-ca, O=ORGANIZATION" \ --project=KEY_PROJECT_ID \ --auto-enable
Create the aggregation CA pool:
gcloud privateca pools create aggregation-ca-pool \ --location=us-central1 \ --tier=enterprise \ --project=KEY_PROJECT_ID \ --no-publish-crl --no-publish-ca-cert
Create the aggregation root CA:
gcloud privateca roots create aggregation-root-ca \ --pool=aggregation-ca-pool \ --location=us-central1 \ --kms-key-version=projects/KEY_PROJECT_ID/locations/us-central1/keyRings/KEY_RING_NAME/cryptoKeys/aggregation-ca-key/cryptoKeyVersions/1 \ --subject="CN=aggregation-ca, O=ORGANIZATION" \ --project=KEY_PROJECT_ID \ --auto-enable
Grant IAM roles to the GKE service agent
The GKE service agent requires access to the resources that you created in Cloud KMS and in CA Service. The service agent uses these resources to sign, verify, and issue credentials in the cluster. You can use the following predefined IAM roles:
- Kubernetes Engine KMS Crypto Key User
(
roles/container.cloudKmsKeyUser
) - CA Service Certificate Manager
(
roles/privateca.certificateManager
)
To grant these roles to the GKE service agent, do the following:
Find the project number of your cluster project:
gcloud projects describe CLUSTER_PROJECT_ID \ --format='value(projectNumber)'
Replace
CLUSTER_PROJECT_ID
with the project ID of your cluster project.Grant the Kubernetes Engine KMS Crypto Key User role on the service account signing key that you created in Create keys:
gcloud kms keys add-iam-policy-binding sa-signing-key \ --location=us-central1 \ --keyring=KEY_RING_NAME \ --member="serviceAccount:service-CLUSTER_PROJECT_NUMBER@container-engine-robot.iam.gserviceaccount.com" \ --role=roles/container.cloudKmsKeyUser \ --project=KEY_PROJECT_ID
Replace
CLUSTER_PROJECT_NUMBER
with the project number of the cluster project.Grant the CA Service Certificate Manager role on the CA pools that you created in Create the CAs:
gcloud privateca pools add-iam-policy-binding cluster-ca-pool \ --location=us-central1 \ --member="serviceAccount:service-CLUSTER_PROJECT_NUMBER@container-engine-robot.iam.gserviceaccount.com" \ --role=roles/privateca.certificateManager \ --project=KEY_PROJECT_ID gcloud privateca pools add-iam-policy-binding etcd-peer-ca-pool \ --location=us-central1 \ --member="serviceAccount:service-CLUSTER_PROJECT_NUMBER@container-engine-robot.iam.gserviceaccount.com" \ --role=roles/privateca.certificateManager \ --project=KEY_PROJECT_ID gcloud privateca pools add-iam-policy-binding etcd-api-ca-pool \ --location=us-central1 \ --member="serviceAccount:service-CLUSTER_PROJECT_NUMBER@container-engine-robot.iam.gserviceaccount.com" \ --role=roles/privateca.certificateManager \ --project=KEY_PROJECT_ID gcloud privateca pools add-iam-policy-binding aggregation-ca-pool \ --location=us-central1 \ --member="serviceAccount:service-CLUSTER_PROJECT_NUMBER@container-engine-robot.iam.gserviceaccount.com" \ --role=roles/privateca.certificateManager \ --project=KEY_PROJECT_ID
Set up CAs and keys on a new cluster
After you create keys, CA pools, root CAs, and grant IAM roles to the GKE service agent, create a new cluster that uses these resources.
The flags that you specify in the cluster creation command require the following resource paths as values:
- Path to a key version in Cloud KMS for the service
account signing key that you created in Create keys. You
specify this path for the
service-account-signing-keys
flag and for theservice-account-verification-keys
flag. - Path to each of the CA pools that you created in Create the CAs.
To configure a new cluster to use your keys and CAs, do the following steps:
Find the path to the latest enabled service account signing key version:
gcloud kms keys versions list \ --key=sa-signing-key \ --keyring=KEY_RING_NAME \ --location=us-central1 \ --project=KEY_PROJECT_ID \ --filter="STATE=ENABLED" --sort-by=~ --format="value(name)" | sed 1q
Replace
KEY_PROJECT_ID
with the project ID of the key project.The output is similar to the following:
projects/KEY_PROJECT_ID/locations/us-central1/keyRings/KEY_RING_NAME/cryptoKeys/sa-signing-key/cryptoKeyVersions/1
Find the paths to each of the CA pools that you created:
gcloud privateca pools list --format="get(name)" \ --project=KEY_PROJECT_ID
The output is similar to the following:
projects/KEY_PROJECT_ID/locations/us-central1/caPools/cluster-ca-pool projects/KEY_PROJECT_ID/locations/us-central1/caPools/etcd-peer-ca-pool projects/KEY_PROJECT_ID/locations/us-central1/caPools/etcd-api-ca-pool projects/KEY_PROJECT_ID/locations/us-central1/caPools/aggregation-ca-pool
Ensure that the output contains all of the CA pools that you created for GKE.
Create a cluster
In this section, you create a cluster with different options specified
depending on which GKE control plane authority features you want to
configure. You can only configure these features on a cluster during cluster
creation. The following commands create Autopilot mode clusters. To
create Standard mode clusters instead, use the same flags with the
gcloud container clusters create
command.
To only configure the CAs and keys that you created in this tutorial, run the following command:
gcloud container clusters create-auto example-cluster \ --location=us-central1 \ --project=CLUSTER_PROJECT_ID \ --cluster-version=VERSION \ --service-account-signing-keys=projects/KEY_PROJECT_ID/locations/us-central1/keyRings/KEY_RING_NAME/cryptoKeys/sa-signing-key/cryptoKeyVersions/1 \ --service-account-verification-keys=projects/KEY_PROJECT_ID/locations/us-central1/keyRings/KEY_RING_NAME/cryptoKeys/sa-signing-key/cryptoKeyVersions/1 \ --cluster-ca=projects/KEY_PROJECT_ID/locations/us-central1/caPools/cluster-ca-pool \ --etcd-peer-ca=projects/KEY_PROJECT_ID/locations/us-central1/caPools/etcd-peer-ca-pool \ --etcd-api-ca=projects/KEY_PROJECT_ID/locations/us-central1/caPools/etcd-api-ca-pool \ --aggregation-ca=projects/KEY_PROJECT_ID/locations/us-central1/caPools/aggregation-ca-pool
Replace the following:
CLUSTER_PROJECT_ID
: the project ID of the cluster project.VERSION
: the GKE version of the cluster. Must be 1.31.1-gke.1846000 or later.
To configure the CAs and keys as well as control plane boot disk encryption and etcd encryption, do the following:
- Perform all of the key configuration steps in Encrypt etcd and control plane boot disks.
- Find the paths to each of the keys by using the instructions in Use encryption keys in a cluster.
Create a cluster:
gcloud container clusters create-auto example-cluster \ --location=us-central1 \ --project=CLUSTER_PROJECT_ID \ --cluster-version=VERSION \ --service-account-signing-keys=projects/KEY_PROJECT_ID/locations/us-central1/keyRings/KEY_RING_NAME/cryptoKeys/sa-signing-key/cryptoKeyVersions/1 \ --service-account-verification-keys=projects/KEY_PROJECT_ID/locations/us-central1/keyRings/KEY_RING_NAME/cryptoKeys/sa-signing-key/cryptoKeyVersions/1 \ --cluster-ca=projects/KEY_PROJECT_ID/locations/us-central1/caPools/cluster-ca-pool \ --etcd-peer-ca=projects/KEY_PROJECT_ID/locations/us-central1/caPools/etcd-peer-ca-pool \ --etcd-api-ca=projects/KEY_PROJECT_ID/locations/us-central1/caPools/etcd-api-ca-pool \ --aggregation-ca=projects/KEY_PROJECT_ID/locations/us-central1/caPools/aggregation-ca-pool \ --control-plane-disk-encryption-key=PATH_TO_DISK_KEY \ --gkeops-etcd-backup-encryption-key=PATH_TO_ETCD_BACKUP_KEY
Replace the following:
CLUSTER_PROJECT_ID
: the project ID of the cluster project.VERSION
: the GKE version of the cluster. Must be 1.31.1-gke.1846000 or later.PATH_TO_DISK_KEY
: the path to your disk encryption key.PATH_TO_ETCD_BACKUP_KEY
: the path to your etcd internal backup encryption key.
You can also use these flags when you create a new Standard mode cluster.
Verify that the cluster uses your specified keys and CAs
This section shows you how to verify the keys and CAs that were used during cluster creation. You can perform this verification using Cloud Logging or by using the Google Cloud CLI.
Use Logging to verify keys and CAs
To verify the keys and CAs using Logging, do the following:
In the Google Cloud console, go to the Logs Explorer page:
Specify the following query:
resource.type="gke_cluster" resource.labels.cluster_name="CLUSTER_NAME" resource.labels.location="CLUSTER_LOCATION" protoPayload.serviceName="container.googleapis.com" protoPayload.methodName="google.container.v1.ClusterManager.CreateCluster" protoPayload.request.cluster.userManagedKeysConfig:*
protoPayload.request.cluster.userManagedKeysConfig:*
filters the results for cluster creation logs that include keys and CAs that you manage.Click Run query.
In the results, expand your cluster creation log. Verify that the paths to keys and CAs are the same as those that you created for that cluster.
Use the gcloud CLI to verify keys and CAs
To verify that the cluster uses the CAs and keys that you created, run the following command:
gcloud container clusters describe example-cluster \
--location=us-central1 \
--project=CLUSTER_PROJECT_ID
The output should include the userManagedKeysConfig
field like in the
following example:
# lines omitted for clarity
userManagedKeysConfig:
sa-signing-key: projects/KEY_PROJECT_ID/locations/us-central1/keyRings/KEY_RING_NAME/cryptoKeys/sa-signing-key/cryptoKeyVersions/1
sa-verification-key: projects/KEY_PROJECT_ID/locations/us-central1/keyRings/KEY_RING_NAME/cryptoKeys/sa-signing-key/cryptoKeyVersions/1
cluster-ca: projects/KEY_PROJECT_ID/locations/us-central1/caPools/cluster-ca-pool
etcd-peer-ca: projects/KEY_PROJECT_ID/locations/us-central1/caPools/etcd-peer-ca-pool
etcd-api-ca: projects/KEY_PROJECT_ID/locations/us-central1/caPools/etcd-api-ca-pool
aggregation-ca: projects/KEY_PROJECT_ID/locations/us-central1/caPools/aggregation-ca-pool
Clean up
To avoid incurring charges to your Google Cloud account for the resources used in this tutorial, either delete the project that contains the resources, or keep the project and delete the individual resources.
Delete the projects
Delete a Google Cloud project:
gcloud projects delete PROJECT_ID
Delete individual resources
Delete the cluster:
gcloud container clusters delete example-cluster \ --location=us-central1 \ --project=CLUSTER_PROJECT_ID
Disable the root CAs:
gcloud privateca roots disable cluster-root-ca \ --location=us-central1 \ --pool=projects/KEY_PROJECT_ID/locations/us-central1/caPools/cluster-ca-pool \ --project=KEY_PROJECT_ID gcloud privateca roots disable etcd-peer-root-ca \ --location=us-central1 \ --pool=projects/KEY_PROJECT_ID/locations/us-central1/caPools/etcd-peer-ca-pool \ --project=KEY_PROJECT_ID gcloud privateca roots disable etcd-api-root-ca \ --location=us-central1 \ --pool=projects/KEY_PROJECT_ID/locations/us-central1/caPools/etcd-api-ca-pool \ --project=KEY_PROJECT_ID gcloud privateca roots disable aggregation-root-ca \ --location=us-central1 \ --pool=projects/KEY_PROJECT_ID/locations/us-central1/caPools/aggregation-ca-pool \ --project=KEY_PROJECT_ID
Delete the root CAs:
gcloud privateca roots delete cluster-root-ca \ --location=us-central1 \ --pool=projects/KEY_PROJECT_ID/locations/us-central1/caPools/cluster-ca-pool \ --project=KEY_PROJECT_ID gcloud privateca roots delete etcd-peer-root-ca \ --location=us-central1 \ --pool=projects/KEY_PROJECT_ID/locations/us-central1/caPools/etcd-peer-ca-pool \ --project=KEY_PROJECT_ID gcloud privateca roots delete etcd-api-root-ca \ --location=us-central1 \ --pool=projects/KEY_PROJECT_ID/locations/us-central1/caPools/etcd-api-ca-pool \ --project=KEY_PROJECT_ID gcloud privateca roots delete aggregation-root-ca \ --location=us-central1 \ --pool=projects/KEY_PROJECT_ID/locations/us-central1/caPools/aggregation-ca-pool \ --project=KEY_PROJECT_ID
Delete the CA pools:
gcloud privateca pools delete cluster-ca-pool --location=us-central1 \ --project=KEY_PROJECT_ID gcloud privateca pools delete etcd-peer-ca-pool --location=us-central1 \ --project=KEY_PROJECT_ID gcloud privateca pools delete etcd-api-ca-pool --location=us-central1 \ --project=KEY_PROJECT_ID gcloud privateca pools delete aggregation-ca-pool --location=us-central1 \ --project=KEY_PROJECT_ID
Delete the keys:
gcloud kms keys versions destroy 1 \ --location=us-central1 \ --keyring=KEY_RING_NAME \ --key=sa-signing-key \ --project=KEY_PROJECT_ID gcloud kms keys versions destroy 1 \ --location=us-central1 \ --keyring=KEY_RING_NAME \ --key=cluster-ca-key \ --project=KEY_PROJECT_ID gcloud kms keys versions destroy 1 \ --location=us-central1 \ --keyring=KEY_RING_NAME \ --key=etcd-peer-ca-key \ --project=KEY_PROJECT_ID gcloud kms keys versions destroy 1 \ --location=us-central1 \ --keyring=KEY_RING_NAME \ --key=etcd-api-ca-key \ --project=KEY_PROJECT_ID gcloud kms keys versions destroy 1 \ --location=us-central1 \ --keyring=KEY_RING_NAME \ --key=aggregation-ca-key \ --project=KEY_PROJECT_ID
You can't delete key rings from Cloud KMS. However, key rings don't incur additional costs.
What's next
- Track identity usage from the moment of issuance
- Explore reference architectures, diagrams, and best practices about Google Cloud. Take a look at our Cloud Architecture Center.