This topic describes how to rotate security credentials for your GKE on AWS management service and user clusters. For more information on the security features of GKE on AWS, see Security.
Before you begin
To complete the steps on this page, you must have access to the directory with your GKE on AWS configuration.
Management service certificates
This section describes how to rotate certificates for your management service.
Management certificate authorities
This section explains how to rotate Certificate authority (CA) signing certificates for GKE on AWS components.
Management API server root CA
To rotate the API server root CA, perform the following steps:
- Change to the directory with your GKE on AWS configuration.
You created this directory when
Installing the management service.
cd anthos-aws
- Open your
anthos-gke.status.yaml
in a text editor. - Remove all the values under the key
certificateAuthority
. This includesencryptedPrivateKey.kmsKeyARN
,encryptedPrivateKey.value
, andencryptedPrivateKey.certificate
. Run
anthos-gke aws management init
to update theanthos-gke.status.yaml
file.anthos-gke aws management init
Run
anthos-gke aws management apply
to update the management service.anthos-gke aws management apply
- From your
anthos-aws
directory, useanthos-gke
to switch context to your management service.cd anthos-aws anthos-gke aws management get-credentials
Other management CAs
This section describes how you can rotate all of the following CAs:
- Authentication webhook CA
- Etcd CA
- Service Account Signer CA
You can rotate these CAs with one of the following methods:
Remove the
certificateAuthority
section from youranthos-gke.status.yaml
.- Change to the directory with your GKE on AWS configuration.
You created this directory when
Installing the management service.
cd anthos-aws
- Open your
anthos-gke.status.yaml
in a text editor. - Remove all the values under the key
certificateAuthority
. This includesencryptedPrivateKey.kmsKeyARN
,encryptedPrivateKey.value
, andencryptedPrivateKey.certificate
. Run
anthos-gke aws management init
to update theanthos-gke.status.yaml
file.anthos-gke aws management init
Run
anthos-gke aws management apply
to update the management service.anthos-gke aws management apply
- Change to the directory with your GKE on AWS configuration.
You created this directory when
Installing the management service.
If a new GKE on AWS version is available, upgrade your GKE on AWS management service.
Recreate the management service EC2 instance.
- From your
anthos-aws
directory, useterraform
to get the ID of your management service. The output includes your management service ID. In the example below, the ID iscd anthos-aws terraform output cluster_id
gke-12345abc
.terraform output cluster_id
gke-12345abc - Open the AWS EC2 Console.
- Click Instances
- Find the instance named
cluster-id-management-0
. - Select Actions -> Instance state -> terminate to remove the instance. EC2 automatically creates a new instance with the same EBS volume.
- From your
Management TLS Client / Server keys and certificates
To rotate the TLS client / server keys and certificates for your management service, you recreate your management service instance. To recreate the instance, perform the following steps:
- From your
anthos-aws
directory, useterraform
to get the ID of your management service. The output includes your management service ID. In the example below, the ID iscd anthos-aws terraform output cluster_id
gke-12345abc
.terraform output cluster_id
gke-12345abc - Open the AWS EC2 Console.
- Click Instances
- Find the instance named
cluster-id-management-0
. - Select Actions -> Instance state -> terminate to remove the instance. EC2 automatically creates a new instance with the same EBS volume.
Google Cloud service accounts
Management service service accounts
To rotate the Google Cloud service accounts for your management service, perform the following steps.
- Create new service accounts and download service account keys following the steps in Prerequisites
- Change to the directory with your GKE on AWS configuration.
You created this directory when
Installing the management service.
cd anthos-aws
If you downloaded keys to a different path, open your
anthos-gke.yaml
file in a text editor. Change the value of.spec.googleCloud.serviceAccountKeys.managementService
,.status.googleCloud.serviceAccountKeys.connectAgent
, and.spec.googleCloud.serviceAccountKeys.node
to the new paths.apiVersion: multicloud.cluster.gke.io/v1 kind: AWSManagementService metadata: name: management spec: googleCloud: serviceAccountKeys: managementService: MANAGEMENT_KEY_PATH connectAgent: CONNECT_KEY_PATH node: NODE_KEY_PATH ...
Run
anthos-gke aws management init
to update theanthos-gke.status.yaml
file.anthos-gke aws management init
Run
anthos-gke aws management apply
to update the management service.anthos-gke aws management apply
User cluster service accounts
To apply these service accounts on your AWSClusters and AWSNodePools, you must upgrade or delete then re-create them.
User cluster certificates
This section describes how to rotate certificates for your user clusters.
User cluster CAs and SSH keys
Most CAs for your user clusters are created when the cluster is created.
When you delete a user cluster, GKE on AWS rotates the following certificates:
- API server root CA
- API front proxy CA
- etcd CA
- Kubernetes service account signer CA
- Control plane to node SSH key pairs
User cluster authentication webhook CA
To rotate the user cluster authentication webhook CA, you edit
your anthos-gke.status.yaml
file and apply the changes.
- Change to the directory with your GKE on AWS configuration.
You created this directory when
Installing the management service.
cd anthos-aws
- Open your
anthos-gke.status.yaml
in a text editor. - Remove all the values under the key
certificateAuthority
. This includesencryptedPrivateKey.kmsKeyARN
,encryptedPrivateKey.value
, andencryptedPrivateKey.certificate
. Run
anthos-gke aws management init
to update theanthos-gke.status.yaml
file.anthos-gke aws management init
Run
anthos-gke aws management apply
to update the management service.anthos-gke aws management apply
User cluster TLS key pairs and certificates
GKE on AWS generates TLS key pairs and certificates when creating an instance. To rotate these pairs, recreate the instance by performing the following steps for your control plane and node pools.
Control plane
- From your
anthos-aws
directory, useanthos-gke
to switch context to your management service.cd anthos-aws anthos-gke aws management get-credentials
Use
kubectl
to get your control plane's AWS EC2 target group from your AWSCluster.env HTTPS_PROXY=http://localhost:8118 \ kubectl get awscluster cluster-name \ -o jsonpath='{.status.targetGroupName}{"\n"}'
The output includes the name of your control plane's EC2 target group. For example,
gke-123456a7-controlplane
.Open the AWS EC2 Console. Choose Target Groups from the left pane.
Click the search bar and find your target group. Click the Name of your target group and then click Targets. The list of your control plane instances appears.
For each instance in the target group, perform the following steps:
Click the instance's Instance ID. The AWS EC2 Instances console appears.
Click on the Instance ID.
Select Actions -> Instance state -> terminate to remove the instance. EC2 automatically creates a new instance with the same EBS volume.
Return to the Target Groups page.
After you have terminated all the instances in the group, return to the Target Groups page.
In the Registered Targets section of the page, find the Status column. Each of your instances should have a Status of Healthy. If any of the instances are healthy, wait several minutes and click the Refresh (
) icon.After all instances in the target group are healthy, proceed to the next step.
Node pools
To rotate your node pool's TLS certificates:
- From your
anthos-aws
directory, useanthos-gke
to switch context to your management service.cd anthos-aws anthos-gke aws management get-credentials
Use
kubectl
to get your node pool's AWS EC2 target group from your AWSNodePool.env HTTPS_PROXY=http://localhost:8118 \ kubectl get awsnodepool -o jsonpath='{.items[*].status.autoScalingGroupName}{"\n"}'
The output includes the name of your node pool's EC2 target group. For example,
gke-123456a7-nodepool-abc123
.Open the AWS EC2 Console. Choose Target Groups from the left pane.
Click the search bar and find your target group. Click the Name of your target group and then click Targets. The list of your control plane instances appears.
For each instance in the target group, perform the following steps:
Click the instance's Instance ID. The AWS EC2 Instances console appears.
Click on the Instance ID.
Select Actions -> Instance state -> terminate to remove the instance. EC2 automatically creates a new instance with the same EBS volume.
Return to the Target Groups page.
After you have terminated all the instances in the group, return to the Target Groups page.
In the Registered Targets section of the page, find the Status column. Each of your instances should have a Status of Healthy. If any of the instances are healthy, wait several minutes and click the Refresh (
) icon.After all instances in the target group are healthy, proceed to the next step.