Ensure compatibility of TLS certificates before upgrading to GKE 1.29

Autopilot Standard

GKE clusters running version 1.29 or later don't support Transportation Layer Security (TLS) certificates signed with the SHA-1 algorithm. To prevent impact to your clusters, you need to replace incompatible certificates of webhook and extension API server backends with certificates using compatible signing algorithms before upgrading your clusters to version 1.29.

Impact to clusters of this removal

GKE pauses automatic upgrades when it detects that a cluster is using certificates that are incompatible with version 1.29. After you replace the certificates with certificates using compatible signing algorithms, or version 1.28 reaches end of life, GKE resumes automatic upgrades.

If you don't replace the incompatible certificates before upgrading to 1.29, you could experience the following issues with your clusters:

GKE webhook backends that use TLS certificates signed with the SHA-1 algorithm will stop working due to authentication failure. Webhook calls will fail for the Kubernetes control plane communicating with your webhooks with incompatible certificates. Depending on your configuration, especially if you use admission webhooks, failure to contact a webhook might block resource creation on your cluster, such as Pod creation, which can be very disruptive.
Calls to APIs served by the extension API servers will fail.

Why Kubernetes is removing this capability

GKE operates open source Kubernetes, which uses the kube-apiserver component to contact your webhook and extension API server backends using TLS. The kube-apiserver component is written in the Go programming language.

From Go version 1.18, Go started rejecting TLS certificates signed with the SHA-1 algorithm, but left a debug switch x509sha1=1 to enable the old behavior to ease the migration process. GKE version 1.24 was the first version built using Go version 1.18. GKE builds of Kubernetes had enabled this debug switch until version 1.29. The switch will be removed in Go 1.24. GKE 1.29 builds Kubernetes with the switch disabled to prepare for the future Go removal of the debug switch. After GKE upgrades your clusters to version 1.29, calls from your cluster's control plane to webhooks or extension API servers in the cluster that provide a TLS certificate signed with SHA-1 algorithm will fail.

Identify affected clusters

GKE monitors your clusters and uses the Recommender service to deliver guidance through insights and recommendations identifying clusters that have webhook or extension API server backends using TLS certificates signed with the SHA-1 algorithm. Or, you can use logs to identify calls to affected backends from your cluster.

How to get insights and recommendations

For clusters running version 1.24 or later, follow the instructions to view insights and recommendations. You can get insights using the gcloud CLI, or the Recommender API, filtering with the subtype DEPRECATION_K8S_SHA_1_CERTIFICATE.

How to get logs

For clusters running 1.24 or later with Cloud Logging enabled, GKE provides a Cloud Audit Logs log to identify calls to affected backends from your cluster. You can use the following filter to search for the logs:

logName =~ "projects/.*/logs/cloudaudit.googleapis.com%2Factivity"
resource.type = "k8s_cluster"
operation.producer = "k8s.io"
"insecure-sha1.invalid-cert.kubernetes.io"

The audit logs include the hostname of the affected backend. To learn more about how to interpret the results, see the next section.

Interpret the guidance from insights and recommendations

A recommendation includes the hostname of the affected backend, and whether it is webhook or extension API server. Hostnames that refer to Services in the cluster follow the format <service-name>.<namespace>.svc.

If the affected backend certificate is from a webhook server, the hostname can either be a Service in the cluster, or a URL. To learn more, see Contacting the webhook.

If the affected certificate is from an extension API server, the hostname is a Service in the cluster. To learn more, see the Contacting the extension apiserver.

After you have identified the affected backend, follow the instructions to inspect the certificate of a Service or Inspect the certificate of a URL backend, depending on the type.

If your clusters have not called servers with affected certificates in the last 30 days, you won't see any recommendations.

Example recommendations

See the following example list of recommendations:

RECOMMENDATION_ID                     PRIMARY_IMPACT_CATEGORY  RECOMMENDATION_STATE  LAST_REFRESH_TIME               PRIORITY  RECOMMENDER_SUBTYPE                DESCRIPTION
26bfcb32-6f2a-407c-874f-8cf55b3af912  RELIABILITY              ACTIVE                2024-02-15T01:09:04.454456273Z  P2        DEPRECATION_K8S_SHA_1_CERTIFICATE  Update the webhook and/or extension API servers that use certificates signed with SHA-1 algorithm to use certificates with compatible signing algorithms prior to upgrading the cluster to version 1.29. [Learn more](https://cloud.google.com/kubernetes-engine/docs/deprecations/sha1-1-29#mitigate_the_risk_of_upgrading_to_129).

To get details of the cluster and Service, describe the recommendation. The output for a Service named example-webhook in the default namespace is similar to the following:

associatedInsights:
- insight: projects/<CLUSTER_PROJECT_NUMBER>/locations/<CLUSTER_LOCATION>/insightTypes/google.container.DiagnosisInsight/insights/d76887a8-9eed-41a0-9459-d49dee43455e
content:
  overview:
    featureDeprecationRecommendation:
    - featureName: x.509_certificate_signature_algorithm
      featureReplacementValue: algorithm [compatible with GKE v1.29](https://cloud.google.com/kubernetes-engine/docs/deprecations/sha1-1-29#compatible-signing-algorithms)
      featureValue: SHA1
      stopServingVersion: '1.29'
      targetType: hostname
      targetValue: example-webhook.default.svc
    targetClusters:
    - clusterId: 3be916a554724c79a2314c8baee3fd57cf1c39df1ad34c3daf291db701b6d541
      clusterUri: //container.googleapis.com/projects/<CLUSTER_PROJECT_NUMBER>/locations/<CLUSTER_LOCATION>/clusters/<CLUSTER_NAME>
description: Update the webhook and/or extension API servers that use certificates
  signed with SHA-1 algorithm to use certificates with compatible signing algorithms
  prior to upgrading the cluster to version 1.29. [Learn more](https://cloud.google.com/kubernetes-engine/docs/deprecations/sha1-1-29#mitigate_the_risk_of_upgrading_to_129).
etag: '"ad50aac8278951d5"'
lastRefreshTime: '2024-02-15T01:09:04.454456273Z'
name: projects/<CLUSTER_PROJECT_NUMBER>/locations/<CLUSTER_LOCATION>/recommenders/google.container.DiagnosisRecommender/recommendations/26bfcb32-6f2a-407c-874f-8cf55b3af912
primaryImpact:
  category: RELIABILITY
  reliabilityProjection:
    risks:
    - SERVICE_DISRUPTION
priority: P2
recommenderSubtype: DEPRECATION_K8S_SHA_1_CERTIFICATE
stateInfo:
  state: ACTIVE
targetResources:
- //container.googleapis.com/projects/<CLUSTER_PROJECT_NUMBER>/locations/<CLUSTER_LOCATION>/clusters/<CLUSTER_NAME>

Inspect the certificate of a Service

Both webhooks and extension API servers can be backed by Services.

After you have identified relevant backend Services to inspect, use the following instructions to inspect the certificate of each Service to check which certificates use the SHA-1 algorithm and need to be updated.

Find the selector and target port of the Service:
```
kubectl describe service --namespace=NAMESPACE SERVICE_NAME
```
Replace NAMESPACE and SERVICE_NAME with the values from targetValue.

The output is similar to the following:
```
Name: example-service
Namespace: default
Labels: run=nginx
Selector: run=nginx
Type: ClusterIP
IP: 172.21.xxx.xxx
Port: 443
TargetPort: 444
```
This output indicates that example-service has the selector run=nginx and target port 444.

Note: When the TargetPort is not included, it is the same value as the Port field.

Find a Pod matching the selector:

kubectl get pods --namespace=NAMESPACE --selector=run=nginx

The output is similar to the following:

NAME          READY   STATUS    RESTARTS   AGE
example-pod   1/1     Running   0          21m

This output indicates that the matching Pod is example-pod.

Set up a port forward from your kubectl localhost to the Pod:

Note: These next commands use port 8888, however you can pick a different port, as this port is chosen arbitrarily.
```
kubectl port-forward --namespace=NAMESPACE pods/example-pod 8888:SERVICE_TARGET_PORT &
```
Replace SERVICE_TARGET_PORT with the TargetPort value from the Service. If TargetPort is not included, use the Port value.
Use openssl to show the certificate that the Service uses:
```
openssl s_client -connect localhost:8888 </dev/null | openssl x509 -noout -text
```
This example output shows a valid certificate signed with the SHA-256 algorithm:
```
Certificate:
    Data:
        ...
        Signature Algorithm: sha256WithRSAEncryption
...
    Signature Algorithm: sha256WithRSAEncryption
```
This example output shows an invalid certificate signed with the SHA-1 algorithm:
```
Certificate:
    Data:
        ...
        Signature Algorithm: sha1WithRSAEncryption
...
    Signature Algorithm: sha1WithRSAEncryption
```
If the output from the certificate is similar, you must update the certificate to use a compatible signing algorithm. For example, if you use the certificate.k8s.io API to manage TLS certificates in your cluster, you can follow the instructions to create a certificate signing request.

Clean up the port forward

To clean up the port forward running in the background, find the process and terminate it.

Run the following command to list the running processes:
```
jobs
```
See the output to get the ID of the process to terminate:
```
[1]+  Running                 kubectl port-forward pods/example-pod 8888:444 &
```
This example output indicates that the process ID is 1.
Terminate the process, replacing PROCESS_ID:
```
kill %PROCESS_ID
```
See the following output:
```
[1]+  Terminated              kubectl port-forward pods/example 8888:444
```
This example output shows that the process was terminated.

Inspect the certificate of a URL backend

If the webhook uses a url backend, directly connect to the hostname specified in the URL. For example, if the URL is https://example.com:123/foo/bar, use the following openssl command to show the certificate that the backend uses:

openssl s_client -connect example.com:123 </dev/null | openssl x509 -noout -text

This example output shows a valid certificate signed with the SHA-256 algorithm:

Certificate:
    Data:
        ...
        Signature Algorithm: sha256WithRSAEncryption
...
    Signature Algorithm: sha256WithRSAEncryption

This example output shows an invalid certificate signed with the SHA-1 algorithm:

Certificate:
    Data:
        ...
        Signature Algorithm: sha1WithRSAEncryption
...
    Signature Algorithm: sha1WithRSAEncryption

If the output from the certificate is similar, you must update the certificate to use a compatible signing algorithm. For example, if you use the certificate.k8s.io API to manage TLS certificates in your cluster, you can follow the instructions to create a certificate signing request.

Mitigate the risk of upgrading to 1.29

After you have identified affected clusters and their backend Services using certificates signed with SHA-1 algorithm, you must update the Services to use certificates with compatible signing algorithms prior to upgrading the clusters to version 1.29.

Affected clusters are automatically detected by GKE and won't be auto-upgraded to version 1.29 until either incompatible certificates are no longer used or version 1.28 reaches end of life. Once 1.28 reaches end of life, the clusters will be auto-upgraded to 1.29.

Compatible signing algorithms

GKE version 1.29 is compatible with supported algorithms in the Go x509 package. This includes the following algorithms:

SHA256WithRSA
SHA384WithRSA
SHA512WithRSA
ECDSAWithSHA256
ECDSAWithSHA384
ECDSAWithSHA512
SHA256WithRSAPSS
SHA384WithRSAPSS
SHA512WithRSAPSS
PureEd25519

To find available algorithms, see the x509.go source file and search for UnknownSignatureAlgorithm SignatureAlgorithm = iota. Algorithms that the Go x509 package supports are listed in the const block with this line. To find unsupported insecure signing algorithms, search for uses of InsecureAlgorithmError in the file.

Resources

See the following resources for additional information on this change: