Version 1.7. This version is supported as outlined in the Anthos version support policy, offering the latest patches and updates for security vulnerabilities, exposures, and issues impacting Anthos clusters on VMware (GKE on-prem). Refer to the release notes for more details. This is the most recent version.

Known issues

This document describes known issues for version 1.7 of Anthos clusters on VMware (GKE on-prem).

ClientConfig custom resource

gkectl update reverts any manual chages that you have made to the ClientConfig custom resource. We strongly recommend that you back up the ClientConfig resource after every manual change.

kubectl describe CSINode and gkectl diagnose snapshot

kubectl describe CSINode and gkectl diagnose snapshot sometimes fail due to the OSS Kubernetes issue on dereferencing nil pointer fields.

OIDC and the CA certificate

The OIDC provider doesn't use the common CA by default. You must explicitly supply the CA certificate.

Upgrading the admin cluster from 1.5 to 1.6.0 breaks 1.5 user clusters that use an OIDC provider and have no value for authentication.oidc.capath in the user cluster configuration file.

To work around this issue, run the following script:

USER_CLUSTER_KUBECONFIG=YOUR_USER_CLUSTER_KUBECONFIG

IDENTITY_PROVIDER=YOUR_OIDC_PROVIDER_ADDRESS

openssl s_client -showcerts -verify 5 -connect $IDENTITY_PROVIDER:443 < /dev/null | awk '/BEGIN CERTIFICATE/,/END CERTIFICATE/{ if(/BEGIN CERTIFICATE/){i++}; out="tmpcert"i".pem"; print >out}'

ROOT_CA_ISSUED_CERT=$(ls tmpcert*.pem | tail -1)

ROOT_CA_CERT="/etc/ssl/certs/$(openssl x509 -in $ROOT_CA_ISSUED_CERT -noout -issuer_hash).0"

cat tmpcert*.pem $ROOT_CA_CERT > certchain.pem CERT=$(echo $(base64 certchain.pem) | sed 's\ \\g') rm tmpcert1.pem tmpcert2.pem

kubectl --kubeconfig $USER_CLUSTER_KUBECONFIG patch clientconfig default -n kube-public --type json -p "[{ \"op\": \"replace\", \"path\": \"/spec/authentication/0/oidc/certificateAuthorityData\", \"value\":\"${CERT}\"}]"

Replace the following:

  • YOUR_OIDC_IDENTITY_PROVICER: The address of your OIDC provider:

  • YOUR_YOUR_USER_CLUSTER_KUBECONFIG: The path of your user cluster kubeconfig file.

gkectl check-config validation fails: can't find F5 BIG-IP partitions

Symptoms

Validation fails because F5 BIG-IP partitions can't be found, even though they exist.

Potential causes

An issue with the F5 BIG-IP API can cause validation to fail.

Resolution

Try running gkectl check-config again.

Disruption for workloads with PodDisruptionBudgets

Upgrading clusters can cause disruption or downtime for workloads that use PodDisruptionBudgets (PDBs).

Nodes fail to complete their upgrade process

If you have Anthos Service Mesh or OSS Istio installed on your cluster, depending on your PodDisruptionBudget settings for the Istio components, user nodes might fail to upgrade to the control plane version after repeated attempts. To prevent this failure, we recommend that you increase the Horizontal Pod Autoscaling minReplicas setting from 1 to 2 for the components in the istio-system namespace before you upgrade. This will ensure that you always have an instance of the ASM control plane running.

If you have Anthos Service Mesh 1.5+ or OSS Istio 1.5+:

kubectl patch hpa -n istio-system istio-ingressgateway -p '{"spec":{"minReplicas": 2}}' --type=merge
kubectl patch hpa -n istio-system istiod -p '{"spec":{"minReplicas": 2}}' --type=merge

If you have Anthos Service Mesh 1.4.x or OSS Istio 1.4.x:

kubectl patch hpa -n istio-system istio-galley -p '{"spec":{"minReplicas": 2}}' --type=merge
kubectl patch hpa -n istio-system istio-ingressgateway -p '{"spec":{"minReplicas": 2}}' --type=merge
kubectl patch hpa -n istio-system istio-nodeagent -p '{"spec":{"minReplicas": 2}}' --type=merge
kubectl patch hpa -n istio-system istio-pilot -p '{"spec":{"minReplicas": 2}}' --type=merge
kubectl patch hpa -n istio-system istio-sidecar-injector -p '{"spec":{"minReplicas": 2}}' --type=merge

Log Forwarder makes an excessive number of OAuth 2.0 requests

With Anthos clusters on VMware, version 1.7.1, you might experience issues with Log Forwarder consuming memory by making excessive OAuth 2.0 requests. Here is a workaround, in which you downgrade the stackdriver-operator version, clean up the disk, and restart Log Forwarder.

Step 0: Download images to your private registry if appropriate

If you use a private registry, follow these steps to download these images to your private registry before proceeding. Omit this step if you do not use a private registry.

Replace PRIVATE_REGISTRY_HOST with the hostname or IP address of your private Docker registry.

stackdriver-operator

docker pull gcr.io/gke-on-prem-release/stackdriver-operator:v0.0.440

docker tag gcr.io/gke-on-prem-release/stackdriver-operator:v0.0.440 \
    PRIVATE_REGISTRY_HOST/stackdriver-operator:v0.0.440

docker push PRIVATE_REGISTRY_HOST/stackdriver-operator:v0.0.440

fluent-bit

docker pull gcr.io/gke-on-prem-release/fluent-bit:v1.6.10-gke.3

docker tag gcr.io/gke-on-prem-release/fluent-bit:v1.6.10-gke.3 \
    PRIVATE_REGISTRY_HOST/fluent-bit:v1.6.10-gke.3

docker push PRIVATE_REGISTRY_HOST/fluent-bit:v1.6.10-gke.3

prometheus

docker pull gcr.io/gke-on-prem-release/prometheus:2.18.1-gke.0

docker tag gcr.io/gke-on-prem-release/prometheus:2.18.1-gke.0 \
    PRIVATE_REGISTRY_HOST/prometheus:2.18.1-gke.0

docker push PRIVATE_REGISTRY_HOST/prometheus:2.18.1-gke.0

Step 1: Downgrade the stackdriver-operator version

  • Run the following command to downgrade your version of stackdriver-operator.
kubectl  --kubeconfig [CLUSTER_KUBECONFIG] -n kube-system patch deployment stackdriver-operator -p \
  '{"spec":{"template":{"spec":{"containers":[{"name":"stackdriver-operator","image":"gcr.io/gke-on-prem-release/stackdriver-operator:v0.0.440"}]}}}}'

Step 2: Clean up the disk buffer for Log Forwarder

  • Deploy the DaemonSet in the cluster to clean up the buffer.
apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: fluent-bit-cleanup
  namespace: kube-system
spec:
  selector:
    matchLabels:
      app: fluent-bit-cleanup
  template:
    metadata:
      labels:
        app: fluent-bit-cleanup
    spec:
      containers:
      - name: fluent-bit-cleanup
        image: debian:10-slim
        command: ["bash", "-c"]
        args:
        - |
          rm -rf /var/log/fluent-bit-buffers/
          echo "Fluent Bit local buffer is cleaned up."
          sleep 3600
        volumeMounts:
        - name: varlog
          mountPath: /var/log
        securityContext:
          privileged: true
      tolerations:
      - key: "CriticalAddonsOnly"
        operator: "Exists"
      - key: node-role.kubernetes.io/master
        effect: NoSchedule
      - key: node-role.gke.io/observability
        effect: NoSchedule
      volumes:
      - name: varlog
        hostPath:
          path: /var/log
  • Verify the disk buffer is cleaned up.
kubectl --kubeconfig [CLUSTER_KUBECONFIG] logs -n kube-system -l app=fluent-bit-cleanup | grep "cleaned up" | wc -l

The output shows the number of nodes in the cluster.

kubectl --kubeconfig [CLUSTER_KUBECONFIG] -n kube-system get pods -l app=fluent-bit-cleanup --no-headers | wc -l

The output shows the number of nodes in the cluster.

  • Delete the cleanup DaemonSet.
kubectl --kubeconfig [CLUSTER_KUBECONFIG] -n kube-system delete ds fluent-bit-cleanup

Step 3: Restart Log Forwarder

kubectl --kubeconfig [CLUSTER_KUBECONFIG] -n kube-system rollout restart ds/stackdriver-log-forwarder

Logs and metrics are not sent to project specified by stackdriver.projectID

In Anthos clusters on VMware 1.7, logs are sent to the parent project of the service account specified in the stackdriver.serviceAccountKeyPath field of your cluster configuration file. The value of stackdriver.projectID is ignored. This issue will be fixed in an upcoming release.

As a workaround, view logs in the parent project of your logging-monitoring service account.

Renewal of certificates might be required before an admin cluster upgrade

Before you begin the admin cluster upgrade process, you should make sure that your admin cluster certificates are currently valid, and renew these certificates if they are not.

Admin cluster certificate renewal process

  1. Make sure that OpenSSL is installed on the admin workstation before you begin.

  2. Get the IP address and SSH keys for the admin master node:

    kubectl --kubeconfig [ADMIN_CLUSTER_KUBECONFIG] get secrets -n kube-system sshkeys \
    -o jsonpath='{.data.vsphere_tmp}' | base64 -d > \
    ~/.ssh/admin-cluster.key && chmod 600 ~/.ssh/admin-cluster.key
    
    export MASTER_NODE_IP=$(kubectl --kubeconfig [ADMIN_CLUSTER_KUBECONFIG] get nodes -o \
    jsonpath='{.items[*].status.addresses[?(@.type=="ExternalIP")].address}' \
    --selector='node-role.kubernetes.io/master')
    
  3. Check if the certificates are expired:

    ssh -i ~/.ssh/admin-cluster.key ubuntu@"${MASTER_NODE_IP}" \
    "sudo kubeadm alpha certs check-expiration"
    

    If the certificates are expired, you must renew them before upgrading the admin cluster.

  4. Back up old certificates:

    This is an optional, but recommended, step.

    # ssh into admin master
    ssh -i ~/.ssh/admin-cluster.key ubuntu@"${MASTER_NODE_IP}"
    
    # on admin master
    sudo tar -czvf backup.tar.gz /etc/kubernetes
    logout
    
    # on worker node
    sudo scp -i ~/.ssh/admin-cluster.key \
    ubuntu@"${MASTER_NODE_IP}":/home/ubuntu/backup.tar.gz .
    
  5. Renew the certificates with kubeadm:

     # ssh into admin master
     ssh -i ~/.ssh/admin-cluster.key ubuntu@"${MASTER_NODE_IP}"
     # on admin master
     sudo kubeadm alpha certs renew all
     

  6. Restart the admin master node:

      # on admin master
      cd /etc/kubernetes
      sudo mkdir tempdir
      sudo mv manifests/*.yaml tempdir/
      sleep 5
      echo "remove pods"
      # ensure kubelet detect those change remove those pods
      # wait until the result of this command is empty
      sudo docker ps | grep kube-apiserver
    
      # ensure kubelet start those pods again
      echo "start pods again"
      sudo mv tempdir/*.yaml manifests/
      sleep 30
      # ensure kubelet start those pods again
      # should show some results
      sudo docker ps | grep -e kube-apiserver -e kube-controller-manager -e kube-scheduler -e etcd
    
      # clean up
      sudo rm -rf tempdir
    
      logout
     
  7. Because the admin cluster kubeconfig file also expires if the admin certificates expire, you should back up this file before expiration.

    • Back up the admin cluster kubeconfig file:

      ssh -i ~/.ssh/admin-cluster.key ubuntu@"${MASTER_NODE_IP}" 
      "sudo cat /etc/kubernetes/admin.conf" > new_admin.conf vi [ADMIN_CLUSTER_KUBECONFIG]

    • Replace client-certificate-data and client-key-data in kubeconfig with client-certificate-data and client-key-data in the new_admin.conf file that you created.

  8. You must validate the renewed certificates, and validate the certificate of kube-apiserver.

    • Check certificates expiration:

      ssh -i ~/.ssh/admin-cluster.key ubuntu@"${MASTER_NODE_IP}" 
      "sudo kubeadm alpha certs check-expiration"

    • Check certificate of kube-apiserver:

      # Get the IP address of kube-apiserver
      cat [ADMIN_CLUSTER_KUBECONFIG] | grep server
      # Get the current kube-apiserver certificate
      openssl s_client -showcerts -connect : 
      | sed -ne '/-BEGIN CERTIFICATE-/,/-END CERTIFICATE-/p'
      > current-kube-apiserver.crt # check expiration date of this cert openssl x509 -in current-kube-apiserver.crt -noout -enddate