Run the pre-upgrade tool

This document shows you how run a standalone tool in preparation for an upgrade. Before upgrading an admin or user cluster that is at Google Distributed Cloud version 1.9 and later versions, we recommend that you run the pre-upgrade tool.

To run the tool, use the bash script in this document that uses a hard-coded Secure Hash Algorithms (SHA). For each release of the tool, this document will be updated with the new SHA. The script creates a Kubernetes Job to run a specific version of preflight checks depending on the version that you are upgrading to.

Pre-upgrade checks

The tool checks the following before you upgrade a user cluster:

Category Description
Cluster Health
  • Validates the PDBs in all namespaces of the admin cluster.
  • Validates the PDBs in all namespaces of the user cluster.
  • Validates that a previous upgrade of the admin cluster finished successfully.
Configurations
  • Recommends the patch version to upgrade to.
  • Checks whether the component access SA key is wiped out as described in the related known issue.
  • If you are upgrading to 1.10, warns if you will need to apply the workaround in this known issue.

The tool checks the following before you upgrade an admin cluster:

Category Description
Cluster Health Validates the PodDisruptionBudgets (PDBs) in all namespaces of the admin cluster.
Configurations

Prepare to run the tool

  1. Upgrade your admin workstation if you have not done so.

  2. Run gkectl prepare to import OS images to vSphere if you have not done so:

    gkectl prepare \
    --bundle-path /var/lib/gke/bundles/gke-onprem-vsphere-TARGET_VERSION.tgz \
    --kubeconfig ADMIN_CLUSTER_KUBECONFIG

    Replace the following:

    • TARGET_VERSION: The Google Distributed Cloud patch version that you want to upgrade to. The format for the version number must be a complete patch version, like 1.13.10-gke.42.

    • ADMIN_CLUSTER_KUBECONFIG: The path to the admin cluster kubeconfig.

  3. If you are using a private registry, download the preflight container image with the provided docker digest and upload the image into the private registry. If you aren't using a private registry, skip to the next step.

    export SRC_IMAGE=gcr.io/gke-on-prem-release/preflight@sha256:9704315c6637750a014d0079ca04a8f97d0ca3735e175020377107c3181f6234
export DST_IMAGE=REGISTRY_ADDRESS/preflight:$(date +%Y-%m%d-%H%M%S)
docker pull $SRC_IMAGE
docker tag $SRC_IMAGE $DST_IMAGE
docker push $DST_IMAGE

    Replace REGISTRY_ADDRESS with the private registry address.

  4. In the following bash script, set values for these placeholders:

    • ADMIN_CLUSTER_KUBECONFIG: The path to the admin cluster kubeconfig.

    • REGISTRY_ADDRESS: If the admin cluster uses a private registry, this is the private registry address that you specified in the previous step. If you aren't using a private registry, specify the public registry: gcr.io/gke-on-prem-release

    #!/bin/bash
UPGRADE_TARGET_VERSION=${1}
CLUSTER_NAME=${2}
ADMIN_KUBECONFIG=ADMIN_CLUSTER_KUBECONFIG
REGISTRY_ADDRESS=REGISTRY_ADDRESS
pre_upgrade_namespace=kube-system
if [[ -z "$CLUSTER_NAME" ]]
then
  echo "Running the pre-ugprade tool before admin cluster upgrade"
else
  echo "Running the pre-ugprade tool before user cluster upgrade"
  pre_upgrade_namespace=$CLUSTER_NAME-gke-onprem-mgmt
fi
kubectl apply --kubeconfig ${ADMIN_KUBECONFIG} -f - <<EOF
apiVersion: v1
kind: ServiceAccount
metadata:
  name: pre-upgrade-job
  namespace: $pre_upgrade_namespace
EOF
kubectl apply --kubeconfig ${ADMIN_KUBECONFIG} -f - <<EOF
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  creationTimestamp: null
  name: pre-upgrade-job-rolebinding-in-$pre_upgrade_namespace
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: onprem-user-cluster-controller-role
subjects:
  - kind: ServiceAccount
    name: pre-upgrade-job
    namespace: $pre_upgrade_namespace
EOF
kubectl apply --kubeconfig ${ADMIN_KUBECONFIG} -f - <<EOF
apiVersion: batch/v1
kind: Job
metadata:
  name: pre-upgrade-$(date +%Y-%m%d-%H%M%S)
  namespace: $pre_upgrade_namespace
  labels:
    onprem.cluster.gke.io/job-usage: preflight
spec:
  ttlSecondsAfterFinished: 2592000
  backoffLimit: 2
  template:
    metadata:
      labels:
        onprem.cluster.gke.io/pod-usage: preflight
    spec:
      containers:
      - name: preflight
        image: $REGISTRY_ADDRESS/preflight@sha256:9704315c6637750a014d0079ca04a8f97d0ca3735e175020377107c3181f6234
        imagePullPolicy: Always
        command:
        - /preflight
        - --upgrade-target-version
        - "$UPGRADE_TARGET_VERSION"
        - --cluster-name
        - "$CLUSTER_NAME"
        - --scenario
        - pre-upgrade
      restartPolicy: Never
      serviceAccountName: pre-upgrade-job
      imagePullSecrets:
      - name: private-registry-creds
EOF

  5. Save the above bash script to a file called pre-upgrade.sh and make it executable:

    chmod +x pre-upgrade.sh

Run the script

  1. The arguments that you provide when you run the script depends on whether you are upgrading a user cluster or an admin cluster:

    • Before upgrading an admin cluster run the script as follows:
    ./pre-upgrade.sh TARGET_VERSION
    • Before upgrading a user cluster:
    ./pre-upgrade.sh TARGET_VERSION USER_CLUSTER_NAME

    Replace USER_CLUSTER_NAME with the name of the user cluster you will be upgrading.

    The output is similar to the following:

    job.batch/pre-upgrade-2023-0822-213551 created

  2. Run the following command on the Pods controlled by the job to get a list of validation results.

    kubectl logs -n JOB_NAMESPACE jobs/JOB_NAME \
    --kubeconfig ADMIN_CLUSTER_KUBECONFIG

    Replace the following:

    • JOB_NAME: This is the name of the job that the script outputs from the previous step.

    • JOB_NAMESPACE: The value that you set depends on whether you are upgrading an admin or user cluster. If you are upgrading an admin cluster, specify kube-system. If you are upgrading a user cluster, specify USER_CLUSTER_NAME-gke-onprem-mgmt.

    Wait a few minutes for the job to complete or to reach the backoff limit and fail. In the results, review the Reason for any check with the status Warning, Unknown or Failure to see if you can resolve the issue.

  3. Before upgrading clusters, run the following command to delete the Job:

    kubectl delete jobs JOB_NAME -n JOB_NAMESPACE \
    --kubeconfig ADMIN_CLUSTER_KUBECONFIG