Update and patch device

The instructions on this page take you through the steps you must perform to update and patch your Google Distributed Cloud (GDC) air-gapped appliance device.

Before you begin

Make sure you meet the requirements listed in this section before following the Upgrade instructions.

You must first complete the following actions regarding the gdcloud command-line interface (CLI) tools:

1. Download the gdcloud CLI tools.
2. Install the gdcloud CLI tools.
3. Upgrade the gdcloud CLI tools as required.
4. Complete user authentication.

Before you run the scripts on the Upgrade instructions section, verify the following prerequisites:

• Your upgrade machine adheres to the corresponding requirements.
• Your appliance is set up and configured.
• You have the required roles to perform these updates.

Upgrade machine requirements and configuration

Follow these steps to meet requirements and enable the configuration of your upgrade machine:

1. Review the Laptop prerequisites to ensure that the laptop you use for the update and patch upgrades meets physical specifications and other requirements.

2. Securely receive the credentials for your initial admin account.

   Running the gdcloud upgrade command automatically configures your upgrade machine or laptop to the network.

3. Complete the configuration: connect your laptop to the uplink management port (port 12) on the switch.

Get your required roles

To perform the upgrades, ask your Organization IAM Admin to grant you the following predefined role:

• Upgrade Debugger

Upgrade instructions

Follow the instructions of this section to configure your access to the GDC, transfer the artifacts to your laptop, and start the upgrade.

Preparation

Follow these steps to configure the access to the GDC:

1. Configure the GDC URL that you used to sign in:

   /root/gdch/release/gdcloud config set
     core/organization_console_url
     `GDC_URL`
   

2. Ensure the web-tls certificate is available in the upgrade machine.

   Follow the CLI steps in Sign in to obtain the web-tls certificate. The following instructions require that the certificate is available at /tmp/web-tls-ca.cert in the upgrade machine.

3. Ensure that /root/gdch/release is present in PATH, so that the upgrade command finds the needed binaries:

   export PATH=/root/gdch/release:$PATH
   

Transfer the artifacts to the upgrade machine

You must have access to the Cloud Storage bucket.

Follow these steps to transfer the artifacts to the laptop you use for the update and patch upgrades:

1. Download the package based on the desired patch version:

   VERSION=x.x.x-gdch.x
   PRODUCT=prod_te
   
   PUBLIC_KEY=$(cat <<-PUBEND
   -----BEGIN PUBLIC KEY-----
   MFkwEwYHKoZIzj0CAQYIKoZIzj0DAQcDQgAEWZrGCUaJJr1H8a36sG4UUoXvlXvZ
   wQfk16sxprI2gOJ2vFFggdq3ixF2h4qNBt0kI7ciDhgpwS8t+/960IsIgw==
   -----END PUBLIC KEY-----
   PUBEND
   )
   
   echo "${PUBLIC_KEY}" > "key.pub"
   DOWNLOADER=gdch-downloader-${PRODUCT}-$VERSION.sh
   
   gsutil cp "gs://${GCS_BUCKET:-private-cloud-release}/$VERSION/$DOWNLOADER*" .
      openssl dgst -sha256 -verify "key.pub" -signature "${DOWNLOADER}.sig" ${DOWNLOADER} && chmod +x $DOWNLOADER && ./$DOWNLOADER --skip-unzip
   

   This command fetches the update package and the downloader script to the current directory, for example, /home/download.

2. Run tree -L 3 /home/download.

   Example output:

   /home/download
   ├── gdch
   │   └── x.x.x-gdch.x
   │       ├── bootstrapper
   │       │   ├── bootstrapper.iso
   │       │   └── bootstrapper.iso.sig
   │       ├── gdch-prod_te-x.x.x-gdch.x-sbom.tar.gz
   │       ├── gdch-prod_te-x.x.x-gdch.x-sbom.tar.gz.sig
   │       ├── prod_te_gdch.tar.gz
   │       └── prod_te_gdch.tar.gz.sig
   ├── gdch-downloader-prod_te-x.x.x-gdch.x.sh
   ├── gdch-downloader-prod_te-x.x.x-gdch.x.sh.1.sig
   ├── gdch-downloader-prod_te-x.x.x-gdch.x.sh.sig
      └── key.pub
   

3. Copy the update to the air-gapped environment, using your USB drive.

   Copy both the downloader script and the gdch directory that you downloaded to your USB drive in step 1 to the upgrade machine in the air-gapped environment.

4. In the upgrade machine, verify the contents and extract the files using the same information you used to download the package. For example, the following code copies the package to the /root directory.

   Unpack the package:

   VERSION=x.x.x-gdch.x
   PRODUCT=prod_te
   DOWNLOADER=gdch-downloader-${PRODUCT}-$VERSION.sh
   openssl dgst -sha256 -verify "key.pub" -signature "${DOWNLOADER}.sig" ${DOWNLOADER} && chmod +x $DOWNLOADER && ./$DOWNLOADER --skip-download
   

   The package is unpacked at the /root/gdch/full-release-x.x.x-gdch.x path.

5. Create a symlink at /root/gdch/release.

   ln -sf /root/gdch/full-release-x.x.x-gdch.x /root/gdch/release
   

   The gdcloud CLI is available at the /root/gdch/release/gdcloud path.

6. Confirm that the version matches the one that you set in VERSION:

   /root/gdch/release/gdcloud version
   

   Sample output:

   gdcloud version: 1.8.0-gdch.0
   

Start the upgrade

Start the appliance upgrade:

gdcloud appliance upgrade
  --login-config-cert=/tmp/root-web-tls-ca.cert
  --config /root/gdch/cell/CELLCFG_FILE_NAME
  --data-interface NETWORK_INTERFACE_NAME

• Replace CELLCFG_FILE_NAME with the cellcfg filename that you saved from the Configure the appliance instructions.
• Replace NETWORK_INTERFACE_NAME with the network interface name on your upgrade machine where port 12 of the switch is connected.

The command runs the following steps:

1. Run preflight checks on the upgrade machine.
2. Set up the network of the upgrade machine.
3. Connect to the admin cluster and perform authentication.
4. Push the artifacts to the artifact registry.
5. Upgrade the appliance software and firmware, comprising:
   1. GDC clusters.
   2. GDC services.
   3. The operating systems (OSes) of physical and virtual nodes.
   4. The switch.

The command drives the updates in sequence, and waits for each step to complete. After completing the steps and if the upgrade is successful, you see a message similar to the following:

The GDC appliance has been successfully upgraded to version X.

Upgrade the firmware version of servers

The firmware package is in the /root/gdch/release/hpe_firmware release path. Run the following instructions, one at a time on all of the servers.

1. Prepare the firmware URL. Run this command from the laptop:

   docker run -d --rm -ti --name firmware-update -p 4321:80  -v ${FW_DIR}:/usr/share/nginx/html ${NGINX_IMAGE_NAME}
   

2. Export the laptop or the bootstrapper mgmt IP:

   export LAPTOP_MGMT_IP=LAPTOP_MGMT_IP
   

3. Prepare the ILO credentials:

   ILO_USER=
   ILO_PASS=
   ILO_IP=
   

4. Check the current active firmware version:

   curl -kqs -u ${ILO_USER}:${ILO_PASS}
   https://${ILO_IP}/redfish/v1/UpdateService/FirmwareInventory/\?\$expand=. | jq '.Members[] |
   select(.Description=="SystemBMC") | .Version'
   

5. Update iLO firmware:

   curl -k -u ${ILO_USER}:${ILO_PASS} -H "Content-Type: application/json" -H "OData-Version: 4.0"
   -X POST --data '{"ImageURI":"http://${LAPTOP_MGMT_IP}:4321/FIRMWARE_PACKAGE.fwpkg"}'
   https://${ILO_IP}/redfish/v1/UpdateService/Actions/UpdateService.SimpleUpdate | jq
   

6. Check the update task status:

    curl -kqs -u ${ILO_USER}:${ILO_PASS} https://${ILO_IP}/redfish/v1/TaskService/Tasks/\?\$expand=. | jq '.Members | last'
    ```
   

7. iLO triggers a self reset. Wait until the action is complete and then verify that the firmware version is active:

   curl -kqs -u ${ILO_USER}:${ILO_PASS} https://${ILO_IP}/redfish/v1/UpdateService/FirmwareInventory/\?\$expand=. | jq '.Members[]
   | select(.Description=="SystemBMC") | .Version'
   

Upgrade the BIOS

1. Check the current active BIOS version:

   curl -kqs -u ${ILO_USER}:${ILO_PASS} https://${ILO_IP}/redfish/v1/UpdateService/FirmwareInventory/\?\$expand=. | jq '.Members[]
   | select(.Description=="SystemRomActive") | .Version'
   

2. Verify the server power state:

   curl -kqs -u ${ILO_USER}:${ILO_PASS} https://${ILO_IP}/redfish/v1/Systems/1 | jq '.PowerState'
   

3. The server must be powered off for the next step. If the power is on, initiate the power off:

   curl -k -u ${ILO_USER}:${ILO_PASS} -H "Content-Type: application/json" -H "OData-Version: 4.0"
   -X POST https://${ILO_IP}/redfish/v1/Systems/1/Actions/ComputerSystem.Reset --data
   '{"ResetType":"ForceOff"}' | jq .
   

4. Verify the server power state. The server might take several minutes to power off:

   curl -kqs -u ${ILO_USER}:${ILO_PASS} https://${ILO_IP}/redfish/v1/Systems/1 | jq '.PowerState'
   

5. Update the BIOS:

   curl -k -u ${ILO_USER}:${ILO_PASS} -H "Content-Type: application/json" -H "OData-Version: 4.0"
   -X POST --data '{"ImageURI":"http://${LAPTOP_MGMT_IP}:4321/FIRMWARE_PACKAGE.signed.flash"}'
   https://${ILO_IP}/redfish/v1/UpdateService/Actions/UpdateService.SimpleUpdate | jq .
   

6. Check the update task status:

   curl -kqs -u ${ILO_USER}:${ILO_PASS} https://${ILO_IP}/redfish/v1/TaskService/Tasks/\?\$expand=. | jq '.Members | last'
   

7. After the task is completed, initiate the power on:

   curl -k -u ${ILO_USER}:${ILO_PASS} -H "Content-Type: application/jsonls " -H "OData-Version: 4.0"
   -X POST https://${ILO_IP}/redfish/v1/Systems/1/Actions/ComputerSystem.Reset --data
   '{"ResetType":"On"}' | jq .
   

8. Check the current active BIOS version:

   curl -kqs -u ${ILO_USER}:${ILO_PASS} https://${ILO_IP}/redfish/v1/UpdateService/FirmwareInventory/\?\$expand=. | jq '.Members[]
   | select(.Description=="SystemRomActive") | .Version'
   

A successful task creation and task execution looks like this:

   ------------------ Success Task Creation ---------------------
   {
   "@odata.context": "/redfish/v1/$metadata#Task.Task",
   "@odata.etag": "W/\"78B88FBB\"",
   "@odata.id": "/redfish/v1/TaskService/Tasks/4",
   "@odata.type": "#Task.v1_3_0.Task",
   "Id": "4",
   "Description": "iLO Task",
   "Messages": [],
   "Name": "Task 4",
   "Payload": {
      "HttpOperation": "POST",
      "JsonBody": "{\"ImageURI\":\"http://xyz.fwpkg\"}",
      "TargetUri": "/redfish/v1/UpdateService/Actions/UpdateService.SimpleUpdate"
   },
   "StartTime": "2022-10-25T16:03:28Z",
   "TaskMonitor": "/redfish/v1/TaskService/TaskMonitors/4",
   "TaskState": "New"
   }

   ------------------ Success Task ---------------------
   {
   "@odata.context": "/redfish/v1/$metadata#Task.Task",
   "@odata.id": "/redfish/v1/TaskService/Tasks/4/",
   "@odata.type": "#Task.v1_3_0.Task",
   "Id": "4",
   "Description": "iLO Task",
   "EndTime": "2022-10-25T16:03:55Z",
   "Messages": [
      {
         "MessageId": "Base.1.4.Success"
      }
   ],
   "Name": "Task 4",
   "Payload": {
      "HttpOperation": "POST",
      "JsonBody": "{\"ImageURI\":\"http://xyz.fwpkg\"}",
      "TargetUri": "/redfish/v1/UpdateService/Actions/UpdateService.SimpleUpdate"
   },
   "StartTime": "2022-10-25T16:03:29Z",
   "TaskMonitor": "/redfish/v1/TaskService/TaskMonitors/4/",
   "TaskState": "Completed",
   "TaskStatus": "OK"
   }