Upgrading Apigee hybrid to version 1.10

This procedure covers upgrading from Apigee hybrid version 1.9.x to Apigee hybrid version 1.10.5 and from previous releases of hybrid 1.10.x to version 1.10.5.

Use the same procedures for minor version upgrades (for example version 1.9 to 1.10) and for patch release upgrades (for example 1.10.0 to 1.10.5).

Upgrading to version 1.10.5 overview

The procedures for upgrading Apigee hybrid are organized in the following sections:

  1. Prepare to upgrade.
  2. Install hybrid runtime version 1.10.5.

Prerequisites

These upgrade instructions assume you have Apigee hybrid version 1.9.x installed and wish to upgrade it to version 1.10.5. If you are updating from an earlier version, see the instructions for Upgrading Apigee hybrid to version 1.9.

Prepare to upgrade to version 1.10

  1. These instructions use the environment variable APIGEECTL_HOME for the directory in your file system where you have installed apigeectl. If needed, change directory into your apigeectl directory and define the variable with the following command:

    Linux 

    export APIGEECTL_HOME=$PWD
echo $APIGEECTL_HOME

    Mac OS 

    export APIGEECTL_HOME=$PWD
echo $APIGEECTL_HOME

    Windows 

    set APIGEECTL_HOME=%CD%
echo %APIGEECTL_HOME%
  2. Make a backup copy of your version 1.9 $APIGEECTL_HOME/ directory. For example: 
    tar -czvf $APIGEECTL_HOME/../apigeectl-v1.9-backup.tar.gz $APIGEECTL_HOME
  3. Back up your Cassandra database following the instructions in Cassandra backup and recovery.

Upgrade your Kubernetes version

Check your Kubernetes platform version and if needed, upgrade your Kubernetes platform to a version that is supported by both hybrid 1.9 and hybrid 1.10. Follow your platform's documentation if you need help.

Install the hybrid 1.10.5 runtime

  1. Be sure you are in the hybrid base directory (the parent of the directory where the apigeectl executable file is located): 
    cd $APIGEECTL_HOME/..

  2. Download the release package for your operating system using the following command. Be sure to select your platform in the following table:

    Linux

    Linux 64 bit:

    curl -LO \
  https://storage.googleapis.com/apigee-release/hybrid/apigee-hybrid-setup/1.10.5/apigeectl_linux_64.tar.gz

    Mac OS

    Mac 64 bit:

    curl -LO \
  https://storage.googleapis.com/apigee-release/hybrid/apigee-hybrid-setup/1.10.5/apigeectl_mac_64.tar.gz

    Windows

    Windows 64 bit:

    curl -LO ^
  https://storage.googleapis.com/apigee-release/hybrid/apigee-hybrid-setup/1.10.5/apigeectl_windows_64.zip
  3. Rename your current apigeectl/ directory to a backup directory name. For example:

    Linux 

    mv $APIGEECTL_HOME/ $APIGEECTL_HOME-v1.9/

    Mac OS 

    mv $APIGEECTL_HOME/ $APIGEECTL_HOME-v1.9/

    Windows 

    rename %APIGEECTL_HOME% %APIGEECTL_HOME%-v1.9

  4. Extract the downloaded gzip file contents into your hybrid base directory. The hybrid base directory is the directory where the renamed apigeectl-v1.9 directory is located:

    Linux 

    tar xvzf filename.tar.gz -C ./

    Mac OS 

    tar xvzf filename.tar.gz -C ./

    Windows 

    tar xvzf filename.zip -C ./

  5. The tar contents are, by default, expanded into a directory with the version and platform in its name. For example: ./apigeectl_1.10.5-xxxxxxx_linux_64. Rename that directory to apigeectl using the following command:

    Linux 

    mv apigeectl_1.10.5-xxxxxxx_linux_64 apigeectl

    Mac OS 

    mv apigeectl_1.10.5-xxxxxxx_mac_64 apigeectl

    Windows 

    rename apigeectl_1.10.5-xxxxxxx_windows_64 apigeectl
  6. Change to the apigeectl directory: 
    cd ./apigeectl

    This directory is the apigeectl home directory. It is where the apigeectl executable command is located.

  7. These instructions use the environment variable $APIGEECTL_HOME for the directory in your file system where the apigeectl utility is installed. If needed, change directory into your apigeectl directory and define the variable with the following command:

    Linux 

    export APIGEECTL_HOME=$PWD
    echo $APIGEECTL_HOME

    Mac OS 

    export APIGEECTL_HOME=$PWD
    echo $APIGEECTL_HOME

    Windows 

    set APIGEECTL_HOME=%CD%
    echo %APIGEECTL_HOME%
  8. Verify the version of apigeectl with the version command: 
    ./apigeectl version
    Version: 1.10.5
  9. Create a hybrid-base-directory/hybrid-files directory, and then move into it. The hybrid-filesdirectory is where configuration files such as the overrides file, certs, and service accounts are located. For example:

    Linux 

    mkdir $APIGEECTL_HOME/../hybrid-files
    cd $APIGEECTL_HOME/../hybrid-files

    Mac OS 

    mkdir $APIGEECTL_HOME/../hybrid-files
    cd $APIGEECTL_HOME/../hybrid-files

    Windows 

    mkdir %APIGEECTL_HOME%/../hybrid-files
    cd %APIGEECTL_HOME%/../hybrid-files
  10. Verify that kubectl is set to the correct context using the following command. The current context should be set to the cluster in which you are upgrading Apigee hybrid. 
    kubectl config get-contexts | grep \*
  11. In the hybrid-files directory:
    1. Update the following symbolic links to $APIGEECTL_HOME. These links allow you to run the newly installed apigeectl command from inside the hybrid-files directory: 
      ln -nfs $APIGEECTL_HOME/tools tools
ln -nfs $APIGEECTL_HOME/config config
ln -nfs $APIGEECTL_HOME/templates templates
ln -nfs $APIGEECTL_HOME/plugins plugins
    2. To check that the symlinks were created correctly, execute the following command and make sure the link paths point to the correct locations: 
      ls -l | grep ^l
  12. Make the following change to your overrides.yaml file to enable the apigee-operator chart ot use the correct tag, 1.10.5-hotfix.1: 
      ao:
    image:
      url: "gcr.io/apigee-release/hybrid/apigee-operators"
      tag: "1.10.5-hotfix.1"
  13. Do a dry run initialization to check for errors: 
    ${APIGEECTL_HOME}/apigeectl init -f OVERRIDES_FILE --dry-run=client

    Where OVERRIDES_FILE is the name of your overrides file, for example ./overrides/overrides.yaml.

  14. If there are no errors, initialize hybrid 1.10.5: 
    $APIGEECTL_HOME/apigeectl init -f OVERRIDES_FILE
  15. Check the initialization status: 
    $APIGEECTL_HOME/apigeectl check-ready -f OVERRIDES_FILE

    On success, the output says: All containers ready. 

    kubectl describe apigeeds -n apigee

    In the output, look for State: running.

  16. Check for errors with a dry run of the apply command using the --dry-run flag: 
    $APIGEECTL_HOME/apigeectl apply -f OVERRIDES_FILE --dry-run=client
  17. If there are no errors, apply your overrides. Select and follow the instructions for production environments or non-prod environments, depending on your installation.

    Production

    For production environments, upgrade each hybrid component individually, and check the status of the upgraded component before proceeding to the next component.

    1. Be sure you are in the hybrid-files directory.
    2. Apply your overrides to upgrade Cassandra: 
      $APIGEECTL_HOME/apigeectl apply -f OVERRIDES_FILE --datastore
    3. Check completion: 
      $APIGEECTL_HOME/apigeectl check-ready -f OVERRIDES_FILE

      Proceed to the next step only when the pods are ready.

    4. Apply your overrides to upgrade Telemetry components and check completion: 
      $APIGEECTL_HOME/apigeectl apply -f OVERRIDES_FILE --telemetry
      $APIGEECTL_HOME/apigeectl check-ready -f OVERRIDES_FILE
    5. Bring up Redis components: 
      $APIGEECTL_HOME/apigeectl apply -f OVERRIDES_FILE --redis
    6. Apply your overrides to upgrade the org-level components (MART, Watcher and Apigee Connect) and check completion: 
      $APIGEECTL_HOME/apigeectl apply -f OVERRIDES_FILE --org
      $APIGEECTL_HOME/apigeectl check-ready -f OVERRIDES_FILE
    7. Apply your overrides to upgrade your environments. You have two choices:
      • Environment by environment: Apply your overrides to one environment at a time and check completion. Repeat this step for each environment: 
        $APIGEECTL_HOME/apigeectl apply -f OVERRIDES_FILE --env ENV_NAME
        $APIGEECTL_HOME/apigeectl check-ready -f OVERRIDES_FILE

        Where ENV_NAME is the name of the environment you are upgrading.

      • All environments at one time: Apply your overrides to all environments at once and check completion: 
        $APIGEECTL_HOME/apigeectl apply -f OVERRIDES_FILE --all-envs
        $APIGEECTL_HOME/apigeectl check-ready -f OVERRIDES_FILE
    8. Apply your overrides to upgrade the virtualhosts components and check completion: 
      $APIGEECTL_HOME/apigeectl apply -f OVERRIDES_FILE --settings virtualhosts
      $APIGEECTL_HOME/apigeectl check-ready -f OVERRIDES_FILE

    Non-prod

    In most non-production, demo, or experimental environments, you can apply the overrides to all components at once. If your non-production environment is large and complex or closely mimics a production environment, you may want to use the instructions for upgrading production environments.

    1. Be sure you are in the hybrid-files directory.
    2. $APIGEECTL_HOME/apigeectl apply -f OVERRIDES_FILE
    3. Check the status: 
      $APIGEECTL_HOME/apigeectl check-ready -f OVERRIDES_FILE

Rolling back an upgrade

Follow these steps to roll back a previous upgrade:

  1. Clean up completed jobs for the hybrid runtime namespace, where NAMESPACE is the namespace specified in your overrides file, if you specified a namespace. If not, the default namespace is apigee: 
    kubectl delete job -n NAMESPACE \
  $(kubectl get job -n NAMESPACE \
  -o=jsonpath='{.items[?(@.status.succeeded==1)].metadata.name}')
  2. Clean up completed jobs for the apigee-system namespace: 
    kubectl delete job -n apigee-system \
  $(kubectl get job -n apigee-system \
  -o=jsonpath='{.items[?(@.status.succeeded==1)].metadata.name}')
  3. Change the APIGEECTL_HOME variable to point to the directory that contains the previous version of apigeectl. For example: 
    export APIGEECTL_HOME=PATH_TO_PREVIOUS_APIGEECTL_DIRECTORY
  4. In the root directory of the installation you want to roll back to, run apigeectl apply, check the status of your pods, and then run apigeectl init. Be sure to use the original overrides file for the version you wish to roll back to:
    1. In the hybrid-files directory, run apigeectl apply: 
      $APIGEECTL_HOME/apigeectl apply -f ORIGINAL_OVERRIDES_FILE

      Where ORIGINAL_OVERRIDES_FILE is the relative path and filename of the overrides file for your previous version hybrid installation, for example, ./overrides/overrides1.9.yaml.

    2. Check the status of your pods: 
      kubectl -n NAMESPACE get pods

      Where NAMESPACE is your Apigee hybrid namespace.

    3. Check the status of apigeeds: 
      kubectl describe apigeeds -n apigee

      Your output should look something like: 

      Status:
  Cassandra Data Replication:
  Cassandra Pod Ips:
    10.8.2.204
  Cassandra Ready Replicas:  1
  Components:
    Cassandra:
      Last Successfully Released Version:
        Revision:  v1-f8aa9a82b9f69613
        Version:   v1
      Replicas:
        Available:  1
        Ready:      1
        Total:      1
        Updated:    1
      State:        running
  Scaling:
    In Progress:         false
    Operation:
    Requested Replicas:  0
  State:                 running

      Proceed to the next step only when the apigeeds pod is running.

    4. Run the following command to make note of what your new replica count values will be for the message processor after the upgrade. If these values do not match what you have set previously, change the values in your overrides file to match your previous configuration. 
      apigeectl apply -f ORIGINAL_OVERRIDES_FILE --dry-run=client --print-yaml --env ENV_NAME 2>/dev/null |grep "runtime:" -A 25 -B 1| grep "autoScaler" -A 2

      Your output should look something like: 

            autoScaler:
        minReplicas: 2
        maxReplicas: 10
    5. Run apigeectl init: 
      $APIGEECTL_HOME/apigeectl init -f ORIGINAL_OVERRIDES_FILE