This procedure covers upgrading from Apigee hybrid version 1.8.x to Apigee hybrid version 1.9.4 and from previous releases of hybrid 1.9.x to version 1.9.4.
Use the same procedures for minor version upgrades (for example version 1.8 to 1.9) and for patch release upgrades (for example 1.9.0 to 1.9.4).
If you are upgrading from Apigee hybrid version 1.7 or older, you must first upgrade to hybrid version 1.8 before upgrading to version 1.9.4. See the instructions for Upgrading Apigee hybrid to version 1.8.
Starting with version 1.9.0, Apigee hybrid only supports Apigee ingress gateway as the ingress layer.
Upgrading to version 1.9.4 overview
The procedures for upgrading Apigee hybrid are organized in the following sections:
Prerequisites
- These upgrade instructions assume you have Apigee hybrid version 1.8.x installed and wish to upgrade it to version 1.9.4. If you are updating from an earlier version, see the instructions for Upgrading Apigee hybrid to version 1.8.
- In Apigee hybrid version 1.8, we introduced Apigee ingress gateway as an alternative ingress layer
to Anthos Service Mesh. Starting with version 1.9.0, Apigee hybrid requires using
Apigee ingress gateway and no longer supports using Anthos Service Mesh for ingress. If the installation you
are upgrading from uses Anthos Service Mesh, you must first migrate to using Apigee ingress gateway before
upgrading to version 1.9.4.
Apigee ingress gateway uses a small subset of Anthos Service Mesh features for the ingress gateway. Management and upgrade of those features are handled automatically by Apigee hybrid. Therefore you do not need expertise with Anthos Service Mesh to install, upgrade, and manage the Apigee hybrid ingress gateway.
See Migrating to Apigee ingress gateway in the hybrid v1.8 documentation for instructions.
Prepare to upgrade to version 1.9
Back up your hybrid installation (recommended)
- These instructions use the environment variable APIGEECTL_HOME for the directory
in your file system where you have installed
apigeectl
. If needed, change directory into yourapigeectl
directory and define the variable with the following command:Linux
export APIGEECTL_HOME=$PWD
echo $APIGEECTL_HOME
Mac OS
export APIGEECTL_HOME=$PWD
echo $APIGEECTL_HOME
Windows
set APIGEECTL_HOME=%CD%
echo %APIGEECTL_HOME%
- Make a backup copy of your version 1.8
$APIGEECTL_HOME/
directory. For example:tar -czvf $APIGEECTL_HOME/../apigeectl-v1.8-backup.tar.gz $APIGEECTL_HOME
- Back up your Cassandra database following the instructions in Cassandra backup and recovery.
Add the Cloud Trace Agent role to the service account for the Apigee runtime. (Optional)
Optional: If you plan to use Cloud trace and you have not
already added the Cloud Trace Agent
role to your hybrid
v1.8 installation,
ensure your service account for your Apigee runtime services has the Cloud Trace Agent
Google Cloud IAM role (roles/cloudtrace.agent
).
For production environments, runtime service account is apigee-runtime
. For
non-production environments, runtime service account is apigee-non-prod
.
You can add the role in the Cloud console > IAM & Admin > Service accounts UI or with the following commands:
- Get the email address for your service account with the following command:
Production
gcloud iam service-accounts list --filter "apigee-runtime"
If the email address matches the pattern
apigee-runtime@$ORG_NAME.iam.gserviceaccount.com
, you can use that pattern in the next step.Non-Prod
gcloud iam service-accounts list --filter "apigee-non-prod"
If it matches the pattern
apigee-non-prod@$ORG_NAME.iam.gserviceaccount.com
, you can use that pattern in the next step. - Assign the Cloud Trace Agent role to the service account:
Production
gcloud projects add-iam-policy-binding $PROJECT_ID \ --member="serviceAccount:apigee-runtime@$PROJECT_ID.iam.gserviceaccount.com" \ --role="roles/cloudtrace.agent"
Non-Prod
gcloud projects add-iam-policy-binding $PROJECT_ID \ --member="serviceAccount:apigee-non-prod@$PROJECT_ID.iam.gserviceaccount.com" \ --role="roles/cloudtrace.agent"
Example
gcloud projects add-iam-policy-binding hybrid-example-project \ --member="serviceAccount:apigee-runtime@hybrid-example-project.iam.gserviceaccount.com" \ --role="roles/cloudtrace.agent"
Where: $PROJECT_ID is the name of the Google Cloud project where Apigee hybrid is installed.
Install Apigee ingress gateway if your installation uses Anthos Service Mesh
Starting with version 1.9, Apigee hybrid no longer supports using Anthos Service Mesh for ingress. If your hybrid installation is using Anthos Service Mesh, you must migrate your current installation to Apigee ingress gateway before installing hybrid version 1.9.
-
Add the
ingressGateways
property to your overrides file.Syntax
ingressGateways: - name: INGRESS_NAME replicaCountMin: REPLICAS_MIN replicaCountMax: REPLICAS_MAX resources: requests: cpu: CPU_COUNT_REQ memory: MEMORY_REQ limits: cpu: CPU_COUNT_LIMIT memory: MEMORY_LIMIT svcAnnotations: # optional. See Known issue 243599452. SVC_ANNOTATIONS_KEY: SVC_ANNOTATIONS_VALUE svcLoadBalancerIP: SVC_LOAD_BALANCER_IP # optional
Example
ingressGateways: - name: prod1 replicaCountMin: 2 replicaCountMax: 100 resources: requests: cpu: 1 memory: 1Gi limits: cpu: 2 memory: 2Gi svcAnnotations: # optional. See Known issue 243599452. networking.gke.io/load-balancer-type: "Internal" svcLoadBalancerIP: 198.252.0.123
- INGRESS_NAME is the name of the ingress deployment. This can be any name
that meets the following requirements:
- Have a maximum length of 17 characters
- Contain only lowercase alphanumeric characters, '-' or '.'
- Start with an alphanumeric character
- End with an alphanumeric character
ingressGateways[].name
in the Configuration property reference. - REPLICAS_MIN and REPLICAS_MAX are the minimum and maximum replica counts for
Apigee ingress gateway in your installation. For more information and default settings, see
ingressGateways[].replicaCountMin
andingressGateways[].replicaCountMax
in the Configuration property reference. - CPU_COUNT_REQ and MEMORY_REQ are the CPU and memory request for each
replica of Apigee ingress gateway in your installation.
For more information and default settings, see
ingressGateways[].resources.requests.cpu
andingressGateways[].resources.requests.memory
in the Configuration property reference. - CPU_COUNT_LIMIT and MEMORY_LIMIT Are the maximum CPU and memory limits for
each replica of Apigee ingress gateway in your installation.
For more information and default settings, see
ingressGateways[].resources.limits.cpu
andingressGateways[].resources.limits.memory
in the Configuration property reference. - SVC_ANNOTATIONS_KEY SVC_ANNOTATIONS_VALUE (optional):
This is a key-value pair that provides annotations for your default ingress service. Annotations are used by your cloud platform to help configure your hybrid installation, for example setting the loadbalancer type to either internal or external. For example:
ingressGateways: svcAnnotations: networking.gke.io/load-balancer-type: "Internal"
Annotations vary from platform to platform. Refer to your platform documentation for required and suggested annotations.
SeeingressGateways[].svcAnnotations
in the Configuration property reference. - SVC_LOAD_BALANCER_IP (optional) Allows you to assign a static IP address for your
load balancer. On platforms that support specifying the load balancer IP address, the load
balancer will be created with this IP address. On platforms that do not allow you to specify the
load balancer IP address, this property is ignored.
If you do not have a static IP address allocated for your load balancer, leave this property out of your overrides file.
SeeingressGateways[].svcLoadBalancerIP
in the Configuration property reference.
- INGRESS_NAME is the name of the ingress deployment. This can be any name
that meets the following requirements:
- Apply the changes to install Apigee ingress gateway with the following commands:
$APIGEECTL_HOME/apigeectl apply -f overrides/overrides.yaml
- Expose the Apigee ingress gateway. Follow the procedures in Expose Apigee ingress gateway.
- Test your new ingress gateway by calling a proxy. Ideally, test all crucial proxies you currently have deployed.
- To switch the traffic, update your DNS records to point to the IP address for your new Apigee ingress gateway.
Depending on your DNS provider, you might be able to gradually shift traffic to the new endpoint.
Tip: You can find the external IP address of Apigee ingress gateway with the following command: kubectl get svc -n apigee -l app=apigee-ingressgateway
Your output should look something like:
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE apigee-ingressgateway-prod-hybrid-37a39bd LoadBalancer 192.0.2.123 233.252.0.123 15021:32049/TCP,80:31624/TCP,443:30723/TCP 16h
- Ensure all the runtime traffic is working by monitoring your dashboards. Only proceed to the next step if everything is working as expected. Make sure no traffic is going through your old ingress gateway (Anthos Service Mesh), as the DNS update may be slow to propagate because of DNS caching.
- To stop Apigee from supplying configuration to Anthos Service Mesh, follow the steps in Stop supplying configuration to ASM in the Managing Apigee ingress gateway guide.
- Retest and monitor API proxy traffic.
- Follow the instructions in the Anthos Service Mesh documentation to Uninstall Anthos Service Mesh from the cluster.
Install the hybrid 1.9.4 runtime
- Be sure you are in the hybrid base directory (the parent of the directory where
the
apigeectl
executable file is located):cd $APIGEECTL_HOME/..
-
Download the release package for your operating system using the following command. Be sure to select your platform in the following table:
Linux
Linux 64 bit:
curl -LO \ https://storage.googleapis.com/apigee-release/hybrid/apigee-hybrid-setup/1.9.4/apigeectl_linux_64.tar.gz
Mac OS
Mac 64 bit:
curl -LO \ https://storage.googleapis.com/apigee-release/hybrid/apigee-hybrid-setup/1.9.4/apigeectl_mac_64.tar.gz
Windows
Windows 64 bit:
curl -LO ^ https://storage.googleapis.com/apigee-release/hybrid/apigee-hybrid-setup/1.9.4/apigeectl_windows_64.zip
- Rename your current
apigeectl/
directory to a backup directory name. For example:Linux
mv $APIGEECTL_HOME/ $APIGEECTL_HOME-v1.8/
Mac OS
mv $APIGEECTL_HOME/ $APIGEECTL_HOME-v1.8/
Windows
rename %APIGEECTL_HOME% %APIGEECTL_HOME%-v1.8
-
Extract the downloaded gzip file contents into your hybrid base directory. The hybrid base directory is the directory where the renamed
apigeectl-v1.8
directory is located:Linux
tar xvzf filename.tar.gz -C ./
Mac OS
tar xvzf filename.tar.gz -C ./
Windows
tar xvzf filename.zip -C ./
-
The tar contents are, by default, expanded into a directory with the version and platform in its name. For example:
./apigeectl_1.9.4-xxxxxxx_linux_64
. Rename that directory toapigeectl
using the following command:Linux
mv apigeectl_1.9.4-xxxxxxx_linux_64 apigeectl
Mac OS
mv apigeectl_1.9.4-xxxxxxx_mac_64 apigeectl
Windows
rename apigeectl_1.9.4-xxxxxxx_windows_64 apigeectl
-
Change to the
apigeectl
directory:cd ./apigeectl
This directory is the
apigeectl
home directory. It is where theapigeectl
executable command is located. - These instructions use the environment variable
$APIGEECTL_HOME
for the directory in your file system where theapigeectl
utility is installed. If needed, change directory into yourapigeectl
directory and define the variable with the following command:Linux
export APIGEECTL_HOME=$PWD
echo $APIGEECTL_HOME
Mac OS
export APIGEECTL_HOME=$PWD
echo $APIGEECTL_HOME
Windows
set APIGEECTL_HOME=%CD%
echo %APIGEECTL_HOME%
- Verify the version of
apigeectl
with theversion
command:./apigeectl version
Version: 1.9.4
- Move to the
hybrid-base-directory/hybrid-files
directory. Thehybrid-files
directory is where configuration files such as the overrides file, certs, and service accounts are located. For example:cd $APIGEECTL_HOME/../hybrid-files
- Verify that
kubectl
is set to the correct context using the following command. The current context should be set to the cluster in which you are upgrading Apigee hybrid.kubectl config get-contexts | grep \*
- In the
hybrid-files
directory:-
Update the following symbolic links to
$APIGEECTL_HOME
. These links allow you to run the newly installedapigeectl
command from inside thehybrid-files
directory:ln -nfs
$APIGEECTL_HOME
/tools toolsln -nfs
$APIGEECTL_HOME
/config configln -nfs
$APIGEECTL_HOME
/templates templatesln -nfs
$APIGEECTL_HOME
/plugins plugins -
To check that the symlinks were created correctly, execute the following command and make
sure the link paths point to the correct locations:
ls -l | grep ^l
-
Update the following symbolic links to
- Do a dry run initialization to check for errors:
${APIGEECTL_HOME}/apigeectl init -f OVERRIDES_FILE --dry-run=client
Where OVERRIDES_FILE is the name of your overrides file, for example
./overrides/overrides.yaml
. - If there are no errors, initialize hybrid 1.9.4:
$APIGEECTL_HOME/apigeectl init -f OVERRIDES_FILE
- Check the initialization status:
$APIGEECTL_HOME/apigeectl check-ready -f OVERRIDES_FILE
On success, the output says:
All containers ready.
kubectl describe apigeeds -n apigee
In the output, look for
State: running
. - Check for errors with a dry run of the
apply
command using the--dry-run
flag:$APIGEECTL_HOME/apigeectl apply -f OVERRIDES_FILE --dry-run=client
- If there are no errors, apply your overrides. Select and follow the instructions for production environments or
non-prod environments, depending on your installation.
Production
For production environments, upgrade each hybrid component individually, and check the status of the upgraded component before proceeding to the next component.
- Be sure you are in the
hybrid-files
directory. - Apply your overrides to upgrade Cassandra:
$APIGEECTL_HOME/apigeectl apply -f OVERRIDES_FILE --datastore
- Check completion:
$APIGEECTL_HOME/apigeectl check-ready -f OVERRIDES_FILE
Proceed to the next step only when the pods are ready.
- Apply your overrides to upgrade Telemetry components and check completion:
$APIGEECTL_HOME/apigeectl apply -f OVERRIDES_FILE --telemetry
$APIGEECTL_HOME/apigeectl check-ready -f OVERRIDES_FILE
- Bring up Redis components:
$APIGEECTL_HOME/apigeectl apply -f OVERRIDES_FILE --redis
- Apply your overrides to upgrade the org-level components (MART, Watcher and Apigee
Connect) and check completion:
$APIGEECTL_HOME/apigeectl apply -f OVERRIDES_FILE --org
$APIGEECTL_HOME/apigeectl check-ready -f OVERRIDES_FILE
- Apply your overrides to upgrade your environments. You have two choices:
- Environment by environment: Apply your overrides to one environment at a time and check completion. Repeat
this step for each environment:
$APIGEECTL_HOME/apigeectl apply -f OVERRIDES_FILE --env ENV_NAME
$APIGEECTL_HOME/apigeectl check-ready -f OVERRIDES_FILE
Where ENV_NAME is the name of the environment you are upgrading.
- All environments at one time: Apply your overrides to all environments at once and check completion:
$APIGEECTL_HOME/apigeectl apply -f OVERRIDES_FILE --all-envs
$APIGEECTL_HOME/apigeectl check-ready -f OVERRIDES_FILE
- Environment by environment: Apply your overrides to one environment at a time and check completion. Repeat
this step for each environment:
- Apply your overrides to upgrade the
virtualhosts
components and check completion:$APIGEECTL_HOME/apigeectl apply -f OVERRIDES_FILE --settings virtualhosts
$APIGEECTL_HOME/apigeectl check-ready -f OVERRIDES_FILE
Non-prod
In most non-production, demo, or experimental environments, you can apply the overrides to all components at once. If your non-production environment is large and complex or closely mimics a production environment, you may want to use the instructions for upgrading production environments.
- Be sure you are in the
hybrid-files
directory. $APIGEECTL_HOME/apigeectl apply -f OVERRIDES_FILE
- Check the status:
$APIGEECTL_HOME/apigeectl check-ready -f OVERRIDES_FILE
- Be sure you are in the
Install 1.9.4-hotfix.1
Follow these steps to install the hotfix release, 1.9.4-hotfix.1
:
- Before doing these steps, you must be on Apigee hybrid version 1.9.4 or a later version. If you are not on 1.9.4 or later, perform an upgrade to 1.9.4 before proceeding.
- Open your
overrides.yaml
file. - In the
istiod
stanza, change the version of the image tag (if present) to version1.17.7
. For example:istiod: image: url: "gcr.io/apigee-release/hybrid/apigee-asm-istiod" tag: "1.17.7-asm.0-distroless"
- Depending on how you chose to install Apigee hybrid, you may have an
ingressGateway
oringressGateways
stanza. Locate the stanza that appears in your overrides file and change the version of the image tag (if present) to version1.17.7
. For example, if you have aningressGateway
stanza:ingressGateway: image: url: "gcr.io/apigee-release/hybrid/apigee-asm-ingress" tag: "1.17.7-asm.0-distroless"
or, if you have an
ingressGateways
stanza:ingressGateways: - name: gateway1 image: url: "gcr.io/apigee-release/hybrid/apigee-asm-ingress" tag: "1.17.7-asm.0-distroless" ... - name: gateway2 image: url: "gcr.io/apigee-release/hybrid/apigee-asm-ingress" tag: "1.17.7-asm.0-distroless" ...
- Save the file.
- Execute the following command to initialize the
istiod
component:$APIGEECTL_HOME/apigeectl init -f OVERRIDES_FILE
$APIGEECTL_HOME/apigeectl check-ready -f OVERRIDES_FILE
- Execute the following command to apply changes to the Apigee ingress component(s). If you have
more than one organization, repeat this command for each one:
$APIGEECTL_HOME/apigeectl apply --org -f OVERRIDES_FILE
$APIGEECTL_HOME/apigeectl check-ready -f OVERRIDES_FILE
- Verify the status of your pods:
kubectl get pods -n YOUR_APIGEE_NAMESPACE
Upgrade your Kubernetes version
Upgrade your Kubernetes platform to the versions supported by hybrid 1.9. Follow your platform's documentation if you need help.
Rolling back an upgrade
Follow these steps to roll back a previous upgrade:
- Clean up completed jobs for the hybrid runtime namespace, where NAMESPACE is the
namespace specified in your overrides file, if you specified a namespace. If not, the default namespace
is
apigee
:kubectl delete job -n NAMESPACE \ $(kubectl get job -n NAMESPACE \ -o=jsonpath='{.items[?(@.status.succeeded==1)].metadata.name}')
- Clean up completed jobs for the
apigee-system
namespace:kubectl delete job -n apigee-system \ $(kubectl get job -n apigee-system \ -o=jsonpath='{.items[?(@.status.succeeded==1)].metadata.name}')
- Change the
APIGEECTL_HOME
variable to point to the directory that contains the previous version ofapigeectl
. For example:export APIGEECTL_HOME=PATH_TO_PREVIOUS_APIGEECTL_DIRECTORY
- In the root directory of the installation you want to roll back to, run
apigeectl apply
, check the status of your pods, and then runapigeectl init
. Be sure to use the original overrides file for the version you wish to roll back to:- In the hybrid-files directory, run
apigeectl apply
:$APIGEECTL_HOME
/apigeectl apply -f ORIGINAL_OVERRIDES_FILEWhere ORIGINAL_OVERRIDES_FILE is the relative path and filename of the overrides file for your previous version hybrid installation, for example,
./overrides/overrides1.8.yaml
. - Check the status of your pods:
kubectl -n NAMESPACE get pods
Where NAMESPACE is your Apigee hybrid namespace.
- Check the status of
apigeeds
:kubectl describe apigeeds -n apigee
Your output should look something like:
Status: Cassandra Data Replication: Cassandra Pod Ips: 10.8.2.204 Cassandra Ready Replicas: 1 Components: Cassandra: Last Successfully Released Version: Revision: v1-f8aa9a82b9f69613 Version: v1 Replicas: Available: 1 Ready: 1 Total: 1 Updated: 1 State: running Scaling: In Progress: false Operation: Requested Replicas: 0 State: running
Proceed to the next step only when the
apigeeds
pod is running. - Run the following command to make note of what your new replica count values will be for
the message processor after the upgrade. If these values do not match what you have set
previously, change the values in your overrides file to match your previous
configuration.
apigeectl apply -f ORIGINAL_OVERRIDES_FILE --dry-run=client --print-yaml --env ENV_NAME 2>/dev/null |grep "runtime:" -A 25 -B 1| grep "autoScaler" -A 2
Your output should look something like:
autoScaler: minReplicas: 2 maxReplicas: 10
- Run
apigeectl init
:$APIGEECTL_HOME
/apigeectl init -f ORIGINAL_OVERRIDES_FILE
- In the hybrid-files directory, run