Upgrading Apigee hybrid to version 1.3.6

Upgrading to version 1.3.6 overview.

The procedures for upgrading Apigee hybrid are organized in the following sections:

Preparation

Create and update service accounts.
Plan environment groups.
Copy and update overrides file.

Upgrade Istio and cert-manager.
Install hybrid runtime version 1.3.
Clean up.

Prerequisite

Apigee hybrid version 1.2. If you are updating from an earlier version see the instructions for Upgrading Apigee hybrid to version 1.2.

Preparation

(Recommended) Make a backup copy of your version 1.2 $APIGEECTL_HOME/ directory. For example:
```
tar -czvf $APIGEECTL_HOME/../apigeectl-v1.2-backup.tar.gz $APIGEECTL_HOME
```
(Recommended) Backup your Cassandra database following the instructions in Cassandra backup and recovery

Upgrade your Kubernetes platform as follows. Follow your platform's documentation if you need help:

Platform	Upgrade to version
GKE	1.15.x
Anthos	1.5
AKS	1.16.x using Anthos attached clusters

If you are not using Apigee Connect in your hybrid installation, enable Apigee Connect.

Check to see if the Apigee Connect API is enabled:

gcloud services list | grep apigeeconnect

apigeeconnect.googleapis.com         Apigee Connect API

If it is not, enable the API:
```
gcloud services enable apigeeconnect.googleapis.com --project $PROJECT_ID
```
Where $PROJECT_ID is your Google Cloud project ID.
On the command line, get your gcloud authentication credentials, as the following example shows:
```
TOKEN=$(gcloud auth print-access-token)
```
To check that your token was populated, use echo, as the following example shows:
```
echo $TOKEN
```
This should display your token as an encoded string.

For more information, see gcloud command-line tool overview.

Check to see if Apigee Connect is enabled for your organization:

curl -H "Authorization: Bearer $TOKEN" \
  "https://apigee.googleapis.com/v1/organizations/$ORG_NAME"

Where $ORG_NAME is the ID of your organization.

If the output contains:

      "name" : "features.mart.connect.enabled",
      "value" : "true"

Apigee Connect is enabled.

If Apigee Connect is not enabled, assign the Apigee Connect Agent role to the MART service account:

gcloud projects add-iam-policy-binding $PROJECT_ID \
  --member serviceAccount:apigee-mart@$PROJECT_ID.iam.gserviceaccount.com \
  --role roles/apigeeconnect.Agent

Enable Apigee Connect with the following command:

curl -H "Authorization: Bearer $TOKEN" -X PUT \
  -H "Content-Type: application/json" \
  -d '{
    "name" : "'"$ORG_NAME"'",
    "properties" : {
      "property" : [ {
        "name" : "features.hybrid.enabled",
        "value" : "true"
      }, {
        "name" : "features.mart.connect.enabled",
        "value" : "true"
      } ]
    }
  }' \
  "https://apigee.googleapis.com/v1/organizations/$ORG_NAME"

If the output contains the two following properties, Apigee Connect was successfully enabled:

      {
        "name": "features.mart.connect.enabled",
        "value": "true"
      },
      {
        "name": "features.hybrid.enabled",
        "value": "true"
      }

Create the apigee-watcher service account. Apigee Watcher is a new service account introduced in v1.3. It whaches the synchronizer for org-level changes and applies those changes to configure the Istio ingress.
From your main hybrid directory:
```
./tools/create-service-account apigee-watcher ./service-accounts
```
Assign Apigee Runtime Agent role to the Watcher service account:
```
gcloud projects add-iam-policy-binding $PROJECT_ID \
  --member serviceAccount:apigee-watcher@$PROJECT_ID.iam.gserviceaccount.com \
  --role roles/apigee.runtimeAgent
```
Where PROJECT_ID is your Google Cloud project ID. If your service account email addresses differ from this pattern, replace them accordingly.
Tip: If you need to get the email address of your watcher service account, run the following command:
```
gcloud iam service-accounts list | grep watcher
```
The output should include a list of all the service accounts and their roles, including:
```
  ...
- members:
  - serviceAccount:apigee-watcher@hybrid13rc5.iam.gserviceaccount.com
  role: roles/apigee.runtimeAgent
  ...
```
Plan your environment groups for routing. Apigee hybrid 1.3 manages base path routing with environment groups instead of routingRules. If you are using routingRules in your hybrid configuration, design environment groups to replicate your routing.
You must create at least one environment group.

See About environment groups.
Click to see an example of using environment groups to replace routingRules

In this example, the following routingRules in the overrides.yaml file route requests to the foo and bar API proxies to the prod-env-1 environment, and the foo-test proxy to the test-env environment:
```
virtualhosts:
  - name: default
    hostAliases:
      - "*.acme.com"
    sslCertPath: ./certs/keystore.pem
    sslKeyPath: ./certs/keystore.key
    routingRules:
      - paths:
        - /foo
        - /bar
      - env: prod-env-1
      - paths:
        - foo-test
      -env: test-env
```
To duplicate that in hybrid version 1.3.6:
1. Deploy the foo and bar proxies to the prod-env-1 environment.
2. Deploy the foo-test and bar proxy to the test-env environment.
3. Create an environment group with a hostalias for the proxies, like api.acme.com.
  Tip: Create an environment group with the same name as the virtualhosts name you used in version 1.2.
4. Revise the virtualhosts stanza in overrides.yaml to
  - Replace virtualhosts:name with the name of the environment group.
  - Remove the virtualhosts:routingRules.
  For example:
```
virtualhosts:
  - name: default # environment group name
    sslCertPath: ./certs/keystore.pem
    sslKeyPath: ./certs/keystore.key
```
Update your overrides file:

Make a copy of your overrides file.

Update gcp and k8sCluster stanzas.

The following configuration properties have been replaced in hybrid version 1.3:

gcpRegion replaced with gcp:region
gcpProjectID replaced with gcp:projectID
gcpProjectIDRuntime replaced with gcp:gcpProjectIDRuntime
k8sClusterName replaced with k8s:clusterName
k8sClusterRegion replaced with k8s:clusterRegion

For example, replace the following structure:

gcpRegion: gcp region
gcpProjectID: gcp project ID
gcpProjectIDRuntime: gcp project ID

k8sClusterName: name
k8sClusterRegion: region

with:

gcp:
 projectID: gcp project ID
 region: gcp region
 gcpProjectIDRuntime: gcp project ID # optional. This is only required if you
                                       # want logger/metrics data to be sent in
                                       # different gcp project.

k8sCluster:
 name: gcp project ID
 region: gcp region

If you do not have a unique instance identifier already in your overrides file, add one:
```
# unique identifier for this installation. 63 chars length limit
instanceID: ID
```
Where ID is a unique identifier for this hybrid installation, like "my-hybrid-131-installation" or "acmecorp-hybrid-131."

Add the Watcher (apigee-watcher) service account to the overrides file:

# Note: the SA should have the "Apigee Runtime Agent" role
watcher:
 serviceAccountPath: "service account file"

Add the Metrics (apigee-metrics) service account to the overrides file:
```
metrics:
 serviceAccountPath: "service account file"
```

Update virtualhosts: stanza to replace routingRuleswith your environment group.

-name: Replace the name with the name of your environment group. You can have multiple name entries, one for each environment group.
hostAliases:[] Delete this line.
Keep (or add) the sslCertPath: and sslKeyPath: entries.
Delete all routingRules entries.

For example:

virtualhosts:
  - name: default
    hostAliases:
      - "*.acme.com"
    sslCertPath: ./certs/keystore.pem
    sslKeyPath: ./certs/keystore.key
    routingRules:
      - paths:
        - /foo
        - /bar
      - env: my-environment

Becomes:

virtualhosts:
  - name: example-env-group
    sslCertPath: ./certs/keystore.pem
    sslKeyPath: ./certs/keystore.key

Update the mart and connectAgent: stanzas.
Note: If you are not planning to use Apigee Connect in your hybrid 1.3 installation, skip this step and follow the tip in step 5 in Apigee Connect after upgrading.
1. Under mart: remove the hostAlias:, sslCertPath:, and sslKeyPath: entries.
2. Add an connectAgent: stanza.
3. Under connectAgent: add a serviceAccountPath: entry and provide the path to the service acount file that has the Apigee Connect Agent role assigned to it (usually the MART service account).
For example:
```
mart:
  hostAlias: "mart.apigee-hybrid-docs.net"
  serviceAccountPath: ./service-accounts/hybrid-project-apigee-mart.json
  sslCertPath: ./certs/fullchain.pem
  sslKeyPath: ./certs/privkey.key
```
Becomes:
```
mart:
  serviceAccountPath: ./service-accounts/hybrid-project-apigee-mart.json

connectAgent:
  serviceAccountPath: ./service-accounts/hybrid-project-apigee-mart.json
```

Upgrade Istio and cert-manager

Apigee hybrid version 1.3 requires cert-manager v0.14.2 to manage and verify certificates and the Istio distribution provided with Anthos Service Mesh (ASM) version 1.5.7 (or newer) to create and manage the runtime ingress gateway.

Upgrade Istio 1.4.6 to ASM 1.5.7 (or newer)

To minimize downtime, the Istio deployments and HPAs must have at least two replicas each. Run the following commands to determine the number of replicas:
```
kubectl -n istio-system get deployments # list of deployments
kubectl -n istio-system get hpa # list of hpa
```

Edit each deployment that has only one replica and increase the replicas: to 2 or more:

kubectl -n istio-system edit deployment name

For example:

spec:
  progressDeadlineSeconds: 600
  replicas: 2

Edit each HPA that has only one replica and increase the minReplicas: to 2 or more:
```
kubectl -n istio-system edit hpa name
```
For example:
```
spec:
  maxReplicas: 5
  minReplicas: 2
```
Download and install ASM following the install instructions in Download and install ASM.

After the installation, run the version command to make sure you have 1.5.x installed correctly:

./bin/istioctl version

client version: 1.5.8-asm.0
apigee-mart-ingressgateway version:
citadel version: 1.4.6
galley version: 1.4.6
ingressgateway version: 1.5.8-asm.0
pilot version: 1.4.6
policy version: 1.4.6
sidecar-injector version: 1.4.6
telemetry version: 1.4.6
pilot version: 1.5.8-asm.0
data plane version: 1.4.6 (1 proxies), 1.5.8-asm.0 (2 proxies)

Upgrade cert-manager

Delete the current cert-manager deployment:

kubectl delete -n cert-manager deployment cert-manager cert-manager-cainjector cert-manager-webhook

Verify your version of Kubernetes:
```
kubectl version
```

Run the following command to install cert-manager from Jetstack:

kubectl apply --validate=false -f https://github.com/jetstack/cert-manager/releases/download/v0.14.2/cert-manager.yaml

Install the hybrid runtime

Store the latest version number in a variable:

export VERSION=$(curl -s \
    https://storage.googleapis.com/apigee-public/apigee-hybrid-setup/current-version.txt?ignoreCache=1)

Check that the variable was populated with a version number. If you want to use a different version, you can save that in environment variable instead. For example:
```
echo $VERSION
  1.3.6
```

Download the release package for your operating system:

Mac 64 bit:

curl -LO \
    https://storage.googleapis.com/apigee-public/apigee-hybrid-setup/$VERSION/apigeectl_mac_64.tar.gz

Linux 64 bit:

curl -LO \
    https://storage.googleapis.com/apigee-public/apigee-hybrid-setup/$VERSION/apigeectl_linux_64.tar.gz

Mac 32 bit:

curl -LO \
    https://storage.googleapis.com/apigee-public/apigee-hybrid-setup/$VERSION/apigeectl_mac_32.tar.gz

Linux 32 bit:

curl -LO \
    https://storage.googleapis.com/apigee-public/apigee-hybrid-setup/$VERSION/apigeectl_linux_32.tar.gz

Rename your current apigeectl/ directory to a backup directory name. For example:
```
mv $APIGEECTL_HOME/ $APIGEECTL_HOME-v1.2/ 
```
Extract the downloaded gzip file contents into your hybrid base directory. For example:
```
tar xvzf filename.tar.gz -C hybrid-base-directory
```
cd to the base directory.
The tar contents are, by default, expanded into a directory with the version and platform in its name. For example: ./apigeectl_1.0.0-f7b96a8_linux_64. Rename that directory to apigeectl:
```
mv apigeectl_1.0.0-f7b96a8_linux_64 apigeectl
```

Delete the apigee-resources-install job from apigee-system:

kubectl -n apigee-system delete job apigee-resources-install

Delete the older CRD:

kubectl delete crd apigeetelemetries.apigee.cloud.google.com

Update the cassandra: stanza in your overrides file with an externalSeedHost property. This property will help ensure your new hybrid version 1.3.6 installation will use the same Kubernetes cluster as your version 1.2 installation. This is a one-time step needed only for upgrade from hybrid version 1.2 to version 1.3.6 (or newer).
1. Find one of the IP addresses of the existing Cassandra in the same Kubernetes cluster where you are upgrading the 1.2.0 installation.
```
kubectl -n namespace get pods -o wide
```
  Where namespace is your Apigee hybrid namespace.
  
  Make note of the IP address of a Cassandra node. For example:
```
kubectl -n apigee get pods -o wide
NAME                  READY   STATUS    RESTARTS   AGE   IP          NODE
apigee-cassandra-0    1/1     Running   0          33d   10.68.8.24   gke-example-cluster-rc5-apigee-data-c8bf1234-09kc
apigee-cassandra-1    1/1     Running   0          16d   10.68.8.33   gke-example-cluster-rc5-apigee-data-c9221ee7-10kc
apigee-cassandra-2    1/1     Running   0          23h   10.68.9.11   gke-example-cluster-rc5-apigee-data-d123e456-11kc
```
2. Add the value for the externalSeedHost property:
```
cassandra:
 externalSeedHost: Cassandra_node_IP
```
  Where Cassandra_node_IP is the IP of the cassandra node (10.68.8.24 in the preceding example).
  
  Note: You will remove the externalSeedHost property in a later step after your new Cassandra is running and verified.
In the new apigeectl/ directory, run apigeectl init, apigeectl apply, and apigeectl check-ready:
1. Initialize hybrid 1.3.6:
```
apigeectl init -f overrides_1.3.yaml
```
  Where overrides_1.3.yaml is your edited overrides.yaml file.
2. In hybrid version 1.3, the syntax of the --dry-run flag depends on the version of kubectl you are running. Check the version of kubectl:
```
gcloud version
```
3. Check for errors with a dry run:
  kubectl version 1.17 and older:
```
apigeectl apply -f overrides_1.3.yaml --dry-run=true
```
  kubectl version 1.18 and newer:
```
apigeectl apply -f overrides_1.3.yaml --dry-run=client
```
4. Apply your overrides. Select and follow the instructions for production environments or demo/experimental environments, depending on your installation.
  During the upgrade process each component will perform a rolling restart. Therefore, for production environments, it is best to apply the upgrade to one component at a time.
  Production
  
  For production environments you should upgrade each hybrid component individually, and check the status of the upgraded component before proceding to the next component.
  1. Apply your overrides to upgrade Cassandra:
    apigeectl apply -f overrides_1.3.yaml --datastore
  2. Check completion:
    kubectl -n namespace get pods
    
    Where namespace is your Apigee hybrid namespace.
    
    Proceed to the next step only when the pods are ready.
  3. Apply your overrides to upgrade Telemetry components and check completion:
    apigeectl apply -f overrides_1.3.yaml --telemetry
    
    kubectl -n namespace get pods
  4. Apply your overrides to upgrade the org-level components (MART, Watcher and Apigee Connect) and check completion:
    apigeectl apply -f overrides_1.3.yaml --org
    
    kubectl -n namespace get pods
  5. Apply your overrides to upgrade your environments. You have two choices:
    
    Apply your overrides to one environment at a time and check completion. Repeat this step for each environment:
    apigeectl apply -f overrides_1.3.yaml --env env_name
    
    kubectl -n namespace get pods
    
    Where env_name is the name of the environment you are upgrading.
    
    Apply your overrides to all environments at once and check completion:
    apigeectl apply -f overrides_1.3.yaml --all-envs
    
    kubectl -n namespace get pods
  Demo/Experimental
  
  In most demo or experimental environments, you can apply the overrides to all components at once. If your demo/experimental environment large and complex or closely mimics a production environment, you may want to use the instructions for upgrading production environments
  1. apigeectl apply -f overrides_1.3.yaml
  2. Check the status:
    apigeectl check-ready -f overrides_1.3.yaml
  For more instructions see GKE Hybrid setup - Step 5: Install hybrid on GKE.
5. Once you have hybrid 1.3 setup up and running, verify that all of the Cassandra nodes (old and new) are part of the same Cassandra cluster. Run the following command on one of the Cassandra nodes:
```
kubectl -n namespace get pods
kubectl -n namespace exec old Cassandra pod -- nodetool status
```
  In the following example output, 10.68.8.24 is from version 1.2.0 and is the node IP you used as the externalSeedHost. 10.68.7.11 is from version 1.3.6:
```
Datacenter: dc-1
================
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
--  Address     Load        Tokens       Owns (effective)  Host ID                               Rack
UN  10.68.8.24  379.41 KiB  256          50.8%             11bbd43b-af64-464b-a96d-0d6dd0521de1  ra-1
UN  10.68.7.11  1.35 MiB    256          49.2%             0b4d9e08-f353-413a-b4a9-7d18a8d07e58  ra-1
```
  If they are not in the same cluster, check the externalSeedHost value.
6. Once all the pods are up and running, remove externalSeedHost from the your overrides file and run apigeectl apply again with --datastore option:
```
apigeectl apply --datastore -f overrides_1.3.6.yaml
```
Clean up

Once you have verified all the pods are up and healthy and the ASM endpoints are valid for the new installation, you can clean up:
- Hybrid 1.2 resources.
- The older Cassandra instance
- Istio 1.4.6 resources.
Delete Hybrid 1.2.0 resources
Note: This procedure requires the apigeectl version 1.2 command in your backed up directory. If you need to download another copy of apigeectl version 1.2, you can use the following command:
```
curl -LO \
    https://storage.googleapis.com/apigee-public/apigee-hybrid-setup/1.2.0/apigeectl_your_system.tar.gz
```
Where your_system is one of linux_32, linux_64, mac_32, or mac_64.
1. Remove the 1.2.0 virtualhost routing details:
```
$APIGEECTL_HOME-v1.2/apigeectl delete -s virtualhost -f 1.2.0_overrides.yaml
```
  Where $APIGEECTL_HOME-v1.2 is the directory where you backed up your apigeectl version 1.2 directory.
2. If the endpoint is still working as expected and you have verified all the 1.3.0 components are working run the following command to delete the hybrid 1.2.0 resources:
```
$APIGEECTL_HOME-v1.2/apigeectl delete -c "mart,connect-agent,synchronizer,runtime,udca,metrics,logger" \
  -f 1.2.0_overrides.yaml
```
Decommission the older Cassandra instance
1. cd into the newly installed apigeectldirectory.
2. Run the tools/cas_cleanup.sh script.
  This script decommissions the old Cassandra pod from the Cassandra ring, deletes old STS, and deletes PVCs.
```
bash cas_cleanup.sh Apigee namespace
```
Delete Istio version 1.4.6 resources

Note: Deleting the Istio Istio version 1.4.6 resources will also delete the MART ingress as well. MART ingress is replaced with Apigee Connect in Apigee hybrid version 1.3.
1. Run the following command to delete the most recent Istio v.1.4.6 resources:
```
kubectl delete all -n istio-system --selector \
  'app in (apigee-mart-istio-ingressgateway, galley, security, istio-nodeagent, istio-mixer, sidecarInjectorWebhook, istio-mixer)'
```
2. Run the following commands to delete older jobs from the Istio 1.4.6 installation:
```
kubectl -n istio-system delete job istio-init-crd-10-1.4.6
kubectl -n istio-system delete job istio-init-crd-11-1.4.6
kubectl -n istio-system delete job istio-init-crd-14-1.4.6
```
Congratulations! You have successfully upgraded to Apigee hybrid version 1.3.6.

Upgrading Apigee hybrid to version 1.3.6

Upgrading to version 1.3.6 overview.

Prerequisite

Preparation

Click to see an example of using environment groups to replace routingRules

Upgrade Istio and cert-manager

Upgrade Istio 1.4.6 to ASM 1.5.7 (or newer)

Upgrade cert-manager

Install the hybrid runtime

Production

Demo/Experimental

Clean up

Delete Hybrid 1.2.0 resources

Decommission the older Cassandra instance

Delete Istio version 1.4.6 resources

Click to see an example of using environment groups to replace `routingRules`