Upgrading Apigee hybrid to version 1.3.6

Stay organized with collections Save and categorize content based on your preferences.

Upgrading to version 1.3.6 overview.

The procedures for upgrading Apigee hybrid are organized in the following sections:

  1. Preparation
    1. Create and update service accounts.
    2. Plan environment groups.
    3. Copy and update overrides file.
  2. Upgrade Istio and cert-manager.
  3. Install hybrid runtime version 1.3.
  4. Clean up.



  1. (Recommended) Make a backup copy of your version 1.2 $APIGEECTL_HOME/ directory. For example:
    tar -czvf $APIGEECTL_HOME/../apigeectl-v1.2-backup.tar.gz $APIGEECTL_HOME
  2. (Recommended) Backup your Cassandra database following the instructions in Cassandra backup and recovery
  3. Upgrade your Kubernetes platform as follows. Follow your platform's documentation if you need help:
    Platform Upgrade to version
    GKE 1.15.x
    Anthos 1.5
    AKS 1.16.x using Anthos attached clusters
  4. If you are not using Apigee Connect in your hybrid installation, enable Apigee Connect.
    1. Check to see if the Apigee Connect API is enabled:
      gcloud services list | grep apigeeconnect
      apigeeconnect.googleapis.com         Apigee Connect API
    2. If it is not, enable the API:
      gcloud services enable apigeeconnect.googleapis.com --project $PROJECT_ID

      Where $PROJECT_ID is your Google Cloud project ID.

    3. On the command line, get your gcloud authentication credentials, as the following example shows:

      TOKEN=$(gcloud auth print-access-token)

      To check that your token was populated, use echo, as the following example shows:

      echo $TOKEN

      This should display your token as an encoded string.

      For more information, see gcloud command-line tool overview.

    4. Check to see if Apigee Connect is enabled for your organization:
      curl -H "Authorization: Bearer $TOKEN" \

      Where $ORG_NAME is the ID of your organization.

      If the output contains:

            "name" : "features.mart.connect.enabled",
            "value" : "true"

      Apigee Connect is enabled.

    5. If Apigee Connect is not enabled, assign the Apigee Connect Agent role to the MART service account:
      gcloud projects add-iam-policy-binding $PROJECT_ID \
        --member serviceAccount:apigee-mart@$PROJECT_ID.iam.gserviceaccount.com \
        --role roles/apigeeconnect.Agent
    6. Enable Apigee Connect with the following command:
      curl -H "Authorization: Bearer $TOKEN" -X PUT \
        -H "Content-Type: application/json" \
        -d '{
          "name" : "'"$ORG_NAME"'",
          "properties" : {
            "property" : [ {
              "name" : "features.hybrid.enabled",
              "value" : "true"
            }, {
              "name" : "features.mart.connect.enabled",
              "value" : "true"
            } ]
        }' \

      If the output contains the two following properties, Apigee Connect was successfully enabled:

              "name": "features.mart.connect.enabled",
              "value": "true"
              "name": "features.hybrid.enabled",
              "value": "true"
  5. Create the apigee-watcher service account. Apigee Watcher is a new service account introduced in v1.3. It whaches the synchronizer for org-level changes and applies those changes to configure the Istio ingress.

    From your main hybrid directory:

    ./tools/create-service-account apigee-watcher ./service-accounts
  6. Assign Apigee Runtime Agent role to the Watcher service account:
    gcloud projects add-iam-policy-binding $PROJECT_ID \
      --member serviceAccount:apigee-watcher@$PROJECT_ID.iam.gserviceaccount.com \
      --role roles/apigee.runtimeAgent

    Where PROJECT_ID is your Google Cloud project ID. If your service account email addresses differ from this pattern, replace them accordingly.

    The output should include a list of all the service accounts and their roles, including:

    - members:
      - serviceAccount:apigee-watcher@hybrid13rc5.iam.gserviceaccount.com
      role: roles/apigee.runtimeAgent
  7. Plan your environment groups for routing. Apigee hybrid 1.3 manages base path routing with environment groups instead of routingRules. If you are using routingRules in your hybrid configuration, design environment groups to replicate your routing.

    You must create at least one environment group.

    See About environment groups.

  8. Update your overrides file:
    1. Make a copy of your overrides file.
    2. Update gcp and k8sCluster stanzas.

      The following configuration properties have been replaced in hybrid version 1.3:

      • gcpRegion replaced with gcp:region
      • gcpProjectID replaced with gcp:projectID
      • gcpProjectIDRuntime replaced with gcp:gcpProjectIDRuntime
      • k8sClusterName replaced with k8s:clusterName
      • k8sClusterRegion replaced with k8s:clusterRegion

      For example, replace the following structure:

      gcpRegion: gcp region
      gcpProjectID: gcp project ID
      gcpProjectIDRuntime: gcp project ID
      k8sClusterName: name
      k8sClusterRegion: region


       projectID: gcp project ID
       region: gcp region
       gcpProjectIDRuntime: gcp project ID # optional. This is only required if you
                                             # want logger/metrics data to be sent in
                                             # different gcp project.
       name: gcp project ID
       region: gcp region
    3. If you do not have a unique instance identifier already in your overrides file, add one:
      # unique identifier for this installation. 63 chars length limit
      instanceID: ID

      Where ID is a unique identifier for this hybrid installation, like "my-hybrid-131-installation" or "acmecorp-hybrid-131."

    4. Add the Watcher (apigee-watcher) service account to the overrides file:
      # Note: the SA should have the "Apigee Runtime Agent" role
       serviceAccountPath: "service account file"
    5. Add the Metrics (apigee-metrics) service account to the overrides file:
       serviceAccountPath: "service account file"
    6. Update virtualhosts: stanza to replace routingRuleswith your environment group.
      1. -name: Replace the name with the name of your environment group. You can have multiple name entries, one for each environment group.
      2. hostAliases:[] Delete this line.
      3. Keep (or add) the sslCertPath: and sslKeyPath: entries.
      4. Delete all routingRules entries.

      For example:

        - name: default
            - "*.acme.com"
          sslCertPath: ./certs/keystore.pem
          sslKeyPath: ./certs/keystore.key
            - paths:
              - /foo
              - /bar
            - env: my-environment


        - name: example-env-group
          sslCertPath: ./certs/keystore.pem
          sslKeyPath: ./certs/keystore.key
    7. Update the mart and connectAgent: stanzas.
      1. Under mart: remove the hostAlias:, sslCertPath:, and sslKeyPath: entries.
      2. Add an connectAgent: stanza.
      3. Under connectAgent: add a serviceAccountPath: entry and provide the path to the service acount file that has the Apigee Connect Agent role assigned to it (usually the MART service account).

      For example:

        hostAlias: "mart.apigee-hybrid-docs.net"
        serviceAccountPath: ./service-accounts/hybrid-project-apigee-mart.json
        sslCertPath: ./certs/fullchain.pem
        sslKeyPath: ./certs/privkey.key


        serviceAccountPath: ./service-accounts/hybrid-project-apigee-mart.json
        serviceAccountPath: ./service-accounts/hybrid-project-apigee-mart.json

Upgrade Istio and cert-manager

Apigee hybrid version 1.3 requires cert-manager v0.14.2 to manage and verify certificates and the Istio distribution provided with Anthos Service Mesh (ASM) version 1.5.7 (or newer) to create and manage the runtime ingress gateway.

Upgrade Istio 1.4.6 to ASM 1.5.7 (or newer)

  1. To minimize downtime, the Istio deployments and HPAs must have at least two replicas each. Run the following commands to determine the number of replicas:
    kubectl -n istio-system get deployments # list of deployments
    kubectl -n istio-system get hpa # list of hpa
  2. Edit each deployment that has only one replica and increase the replicas: to 2 or more:
    kubectl -n istio-system edit deployment name

    For example:

      progressDeadlineSeconds: 600
      replicas: 2
  3. Edit each HPA that has only one replica and increase the minReplicas: to 2 or more:
    kubectl -n istio-system edit hpa name

    For example:

      maxReplicas: 5
      minReplicas: 2
  4. Download and install ASM following the install instructions in Download and install ASM.
  5. After the installation, run the version command to make sure you have 1.5.x installed correctly:
    ./bin/istioctl version
    client version: 1.5.8-asm.0
    apigee-mart-ingressgateway version:
    citadel version: 1.4.6
    galley version: 1.4.6
    ingressgateway version: 1.5.8-asm.0
    pilot version: 1.4.6
    policy version: 1.4.6
    sidecar-injector version: 1.4.6
    telemetry version: 1.4.6
    pilot version: 1.5.8-asm.0
    data plane version: 1.4.6 (1 proxies), 1.5.8-asm.0 (2 proxies)

Upgrade cert-manager

  1. Delete the current cert-manager deployment:
    kubectl delete -n cert-manager deployment cert-manager cert-manager-cainjector cert-manager-webhook
  2. Verify your version of Kubernetes:
    kubectl version
  3. Run the following command to install cert-manager from Jetstack:
    kubectl apply --validate=false -f https://github.com/jetstack/cert-manager/releases/download/v0.14.2/cert-manager.yaml 

Install the hybrid runtime

  1. Store the latest version number in a variable:
    export VERSION=$(curl -s \
  2. Check that the variable was populated with a version number. If you want to use a different version, you can save that in environment variable instead. For example:
    echo $VERSION
  3. Download the release package for your operating system:

    Mac 64 bit:

    curl -LO \

    Linux 64 bit:

    curl -LO \

    Mac 32 bit:

    curl -LO \

    Linux 32 bit:

    curl -LO \
  4. Rename your current apigeectl/ directory to a backup directory name. For example:
  5. Extract the downloaded gzip file contents into your hybrid base directory. For example:

    tar xvzf filename.tar.gz -C hybrid-base-directory
  6. cd to the base directory.
  7. The tar contents are, by default, expanded into a directory with the version and platform in its name. For example: ./apigeectl_1.0.0-f7b96a8_linux_64. Rename that directory to apigeectl:

    mv apigeectl_1.0.0-f7b96a8_linux_64 apigeectl
  8. Delete the apigee-resources-install job from apigee-system:
    kubectl -n apigee-system delete job apigee-resources-install
  9. Delete the older CRD:
    kubectl delete crd apigeetelemetries.apigee.cloud.google.com
  10. Update the cassandra: stanza in your overrides file with an externalSeedHost property. This property will help ensure your new hybrid version 1.3.6 installation will use the same Kubernetes cluster as your version 1.2 installation. This is a one-time step needed only for upgrade from hybrid version 1.2 to version 1.3.6 (or newer).
    1. Find one of the IP addresses of the existing Cassandra in the same Kubernetes cluster where you are upgrading the 1.2.0 installation.
      kubectl -n namespace get pods -o wide

      Where namespace is your Apigee hybrid namespace.

      Make note of the IP address of a Cassandra node. For example:

      kubectl -n apigee get pods -o wide
      NAME                  READY   STATUS    RESTARTS   AGE   IP          NODE
      apigee-cassandra-0    1/1     Running   0          33d   gke-example-cluster-rc5-apigee-data-c8bf1234-09kc
      apigee-cassandra-1    1/1     Running   0          16d   gke-example-cluster-rc5-apigee-data-c9221ee7-10kc
      apigee-cassandra-2    1/1     Running   0          23h   gke-example-cluster-rc5-apigee-data-d123e456-11kc
    2. Add the value for the externalSeedHost property:
       externalSeedHost: Cassandra_node_IP

      Where Cassandra_node_IP is the IP of the cassandra node ( in the preceding example).

  11. In the new apigeectl/ directory, run apigeectl init, apigeectl apply, and apigeectl check-ready:
    1. Initialize hybrid 1.3.6:
      apigeectl init -f overrides_1.3.yaml

      Where overrides_1.3.yaml is your edited overrides.yaml file.

    2. In hybrid version 1.3, the syntax of the --dry-run flag depends on the version of kubectl you are running. Check the version of kubectl:
      gcloud version
    3. Check for errors with a dry run:

      kubectl version 1.17 and older:

      apigeectl apply -f overrides_1.3.yaml --dry-run=true

      kubectl version 1.18 and newer:

      apigeectl apply -f overrides_1.3.yaml --dry-run=client
    4. Apply your overrides. Select and follow the instructions for production environments or demo/experimental environments, depending on your installation.


      For production environments you should upgrade each hybrid component individually, and check the status of the upgraded component before proceding to the next component.

      1. Apply your overrides to upgrade Cassandra:
        apigeectl apply -f overrides_1.3.yaml --datastore
      2. Check completion:
        kubectl -n namespace get pods

        Where namespace is your Apigee hybrid namespace.

        Proceed to the next step only when the pods are ready.

      3. Apply your overrides to upgrade Telemetry components and check completion:
        apigeectl apply -f overrides_1.3.yaml --telemetry
        kubectl -n namespace get pods
      4. Apply your overrides to upgrade the org-level components (MART, Watcher and Apigee Connect) and check completion:
        apigeectl apply -f overrides_1.3.yaml --org
        kubectl -n namespace get pods
      5. Apply your overrides to upgrade your environments. You have two choices:
        • Apply your overrides to one environment at a time and check completion. Repeat this step for each environment:
          apigeectl apply -f overrides_1.3.yaml --env env_name
          kubectl -n namespace get pods

          Where env_name is the name of the environment you are upgrading.

        • Apply your overrides to all environments at once and check completion:
          apigeectl apply -f overrides_1.3.yaml --all-envs
          kubectl -n namespace get pods


      In most demo or experimental environments, you can apply the overrides to all components at once. If your demo/experimental environment large and complex or closely mimics a production environment, you may want to use the instructions for upgrading production environments

      1. apigeectl apply -f overrides_1.3.yaml
      2. Check the status:
        apigeectl check-ready -f overrides_1.3.yaml

      For more instructions see GKE Hybrid setup - Step 5: Install hybrid on GKE.

    5. Once you have hybrid 1.3 setup up and running, verify that all of the Cassandra nodes (old and new) are part of the same Cassandra cluster. Run the following command on one of the Cassandra nodes:
      kubectl -n namespace get pods
      kubectl -n namespace exec old Cassandra pod -- nodetool status

      In the following example output, is from version 1.2.0 and is the node IP you used as the externalSeedHost. is from version 1.3.6:

      Datacenter: dc-1
      |/ State=Normal/Leaving/Joining/Moving
      --  Address     Load        Tokens       Owns (effective)  Host ID                               Rack
      UN  379.41 KiB  256          50.8%             11bbd43b-af64-464b-a96d-0d6dd0521de1  ra-1
      UN  1.35 MiB    256          49.2%             0b4d9e08-f353-413a-b4a9-7d18a8d07e58  ra-1

      If they are not in the same cluster, check the externalSeedHost value.

    6. Once all the pods are up and running, remove externalSeedHost from the your overrides file and run apigeectl apply again with --datastore option:
      apigeectl apply --datastore -f overrides_1.3.6.yaml

    Clean up

    Once you have verified all the pods are up and healthy and the ASM endpoints are valid for the new installation, you can clean up:

    • Hybrid 1.2 resources.
    • The older Cassandra instance
    • Istio 1.4.6 resources.

    Delete Hybrid 1.2.0 resources

    1. Remove the 1.2.0 virtualhost routing details:
      $APIGEECTL_HOME-v1.2/apigeectl delete -s virtualhost -f 1.2.0_overrides.yaml

      Where $APIGEECTL_HOME-v1.2 is the directory where you backed up your apigeectl version 1.2 directory.

    2. If the endpoint is still working as expected and you have verified all the 1.3.0 components are working run the following command to delete the hybrid 1.2.0 resources:
      $APIGEECTL_HOME-v1.2/apigeectl delete -c "mart,connect-agent,synchronizer,runtime,udca,metrics,logger" \
        -f 1.2.0_overrides.yaml

    Decommission the older Cassandra instance

    1. cd into the newly installed apigeectldirectory.
    2. Run the tools/cas_cleanup.sh script.

      This script decommissions the old Cassandra pod from the Cassandra ring, deletes old STS, and deletes PVCs.

      bash cas_cleanup.sh Apigee namespace

    Delete Istio version 1.4.6 resources

    1. Run the following command to delete the most recent Istio v.1.4.6 resources:
      kubectl delete all -n istio-system --selector \
        'app in (apigee-mart-istio-ingressgateway, galley, security, istio-nodeagent, istio-mixer, sidecarInjectorWebhook, istio-mixer)'
    2. Run the following commands to delete older jobs from the Istio 1.4.6 installation:
      kubectl -n istio-system delete job istio-init-crd-10-1.4.6
      kubectl -n istio-system delete job istio-init-crd-11-1.4.6
      kubectl -n istio-system delete job istio-init-crd-14-1.4.6

    Congratulations! You have successfully upgraded to Apigee hybrid version 1.3.6.