Apigee hybrid release notes

This page documents production updates to Apigee hybrid in 2022 and later. We recommend that users periodically check this list for any new announcements, or subscribe to this page using a feed reader to get notifications of updates.

What is a feed reader?

Really simple syndication (RSS) feed readers aggregate content from websites that you specify.

Feed reader notifications can be email-, browser-, desktop-, or mobile-based. Some readers are free, or have free versions, and some require a subscription.

A few examples:

More information on RSS:

See also:

Subscribe:

You can see the latest product updates for all of Google Cloud on the Google Cloud page, browse and filter all release notes in the Google Cloud console, or programmatically access release notes in BigQuery.

To get the latest product updates delivered to you, add the URL of this page to your feed reader, or add the feed URL directly.

November 12, 2024

v1.13.2

hybrid v1.13.2

On November 12, 2024 we released an updated version of the Apigee hybrid software, 1.13.2.

Bug ID Description
373722434 Fixed support for backups to GCS buckets with retention policies.
361044374 Fixes assign message not correctly highlighting the set payload action in the debug trace.
355122464 This release contains a few error-handling fixes for CSI backup and restore.
237656263 Fix added to make use of asynchronous ServiceCallout execution when the ServiceCallout policy <Response> element is not present.

Procedure:

  1. In the apigee-env/values.yaml file set conf_system_servicecallout.expects.response to false under runtime:cwcAppend:. For example: 
    # Apigee Runtime.
runtime:
  cwcAppend:
    conf_system_servicecallout.expects.response: false
  2. Upgrade the apigee-env chart for each environment to apply the change. For example: 
    helm upgrade ENV_RELEASE_NAME apigee-env/ \
  --install \
  --namespace APIGEE_NAMESPACE \
  --set env=ENV_NAME \
  -f OVERRIDES_FILE
Bug ID Description
N/A Security fixes for apigee-redis.
This addresses the following vulnerabilities:

November 01, 2024

v1.12.3

hybrid v1.12.3

On November 1, 2024 we released an updated version of the Apigee hybrid software, 1.12.3.

Bug ID Description
368646378 Fixed an issue affecting control Plane connectivity testing in Guardrails.
361044374 Fixes assign message not correctly highlighting the set payload action in the debug trace.
335357961 Fixed an issue where Apigee hybrid could claim uploads of backups with the Cloud provider when no bucket had been configured
181569113 Fixed an issue in new debug session creation.
Bug ID Description
376104926 Security fixes for apigee-kube-rbac-proxy.
This addresses the following vulnerabilities:
N/A Security fixes for apigee-open-telemetry-collector.
This addresses the following vulnerability:

October 04, 2024

v1.13.1

hybrid v1.13.1

On October 4, 2024 we released an updated version of the Apigee hybrid software, 1.13.1.

Cassandra credential rotation in Vault

Starting in version v1.3.1, You can set up automatic Cassandra credential rotation when your credentials are stored in Hashicorp Vault. See Rotating Cassandra credentials in Hashicorp Vault.

New analytics and debug data pipeline for data residency-enabled orgs

Newly created Apigee hybrid v1.13.1 orgs created with data residency enabled can use a new data pipeline to collect analytics and debug data and allow various runtime components to write data directly to our control plane. You cannot use the new data pipeline with non data residency-enabled orgs; only new orgs created on hybrid v1.13.1 can use this new feature. For details, see Using data residency with Apigee hybrid.

Bug ID Description
364282883 Remove check for dc-expansion flag and add timeout to multi-region seed host connection test.
362305438 You can now add additional env variables to the runtime component.
353527851 WebSocket connection drops when using VerifyJwt or OAuthV2 VerifyJWTAccessToken operations.
351440306 An issue was fixed where trace could not be viewed in the UI for orgs with DRZ enabled.
338638343 An ID is now added at the end of apigee-env and virtualhost guardrails pods to make the pod names unique.
Bug ID Description
N/A Security fixes for apigee-open-telemetry-collector.
This addresses the following vulnerability:

September 13, 2024

v1.12.2

hybrid v1.12.2

On September 13, 2024 we released an updated version of the Apigee hybrid software, 1.12.2.

Bug ID Description
362305438 You can now add additional env variables to the runtime component.
347798999 You can now configure forward proxy for opentelemetry pods in Apigee hybrid.
Bug ID Description
N/A Security fixes for apigee-cassandra-backup-utility and apigee-hybrid-cassandra.
This addresses the following vulnerability:

September 05, 2024

1.13.0-hotfix.1

hybrid 1.13.0-hotfix.1

On September 5, 2024 we released an updated version of the Apigee hybrid software, 1.13.0-hotfix.1.

Apply this hotfix following the steps in Upgrading Apigee hybrid to version 1.13:

  1. Prepare for the Helm charts upgrade
  2. Install the Apigee hybrid Helm charts
Bug ID Description
362690729 Fix for aggressive scaling of runtime pods & cpu spike.
362979563 Fix for Ingress Health Check failure /healthz/ingress - route_not_found.

August 23, 2024

v1.13.0

hybrid v1.13.0

On August 23, 2024 we released an updated version of the Apigee hybrid software, v1.13.0.

Storing additional secrets in an external secret store

Starting in version v1.13, You can now store AX Hash Salt, Redis password, and Encryption keys in an external secret store like Hashicorp Vault. See Storing Secrets in Vault.

Apigee Operator now runs in the Apigee Kubernetes namespace

Starting in version v1.13, apigee-operator runs in the same name space as the other Apigee hybrid components instead of the apigee-system namespace. You can use apigee or your own custom Apigee namespace. See Upgrading Apigee hybrid to version v1.13.0 and Step 3: Create the apigee namespace.

Improved backup and restore

Starting in version v1.13.0, Apigee hybrid introduces a new backup and restore system. The new system removes the need for pod exec permission and use of a Kubernetes ClusterRole, and requires fewer Kubernetes Service Accounts when using Workload Identity. The new system replaces use of the apigee-cassandra-backup-utility image by using the apigee-hybrid-cassandra-client image. The apigee-cassandra-backup-utility image will no longer be provided starting with this release. See Scheduling backups in a remote server.

Leader election enabled for apigee-watcher component

Starting in version v1.13.0, leader election is enabled for the apigee-watcher component. For proper functioning of the leader election, make sure that the apigee-watcher component uses only one replica set.

Bug ID Description
352070616 Update Go language version.
351868444 Tolerations are now working for Redis's Envoy pod. (Fixed in 1.12.1-hotfix.1)
347997965 Upgrading to Apigee Hybrid 1.11.2 and 1.10.5 can cause missing metrics. (Fixed in 1.11.2-hotfix.1)
347798999 Fixed issue preventing configuration of forward proxy for opentelemetry pods. (Fixed in Apigee hybrid v 1.12.1)
346589998 Check Cassandra DNS hostname resolution during Hybrid region expansion.
345501069 Fixed issue with Hybrid Guardrails resource configuration preventing the Guardails pod from starting. (Fixed in Apigee hybrid v 1.12.1)
341797795 Autofill the Hybrid Guardrails checkpoint value if a checkpoint is not provided. (Fixed in Apigee hybrid v 1.12.1)
340889560 Added csi to the apigee-logger SCC. (Fixed in Apigee hybrid v 1.12.0-hotfix.1)
340248314 Added support for targetCPUUtilizationPercentage to apigeeIngressGateway and ingressGateways. The default value is 75. (Fixed in Apigee hybrid v 1.12.1)
339849002 Hashicorp Vault integration issues fixed for Google Service Account for Cassandra Backup/Restore. (Fixed in Apigee hybrid v 1.12.0-hotfix.1)
324779388 Improved error handling for backup and restore. (Fixed in Apigee hybrid v 1.12.1)
311489774 Removed inclusion of Java in Cassandra client image.. (Fixed in Apigee hybrid v 1.12.1)
310338146 Fixed invalid download directory output from the create-service-account tool. (Fixed in Apigee hybrid v 1.12.1)
300135626 Removed inclusion of Java in Cassandra Backup Utility image. (Fixed in Apigee hybrid v 1.12.1)
297539870 HTTPTargetConnection property io.timeout.millis is now honored correctly when using websockets. (Fixed in 1.11.2-hotfix.2)
239523766 Remove "Unable to evaluate jsonVariable, returning null" logging string from ExtractVariables Policy (Fixed in Apigee hybrid v 1.12.1)
181569113 Fixed an issue in new debug session creation. (Fixed in Apigee hybrid v 1.11.2)
Bug ID Description
N/A Security fixes for apigee-asm-istiod.
This addresses the following vulnerability:
N/A Security fixes for apigee-cassandra-backup-utility.
This addresses the following vulnerability:
N/A Security fixes for apigee-connect-agent.
This addresses the following vulnerabilities:
N/A Security fixes for apigee-envoy.
This addresses the following vulnerability:
N/A Security fixes for apigee-fluent-bit.
This addresses the following vulnerability:
N/A Security fixes for apigee-hybrid-cassandra.
This addresses the following vulnerabilities:
N/A Security fixes for apigee-hybrid-cassandra-client.
This addresses the following vulnerability:
N/A Security fixes for apigee-mart-server.
This addresses the following vulnerabilities:
N/A Security fixes for apigee-mint-task-scheduler.
This addresses the following vulnerabilities:
N/A Security fixes for apigee-operators.
This addresses the following vulnerabilities:
N/A Security fixes for apigee-prom-prometheus.
This addresses the following vulnerability:
N/A Security fixes for apigee-prometheus-adapter.
This addresses the following vulnerability:
N/A Security fixes for apigee-redis.
This addresses the following vulnerabilities:
N/A Security fixes for apigee-udca.
This addresses the following vulnerability:
N/A Security fixes for apigee-watcher.
This addresses the following vulnerability:
N/A Security fixes for cassandra-backup-utility.
This addresses the following vulnerability:

July 31, 2024

1.11.2-hotfix.2

hybrid 1.11.2-hotfix.2

On July 31, 2024 we released an updated version of the Apigee hybrid software, 1.11.2-hotfix.2.

Note: This release reflects a change to the component images and not the Helm chart templates. If your hybrid installation is currently on Apigee hybrid v1.11.2, Apply this hotfix with the following steps:

  1. In your overrides file, update the image.url and image.tag properties of ao and runtime: 

    ao:
  image:
    url: "gcr.io/apigee-release/hybrid/apigee-operators"
    tag: "1.11.2-hotfix.2"
runtime:
  image:
    url: "gcr.io/apigee-release/hybrid/apigee-runtime"
    tag: "1.11.2-hotfix.2"

  2. Install the hotfix release:

    • For Helm-managed releases, update the apigee-operator with the helm upgrade command and your current overrides files: 

      helm upgrade operator apigee-operator/ \
  --namespace apigee-system \
  --atomic \
  -f overrides.yaml

      For each environment in your Apigee org:

      helm upgrade ENV_NAME apigee-env/ \
  --namespace apigee \
  --atomic \
  -f overrides.yaml

    • For apigeectl-managed releases:

      1. install the hotfix release with apigeectl init using your updated overrides file: 

        ${APIGEECTL_HOME}/apigeectl init -f OVERRIDES_FILE --dry-run=client

        Followed by:

        ${APIGEECTL_HOME}/apigeectl init -f OVERRIDES_FILE

      2. Apply the hotfix release with apigeectl apply: 

        ${APIGEECTL_HOME}/apigeectl apply -f OVERRIDES_FILE --all-envs --dry-run=client

        Followed by:

        ${APIGEECTL_HOME}/apigeectl apply -f OVERRIDES_FILE --all-envs
Bug ID Description
351868444 Tolerations are now working for Redis's Envoy pod.
297539870 HTTPTargetConnection property io.timeout.millis is now honored correctly when using websockets.

July 30, 2024

1.12.1-hotfix.1

hybrid 1.12.1-hotfix.1

On July 30, 2024 we released an updated version of the Apigee hybrid software, 1.12.1-hotfix.1.

Note: This release reflects a change to the component images and not the Helm chart templates. If your hybrid installation is currently on Apigee hybrid v1.12.1, Apply this hotfix with the following steps:

  1. In your overrides file, update the ao.image.url and ao.image.tag properties: 

    ao:
  image:
    url: "gcr.io/apigee-release/hybrid/apigee-operators"
    tag: "1.12.1-hotfix.1"

  2. Install the hotfix release. Update the apigee-operator component with the helm upgrade command and your current overrides files: 

       helm upgrade operator apigee-operator/ \
     --namespace apigee-system \
     --atomic \
     -f overrides.yaml
Bug ID Description
351868444 Tolerations are now working for Redis's Envoy pod.

June 28, 2024

v1.12.1

hybrid v1.12.1

On June 28, 2024 we released an updated version of the Apigee hybrid software, 1.12.1.

Bug ID Description
347798999 Fixed an issue preventing configuration of forward proxies for OpenTelemetry collector pods.
345501069 Fixed issue with Hybrid Guardrails resource configuration preventing the Guardails pod from starting.
341797795 Autofill the Hybrid Guardrails checkpoint value if a checkpoint is not provided.
340248314 Added support for targetCPUUtilizationPercentage to apigeeIngressGateway and ingressGateways. The default value is 75.
324779388 Improved error handling for backup and restore.
311489774 Removed inclusion of Java in Cassandra client image..
310338146 Fixed invalid download directory output from the create-service-account tool.
300135626 Removed inclusion of Java in Cassandra Backup Utility image.
239523766 Remove "Unable to evaluate jsonVariable, returning null" logging string from ExtractVariables Policy
Bug ID Description
345791712 Security fix for fluent-bit.
This addresses the following vulnerability:
335910066 Security fixes for apigee-kube-rbac-proxy.
This addresses the following vulnerability:
335909737 Security fixes for apigee-asm-ingress.
This addresses the following vulnerabilities:
335909397 Security fixes for apigee-open-telemetry-collector.
This addresses the following vulnerability:
335908990 Security fixes for apigee-asm-istiod.
This addresses the following vulnerabilities:
335908985 Security fix for apigee-prometheus-adapter.
This addresses the following vulnerabilities: .
335908657 Security fixes for apigee-prom-prometheus.
This addresses the following vulnerabilities:
335908139 Security fix for fluent-bit.
This addresses the following vulnerability:
332821083 Security fix for apigee-operators.
This addresses the following vulnerability:
317528509 Security fixes for apigee-synchronizer.
This addresses the following vulnerabilities:
308835165 Security fix for apigee-synchronizer.
This addresses the following vulnerability:
N/A Security fixes for apigee-asm-ingress.
This addresses the following vulnerabilities:
N/A Security fixes for apigee-asm-istiod.
This addresses the following vulnerabilities:
N/A Security fixes for apigee-cassandra-backup-utility.
This addresses the following vulnerabilities:
N/A Security fixes for apigee-fluent-bit.
This addresses the following vulnerabilities:
N/A Security fixes for apigee-hybrid-cassandra.
This addresses the following vulnerabilities:
N/A Security fixes for apigee-kube-rbac-proxy.
This addresses the following vulnerabilities:
N/A Security fixes for apigee-prometheus-adapter.
This addresses the following vulnerabilities:
N/A Security fixes for apigee-stackdriver-prometheus-sidecar.
This addresses the following vulnerabilities:
1.11.2-hotfix.1

hybrid 1.11.2-hotfix.1

On June 28, 2024 we released an updated version of the Apigee hybrid software, 1.11.2-hotfix.1.

Note: This release reflects a change to the Helm chart templates and not a change to the images. If your hybrid installation is currently on Apigee hybrid v1.11.2, Apply this hotfix with the following steps:

  1. In your overrides file, update the ao.image url and tag: 

    ao:
  image:
    url: "gcr.io/apigee-release/hybrid/apigee-operators"
    tag: "1.11.2-hotfix.1"

  2. Install the hotfix release:

    • For Helm-managed releases, update the apigee-operator with the helm upgrade command and your current overrides files: 

      helm upgrade operator apigee-operator/ \
  --namespace apigee-system \
  --atomic \
  -f overrides.yaml

    • For apigeectl-managed releases, install the hotfix release with apigeectl init using your updated overrides files: 

      ${APIGEECTL_HOME}/apigeectl init -f OVERRIDES_FILE --dry-run=client

      Followed by:

      ${APIGEECTL_HOME}/apigeectl init -f OVERRIDES_FILE
Bug ID Description
347997965 Upgrading to Apigee Hybrid 1.11.2 and 1.10.5 can cause missing metrics.
1.10.5-hotfix.1

hybrid 1.10.5-hotfix.1

On June 28, 2024 we released an updated version of the Apigee hybrid software, 1.10.5-hotfix.1.

Note: This release reflects a change to the Helm chart templates and not a change to the images. If your hybrid installation is currently on Apigee hybrid v1.10.5, Apply this hotfix with the following steps:

  1. In your overrides file, update the ao.image url and tag: 

    ao:
  image:
    url: "gcr.io/apigee-release/hybrid/apigee-operators"
    tag: "1.10.5-hotfix.1"

  2. Install the hotfix release with apigeectl init using your updated overrides files: 

    ${APIGEECTL_HOME}/apigeectl init -f OVERRIDES_FILE --dry-run=client

    Followed by:

    ${APIGEECTL_HOME}/apigeectl init -f OVERRIDES_FILE
Bug ID Description
347997965 Upgrading to Apigee Hybrid 1.11.2 and 1.10.5 can cause missing metrics.

June 17, 2024

v1.10.5

hybrid v1.10.5

On June 17, 2024 we released an updated version of the Apigee hybrid software, 1.10.5.

Bug ID Description
329540114 Security fix for apigee-installer.
This addresses the following vulnerability:
317528509 Security fix for apigee-synchronizer.
This addresses the following vulnerabilities:
N/A Security fixes for apigee-synchronizer.
This addresses the following vulnerability:
N/A Security fixes for apigee-asm-ingress and apigee-asm-istiod.
This addresses the following vulnerability:
N/A Security fixes for apigee-cassandra-backup-utility.
This addresses the following vulnerabilities:
N/A Security fixes for apigee-connect-agent.
This addresses the following vulnerability:
N/A Security fixes for apigee-diagnostics-collector.
This addresses the following vulnerabilities:
N/A Security fixes for apigee-fluent-bit.
This addresses the following vulnerabilities:
N/A Security fixes for apigee-hybrid-cassandra.
This addresses the following vulnerabilities:
N/A Security fixes for apigee-hybrid-cassandra-client.
This addresses the following vulnerabilities:
N/A Security fixes for apigee-kube-rbac-proxy.
This addresses the following vulnerabilities:
N/A Security fixes for apigee-mart-server.
This addresses the following vulnerabilities:
N/A Security fixes for apigee-mint-task-scheduler.
This addresses the following vulnerabilities:
N/A Security fixes for apigee-prom-prometheus.
This addresses the following vulnerabilities:
N/A Security fixes for apigee-prometheus-adapter.
This addresses the following vulnerabilities:
N/A Security fixes for apigee-runtime.
This addresses the following vulnerabilities:
N/A Security fixes for apigee-udca.
This addresses the following vulnerabilities:

June 10, 2024

v1.11.2

hybrid v1.11.2

On June 10, 2024 we released an updated version of the Apigee hybrid software, 1.11.2.

Bug ID Description
340248314 Added support for targetCPUUtilizationPercentage to apigeeIngressGateway and ingressGateways for hybrid installations managed with Helm. The default value is 75.
Note: targetCPUUtilizationPercentage is not supported for apigeectl.
324779388 Improved error handling for backup and restore.
311489774 Removed inclusion of Java and Python installations in Cassandra client image.
300135626 Removed inclusion of Java and Python installations in Cassandra Backup Utility image.
181569113 Fixed an issue in new debug session creation.
Bug ID Description
345520525 Security fixes for apigee-asm-ingress. and apigee-asm-istiod.
This addresses the following vulnerabilities:
335908139 Security fixes for apigee-fluent-bit.
This addresses the following vulnerability:
333121802 Security fixes for apigee-cassandra-backup-utility. and apigee-hybrid-cassandra.
This addresses the following vulnerability:
317528509 Security fix for apigee-synchronizer.
This addresses the following vulnerabilities:
317447390 Security fix for apigee-operators.
This addresses the following vulnerability:
329762216 Security fix for apigee-installer.
This addresses the following vulnerability:
308835165 Security fixes for apigee-synchronizer.
This addresses the following vulnerability:
308926079 Security fixes for apigee-kube-rbac-proxy.
This addresses the following vulnerabilities:
300091388 Security fixes for Apigee Connect Agent.
This addresses the following vulnerability:
N/A Security fixes for apigee-cassandra-backup-utility.
This addresses the following vulnerability:
N/A Security fixes for apigee-diagnostics-collector.
This addresses the following vulnerabilities:
N/A Security fixes for apigee-fluent-bit.
This addresses the following vulnerabilities:
N/A Security fixes for apigee-hybrid-cassandra-client.
This addresses the following vulnerabilities:
N/A Security fixes for apigee-hybrid-cassandra.
This addresses the following vulnerabilities:
N/A Security fixes for apigee-kube-rbac-proxy.
This addresses the following vulnerabilities:
N/A Security fixes for apigee-mart-server.
This addresses the following vulnerabilities:
N/A Security fixes for apigee-mint-task-scheduler.
This addresses the following vulnerabilities:
N/A Security fixes for apigee-prom-prometheus.
This addresses the following vulnerabilities:
N/A Security fixes for apigee-prometheus-adapter.
This addresses the following vulnerabilities:
N/A Security fixes for apigee-redis.
This addresses the following vulnerabilities:
N/A Security fixes for apigee-runtime.
This addresses the following vulnerabilities:
N/A Security fixes for apigee-synchronizer.
This addresses the following vulnerabilities:
N/A Security fixes for apigee-udca.
This addresses the following vulnerabilities:
N/A Security fixes for apigee-watcher.
This addresses the following vulnerabilities:

May 28, 2024

1.12.0-hotfix.1

ANNOUNCEMENT

hybrid 1.12.0-hotfix.1

On May 28, 2024 we released an updated version of the Apigee hybrid software, 1.12.0-hotfix.1.

Note: This release reflects a change to the Helm chart templates and not a change to the images. If your hybrid installation is currently on Apigee hybrid v1.12.0, you can install this hotfix release by downloading the charts with the version tag 1.12.0-hotfix.1 and updating the apigee-operator and apigee-datastore charts with the helm upgrade command and your current overrides files.

For example: 

export CHART_REPO=oci://us-docker.pkg.dev/apigee-release/apigee-hybrid-helm-charts
export CHART_VERSION=1.12.0-hotfix.1
helm pull $CHART_REPO/apigee-operator --version $CHART_VERSION --untar
helm pull $CHART_REPO/apigee-datastore --version $CHART_VERSION --untar

helm upgrade operator apigee-operator/ \
  --namespace apigee-system \
  --atomic \
  -f overrides.yaml 
helm upgrade datastore apigee-datastore/ \
  --namespace apigee \
  --atomic \
  -f overrides.yaml
Bug ID Description
340889560 Added csi to the apigee-logger SCC.
339849002 Hashicorp Vault integration issues fixed for Google Service Account for Cassandra Backup/Restore.

April 17, 2024

v1.12.0

hybrid v1.12.0

On April 17, 2024 we released an updated version of the Apigee hybrid software, v1.12.0.

For information on upgrading, see Upgrading Apigee hybrid to version v1.12.0. For information on new installations, see The big picture.

A new suite of metrics for monitoring Apigee proxies and target endpoints is now available for Hybrid 1.12.

You can now add your own contractEncryptionKey for new Apigee hybrid installations. For details, see Data encryption.

The JAR file dependencies required to create a Java callout are now hosted securely in Artifact Registry.

For more information on downloading the JAR dependencies from Artifact Regsitry, see Compile your code with Maven.

Hybrid 1.12 validates required conditions are satisfied before allowing Runtime services to be created. See Diagnosing issues with guardrails.

Apigee hybrid now supports Workload Identity Federation for component authentication on AKS and EKS installations. See Enabling Workload Identity Federation on AKS and EKS.

Hybrid v1.12 now supports storing service account keys in Hashicorp Vault. See Storing service account keys in Hashicorp Vault.

The apigeectl command-line tool is deprecated. as of April 17, 2024. The apigeectl tool is not supported for Apigee hybrid v1.12. Support for apigeectl for hybrid v1.10 and v1.11 will end on April 17, 2025. For more information, see apigeectl deprecation.

The Proxyv2 and targetv2 metrics suite is deprecated. The Apigee hybrid v1.12 release supports the new proxy and target metrics by default. Support for Proxyv2 and targetv2 metrics in hybrid v 1.10 and v1.11 will end on April 17, 2025. For more information, see ProxyV2 and TargetV2 deprecation.

Bug ID Description
284034011 Modified Apigee Watcher and Apigee Ingress to leverage a sidecar instead of pod/exec for collecting ingress routing status.
298202120 The Datastore component now uses Cassandra 4.
311705715 Use a non-default service account for the remove-dc component. (Fixed in Apigee hybrid 1.10.3-hotfix.4, 1.10.4, and 1.11.1)
306341401 Fixed regression where virtualhost cipherSuites overrides weren't being used. (Fixed in Apigee hybrid 1.10.4 and 1.11.1)
302186503 Added the missing HTTP proxy template settings to the Apigee Hybrid Helm datastore component. (Fixed in Apigee hybrid 1.10.4)
300542690 Added dedicated service accounts for Apigee Connect, Redis, and UDCA to prevent Kubernetes from automatically injecting credentials for a specified Service Account or the default Service Account. (Fixed in Apigee hybrid 1.10.4)
277353680 Fixed issue causing target server HealthMonitors to continue beyond revision or deletion of the proxy.

Target health checks are now terminated as soon as the proxy is removed from the runtime (undeployed or deleted). Note: There may be a delay between removal of the proxy and termination of the target server health checks. (Fixed in Apigee hybrid 1.10.4)

These security bugs were fixed in Apigee hybrid v1.12.0

Bug ID Description
N/A Security fixes for apigee-cassandra-backup-utility
This addresses the following vulnerabilities:
N/A Security fixes for apigee-hybrid-cassandra-client
This addresses the following vulnerabilities:
N/A Security fixes for apigee-hybrid-cassandra
This addresses the following vulnerabilities:
N/A Security fixes for apigee-kube-rbac-proxy
This addresses the following vulnerabilities:
N/A Security fixes for apigee-prometheus-adapter
This addresses the following vulnerability:
N/A Security fixes for apigee-udca
This addresses the following vulnerabilities:
N/A Security fixes for apigee-watcher
This addresses the following vulnerability:
N/A Security fixes for apigee-connect-agent
This addresses the following vulnerability:
N/A Security fixes for apigee-fluent-bit
This addresses the following vulnerabilities:

These security bugs were fixed in Apigee hybrid v1.10.4.

Bug ID Description
315034009 Security fixes: apigee-asm-ingress and apigee-asm-istiod (ingressgateway and ingressgateway-controller) are upgraded to Service Mesh version 1.17.8-asm.4.
This addresses the following vulnerabilities:
311167948 A security issue was addressed.
303460289 Security fixes for apigee-prometheus-adapter.
This addresses the following vulnerabilities:
303459588 Security fixes for apigee-prom-prometheus.
This addresses the following vulnerabilities:
300319489 Security fixes for fluentd.
This addresses the following vulnerabilities:
294892189 Security fixes for apigee-diagnostics-collector.
This addresses the Guava vulnerability:
N/A Security fixes for apigee-cassandra-backup-utility and apigee-prom-prometheus.
This addresses the following vulnerabilities:
N/A Security fixes for apigee-fluent-bit.
This addresses the following vulnerabilities:
N/A Security fixes for apigee-hybrid-cassandra and apigee-hybrid-cassandra-client.
This addresses the following vulnerabilities:
N/A Security fixes for apigee-installer, apigee-operators, and apigee-watcher.
This addresses the following vulnerabilities:
N/A Security fixes for apigee-kube-rbac-proxy.
This addresses the following vulnerabilities:

These security bugs were fixed in Apigee hybrid v1.11.1.

Bug ID Description
315034009 Security fixes: apigee-asm-ingress and apigee-asm-istiod (ingressgateway and ingressgateway-controller) are upgraded to Service Mesh version 1.17.8-asm.4.
This addresses the following vulnerabilities:
303460289 Security fixes to apigee-prometheus-adapter.
This addresses the following vulnerabilities:
303459588 Security fixes to apigee-prom-prometheus.
This addresses the following vulnerabilities:
303292806 Restrict connections from the Cassandra backup utility to Cassandra server pods in the apigee namespace.
N/A Security fixes to apigee-cassandra-backup-utility.
This addresses the following vulnerabilities:
N/A Security fixes to apigee-fluent-bit.
This addresses the following vulnerabilities:
N/A Security fixes to apigee-hybrid-cassandra.
This addresses the following vulnerabilities:
N/A Security fixes to apigee-hybrid-cassandra-client.
This addresses the following vulnerabilities:
N/A Security fixes to apigee-kube-rbac-proxy.
This addresses the following vulnerabilities:
N/A Security fixes to apigee-installer, apigee-operators, and apigee-watcher.
This addresses the following vulnerabilities:

These security bugs were fixed in Apigee hybrid v1.11.1-hotfix.1.

Bug ID Description
324460830 Security fix for apigee-ingress.
This addresses the following vulnerabilities:

These security bugs were fixed in Apigee hybrid v1.10.4-hotfix.1.

Bug ID Description
324460830 Security fix for apigee-ingress.
This addresses the following vulnerabilities:

February 09, 2024

1.11.1-hotfix.1

hybrid 1.11.1-hotfix.1

On February 9, 2024 we released an updated version of the Apigee hybrid software, v1.11.1-hotfix.1.

This release addresses the security concerns in GCP-2024-007 from Google Anthos Service Mesh.

Bug ID Description
324460830 Security fix for apigee-ingress.
This addresses the following vulnerabilities:
1.10.4-hotfix.1

hybrid 1.10.4-hotfix.1

On February 9, 2024 we released an updated version of the Apigee hybrid software, v1.10.4-hotfix.1.

This release addresses the security concerns in GCP-2024-007 from Google Anthos Service Mesh.

Bug ID Description
324460830 Security fix for apigee-ingress.
This addresses the following vulnerabilities:

December 21, 2023

v1.11.1

hybrid v1.11.1

On December 21, 2023 we released an updated version of the Apigee hybrid software, 1.11.1.

Bug ID Description
311705715 Use a non-default service account for the remove-dc component.
306341401 Fixed regression where virtualhost cipherSuites overrides weren't being used.
Bug ID Description
315034009 Security fixes: apigee-asm-ingress and apigee-asm-istiod (ingressgateway and ingressgateway-controller) are upgraded to Service Mesh version 1.17.8-asm.4.
This addresses the following vulnerabilities:
303460289 Security fixes to apigee-prometheus-adapter.
This addresses the following vulnerabilities:
303459588 Security fixes to apigee-prom-prometheus.
This addresses the following vulnerabilities:
303292806 Restrict connections from the Cassandra backup utility to Cassandra server pods in the apigee namespace.
N/A Security fixes to apigee-cassandra-backup-utility.
This addresses the following vulnerabilities:
N/A Security fixes to apigee-fluent-bit.
This addresses the following vulnerabilities:
N/A Security fixes to apigee-hybrid-cassandra.
This addresses the following vulnerabilities:
N/A Security fixes to apigee-hybrid-cassandra-client.
This addresses the following vulnerabilities:
N/A Security fixes to apigee-kube-rbac-proxy.
This addresses the following vulnerabilities:
N/A Security fixes to apigee-installer, apigee-operators, and apigee-watcher.
This addresses the following vulnerabilities:

December 18, 2023

v1.10.4

hybrid v1.10.4

On December 18, 2023 we released an updated version of the Apigee hybrid software, 1.10.4.

Bug ID Description
311705715 Use a non-default service account for the remove-dc component.
306341401 Fixed regression where virtualhost cipherSuites overrides weren't being used.
302186503 Add the missing HTTP proxy template settings to the Apigee Hybrid Helm datastore component.
300542690 Added dedicated service accounts for Apigee Connect, Redis, and UDCA to prevent Kubernetes from automatically injecting credentials for a specified Service Account or the default Service Account.
277353680 Fixed issue causing target server HealthMonitors to continue beyond revision or deletion of the proxy.

Target health checks are now terminated as soon as the proxy is removed from the runtime (undeployed or deleted). Note: There may be a delay between removal of the proxy and termination of the target server health checks.
Bug ID Description
315034009 Security fixes: apigee-asm-ingress and apigee-asm-istiod (ingressgateway and ingressgateway-controller) are upgraded to Service Mesh version 1.17.8-asm.4.
This addresses the following vulnerabilities:
311167948 A security issue was addressed.
303460289 Security fixes for apigee-prometheus-adapter.
This addresses the following vulnerabilities:
303459588 Security fixes for apigee-prom-prometheus.
This addresses the following vulnerabilities:
300319489 Security fixes for fluentd.
This addresses the following vulnerabilities:
294892189 Security fixes for apigee-diagnostics-collector.
This addresses the Guava vulnerability:
N/A Security fixes for apigee-cassandra-backup-utility and apigee-prom-prometheus.
This addresses the following vulnerabilities:
N/A Security fixes for apigee-fluent-bit.
This addresses the following vulnerabilities:
N/A Security fixes for apigee-hybrid-cassandra and apigee-hybrid-cassandra-client.
This addresses the following vulnerabilities:
N/A Security fixes for apigee-installer, apigee-operators, and apigee-watcher.
This addresses the following vulnerabilities:
N/A Security fixes for apigee-kube-rbac-proxy.
This addresses the following vulnerabilities:

December 01, 2023

1.10.3-hotfix.4

hybrid 1.10.3-hotfix.4

On December 1, 2023 we released an updated version of the Apigee hybrid software, v1.10.3-hotfix.4.

Bug ID Description
311705715 Mount a dedicated service account to the remove-dc component.
Bug ID Description
311167948 A security issue was addressed.

November 17, 2023

v1.11.0

hybrid v1.11.0

On November 17, 2023 we released an updated version of the Apigee hybrid software, v1.11.0.

Helm charts management for Apigee hybrid

Starting in version v1.11.0, you have the choice of installing and managing your clusters with either Helm or apigeectl. You cannot manage a cluster with both. Apigee recommends using Helm for new hybrid installations. See Apigee hybrid Helm charts reference.

Vault integration for Cassandra credentials (preview)

Starting in version v1.11.0, you can store Cassandra credentials in Hashicorp Vault.
Note: Using Vault requires Helm management of your Apigee installation.
See Storing Cassandra credentials in Hashicorp Vault.

Vault integration is in preview as of the Apigee hybrid 1.11.0 release.

Apigee Advance API Security Actions for Apigee hybrid

Advanced API Security's new Security Actions feature is now available in Apigee hybrid.

With this release, the HeaderName element is available as a child element of Authentication. This element appears in the ServiceCallout and ExternalCallout policies, and in the TargetEndpoint proxy configuration.

By default, when an Authentication configuration is present, Apigee generates and injects a bearer token into the Authorization header, in the message sent to the target system. The new HeaderName element allows you to specify the name of a different header to hold that bearer token.

Bug ID Description
295929616 Installation of Hybrid 1.10.x would fail on OpenShift due to out of memory issues. (Fixed in Apigee hybrid v1.10.3)
294069799 Updated the security context settings for the Apigee Hybrid Backup and Restore pod.
292571089 An error with support for CSI backup and restore for Cassandra was fixed. (Fixed in Apigee hybrid v1.10.3)
292118812 Fixed UDCA regression in Hybrid 1.10.1 where UDCA would ignore forward proxy configuration. (Fixed in Apigee hybrid v1.10.2)
289254725 Implemented a fix to prevent failure of proxy deployments that include the OASValidation policy. (Fixed in Apigee hybrid v1.10.1)
287321226 Security context has been corrected for apigee-prom-prometheus to avoid privilege escalation. (Fixed in Apigee hybrid v1.10.3)
277353680 Fixed issue causing target server HealthMonitors to continue beyond revision or deletion of the proxy.

Target health checks are now terminated as soon as the proxy is removed from the runtime (undeployed or deleted). Note: There may be a delay between removal of the proxy and termination of the target server health checks. (Fixed in Apigee X)
240180122 Disable privilege escalation on the cassandra container by moving the ulimit settings to the newly introduced initContainer "apigee-cassandra-ulimit-init".

If you are using security controls with gatekeeper, ensure that apigee-cassandra-ulimit-init initContainer can runAs user, group as 0 and allow capabilities IPC_LOCK and SYS_RESOURCES. (Fixed in Apigee hybrid v1.11.0)
205666368 Fixed issue with default validation of TLS target endpoint certificates.

To enable strict SSL on southbound connections to a proxy target endpoint, add the tag <Enforce>true</Enforce> in the target <SSLInfo> block.

See About setting TLS options in a target endpoint or target server.

See also Known Issue #205666368.

(Fixed in Apigee hybrid v1.10.3-hotfix.1)
158132963 Added improvements to capture relevant target flow variables in trace and analytics in case of target timeouts. (Fixed in Apigee hybrid v1.10.2)
Bug ID Description
303292806 Set backup utility to only connect to Cassandra server pods in the apigee namespace. (Fixed in Apigee hybrid v1.10.3-hotfix.3)
300542690 Added dedicated service accounts for Apigee Connect, Redis, and UDCA to prevent Kubernetes from automatically injecting credentials for a specified ServiceAccount or the default ServiceAccount. (Fixed in Apigee hybrid v1.10.3-hotfix.3)
297938600,
297938559,
297938486,
294892344		 Security fixes for apigee-diagnostics-collector. (Fixed in Apigee hybrid v1.10.3)
This addresses the following vulnerabilities:
297938498,
297938487		 Security fixes for apigee-fluent-bit.(Fixed in Apigee hybrid v1.10.3)
This addresses the following vulnerabilities:
297938441 Security fixes for apigee-runtime. (Fixed in Apigee hybrid v1.10.3)
This addresses the following vulnerabilities:
297286274 Security fixes for apigee-installer. (Fixed in Apigee hybrid v1.10.3)
This addresses the following vulnerabilities:
296719459,
296719400,
296719348,
296719307,
296719306,
296719188,
296719187,
296719186,
296719115,
296719018,
296718937,
296718918,
296718917,
296718916,
296716670,
296716669,
296716472,
296716471,
296715155		 Security fixes for apigee-hybrid-cassandra. (Fixed in Apigee hybrid v1.10.3)
This addresses the following vulnerabilities:
296717666,
296717283,
296716668,
296716667,
296716650,
296716635,
296716634,
296716633,
296716470,
296716234,
296715734,
296715733,
296715154,
296715153		 Security fixes for apigee-hybrid-cassandra-client. (Fixed in Apigee hybrid v1.10.3)
This addresses the following vulnerabilities:
296717665,
296717664,
296717663,
296717662,
296717185,
296716666,
296716649,
296716632,
296716468,
296716467,
296716232,
296715152,
296715151,
296714218		 Security fixes for apigee-cassandra-backup-utility. (Fixed in Apigee hybrid v1.10.3)
This addresses the following vulnerabilities:
295936113 Security fixes for apigee-mart-server. (Fixed in Apigee hybrid v1.10.3)
This addresses the following vulnerability:
294906706 Security fixes for apigee-prom-prometheus. (Fixed in Apigee hybrid v1.10.3)
This addresses the following vulnerabilities:
293925856 Security fixes for apigee-prometheus-adapter. (Fixed in Apigee hybrid v1.10.3)
This addresses the following vulnerabilities:
293348130 Security fixes for apigee-udca. (Fixed in Apigee hybrid v1.10.2)
This addresses the following vulnerabilities:
291994501 Security fixes for apigee-operator and apigee-watcher. (Fixed in Apigee hybrid v1.10.2)
This addresses the following vulnerabilities:
291994501 Security fixes for apigee-installer. (Fixed in Apigee hybrid v1.10.2)
This addresses the following vulnerabilities:
290829031 Security fixes for apigee-hybrid-cassandra, apigee-cassandra-client, and cassandra-backup-utility. (Fixed in Apigee hybrid v1.10.2)
This addresses the following vulnerabilities:
290829028 Security fixes for Apigee Connect and apigee-connect-agent and apigee-redis. (Fixed in Apigee hybrid v1.10.2)
This addresses the following vulnerabilities:
290068742 Security fixes for apigee-udca. (Fixed in Apigee hybrid v1.10.1)
This addresses the following vulnerability:
290067464, 297938583 Security fixes for apigee-stackdriver-logging-agent. (Fixed in Apigee hybrid v1.10.1)
This addresses the following vulnerabilities:
290065830 Security fixes for apigee-udca. (Fixed in Apigee hybrid v1.10.1)
This addresses the following vulnerability:
281561243 Security fixes for apigee-diagnostics-collector, apigee-mart-server, apigee-mint-task-scheduler, apigee-runtime, and apigee-synchronizer. (Fixed in Apigee hybrid v1.10.1)
This addresses the following vulnerability:
N/A Security fixes for apigee-prometheus-adapter. (Fixed in Apigee hybrid hybrid v1.11)
This addresses the following vulnerabilities:
N/A Security fixes for apigee-prom-prometheus/master. (Fixed in Apigee hybrid v1.11)
This addresses the following vulnerabilities:
N/A Security fixes for apigee-kube-rbac-proxy. (Fixed in Apigee hybrid hybrid v1.11)
This addresses the following vulnerabilities:
N/A Security fixes for apigee-hybrid-cassandra. (Fixed in Apigee hybrid hybrid v1.11)
This addresses the following vulnerabilities:
N/A Security fixes for apigee-fluent-bit. (Fixed in Apigee hybrid hybrid v1.11)
This addresses the following vulnerabilities:
N/A Security fixes for apigee-diagnostics-collector, apigee-mart-server, apigee-mint-task-scheduler, apigee-runtime, and apigee-synchronizer. (Fixed in Apigee hybrid hybrid v1.11)
This addresses the following vulnerabilities:
N/A Security fixes for apigee-cassandra-backup-utility, apigee-hybrid-cassandra-client, and apigee-connect-agent. (Fixed in Apigee hybrid v1.11)
This addresses the following vulnerabilities:
N/A Security fixes for apigee-asm-ingress and apigee-asm-istiod. (Fixed in Apigee hybrid v1.11)
This addresses the following vulnerabilities:

October 17, 2023

1.10.3-hotfix.3

hybrid v1.10.3-hotfix.3

On October 17, 2023 we released an updated version of the Apigee hybrid software, v1.10.3-hotfix.3.

Bug ID Description
303292806 Set backup utility to only connect to Cassandra server pods in the apigee namespace.
300542690 Added dedicated service accounts for Apigee Connect, Redis, and UDCA to prevent Kubernetes from automatically injecting credentials for a specified ServiceAccount or the default ServiceAccount.

October 13, 2023

1.10.3-hotfix.2 & 1.9.4-hotfix.1

hybrid v1.10.3-hotfix.2

On October 13, 2023 we released an updated version of the Apigee hybrid software, v1.10.3-hotfix.2.

Bug ID Description
304681330 Security fix for apigee-ingress.
This addresses the following vulnerability:
CVE-2023-44487
305127632 Security bulletin published.
GCP-2023-032

hybrid v1.9.4-hotfix.1

On October 13, 2023 we released an updated version of the Apigee hybrid software, v1.9.4-hotfix.1.

Bug ID Description
304681330 Security fix for apigee-ingress.
This addresses the following vulnerability:
CVE-2023-44487
305127632 Security bulletin published.
GCP-2023-032

Description

A Denial-of-Service (DoS) vulnerability was recently discovered in multiple implementations of the HTTP/2 protocol (CVE-2023-44487), including the Apigee Ingress (Anthos Service Mesh) server used by Apigee hybrid. The vulnerability could lead to a DoS of Apigee API management functionality.

Affected Products

Apigee hybrid instances that allow HTTP/2 requests to reach the Apigee Ingress are affected. Customers should verify if the load balancers fronting their Apigee hybrid ingresses allow for HTTP/2 requests to reach the Apigee Ingress service.

What Should I Do?

Apigee hybrid customers will need to upgrade to one of the following patch versions:

What Vulnerabilities Are Addressed By These Patches?

The vulnerability, CVE-2023-44487, allows an attacker to execute a denial-of-service attack on Apigee ingresses.

September 14, 2023

1.10.3-hotfix.1

hybrid 1.10.3-hotfix.1

On September 14, 2023 we released an updated version of the Apigee hybrid software, 1.10.3-hotfix.1.

Bug ID Description
205666368 Fixed issue with default validation of TLS target endpoint certificates.

To enable strict SSL on southbound connections to a proxy target endpoint, add the tag <Enforce>true</Enforce> in the target <SSLInfo> block.

See About setting TLS options in a target endpoint or target server.

See also Known Issue #205666368.

August 31, 2023

v1.10.3

hybrid v1.10.3

On August 31, 2023 we released an updated version of the Apigee hybrid software, v1.10.3.

Bug ID Description
295929616 Installation of Hybrid 1.10.x would fail on OpenShift due to out of memory issues.
292571089 An error with support for CSI backup and restore for Cassandra was fixed.
287321226 Security context has been corrected for apigee-prom-prometheus to avoid privilege escalation.
Bug ID Description
296717665,
296717664,
296717663,
296717662,
296717185,
296716666,
296716649,
296716632,
296716468,
296716467,
296716232,
296715152,
296715151,
296714218		 Security fixes for apigee-cassandra-backup-utility.
This addresses the following vulnerabilities:
297938600,
297938559,
297938486,
294892344		 Security fixes for apigee-diagnostics-collector.
This addresses the following vulnerabilities:
297938498,
297938487		 Security fixes for apigee-fluent-bit.
This addresses the following vulnerabilities:
296719459,
296719400,
296719348,
296719307,
296719306,
296719188,
296719187,
296719186,
296719115,
296719018,
296718937,
296718918,
296718917,
296718916,
296716670,
296716669,
296716472,
296716471,
296715155		 Security fixes for apigee-hybrid-cassandra.
This addresses the following vulnerabilities:
296717666,
296717283,
296716668,
296716667,
296716650,
296716635,
296716634,
296716633,
296716470,
296716234,
296715734,
296715733,
296715154,
296715153		 Security fixes for apigee-hybrid-cassandra-client.
This addresses the following vulnerabilities:
297286274 Security fixes for apigee-installer.
This addresses the following vulnerabilities:
295936113 Security fixes for apigee-mart-server.
This addresses the following vulnerability:
297938441 Security fixes for apigee-runtime.
This addresses the following vulnerabilities:
294906706 Security fixes for apigee-prom-prometheus.
This addresses the following vulnerabilities:
293925856 Security fixes for apigee-prometheus-adapter.
This addresses the following vulnerabilities:

July 31, 2023

v1.10.2

hybrid v1.10.2

On July 31, 2023 we released an updated version of the Apigee hybrid software, v1.10.2.

Bug ID Description
292118812 Fixed UDCA regression in Hybrid 1.10.1 where UDCA would ignore forward proxy configuration.
205666368 Fixed issue with default validation of TLS target endpoint certificates.

To enable strict SSL on southbound connections to a proxy target endpoint, add the tag <Enforce>true</Enforce> in the target <SSLInfo> block.

See About setting TLS options in a target endpoint or target server.

See also Known Issue #205666368.

158132963 Added improvements to capture relevant target flow variables in trace and analytics in case of target timeouts.
Bug ID Description
293348130 Security fixes for apigee-udca.
This addresses the following vulnerabilities:
291994501 Security fixes for apigee-operator and apigee-watcher.
This addresses the following vulnerabilities:
291994501 Security fixes for apigee-installer.
This addresses the following vulnerabilities:
290829031 Security fixes for apigee-hybrid-cassandra, apigee-cassandra-client, and cassandra-backup-utility.
This addresses the following vulnerabilities:
290829028 Security fixes for Apigee Connect and apigee-connect-agent and apigee-redis.
This addresses the following vulnerabilities:

July 21, 2023

v1.9.4

hybrid v1.9.4

On July 21, 2023 we released an updated version of the Apigee hybrid software, v1.9.4.

Bug ID Description
289254725 Implemented a fix to prevent failure of proxy deployments that include the OASValidation policy.
279712107 Added the ability to annotate apigee-ingressgateway-manager pods through overrides.yaml file.
See istiod.annotations for details.
272212164 Cassandra CSI backup could clash with Azure default configuration. The CSI backup script has been fixed to prevent a resource naming issue that could cause backups to fail.
158132963 Added improvements to capture relevant target flow variables in trace and analytics in case of target timeouts.
Bug ID Description
290709899 Security fixes for apigee-diagnostics-collector, apigee-mart-server, apigee-mint-task-scheduler, apigee-runtime, and apigee-synchronizer.
This addresses the following vulnerability:
290829028 Security fixes for Apigee Connect and apigee-connect-agent and apigee-redis.
This addresses the following vulnerabilities:

July 13, 2023

v1.10.1

hybrid v1.10.1

On July 13, 2023 we released an updated version of the Apigee hybrid software, v1.10.1.

Bug ID Description
289254725 Implemented a fix to prevent failure of proxy deployments that include the OASValidation policy.
Bug ID Description
281561243 Security fixes for apigee-diagnostics-collector, apigee-mart-server, apigee-mint-task-scheduler, apigee-runtime, and apigee-synchronizer.
This addresses the following vulnerability:
290067464 Security fixes for apigee-stackdriver-logging-agent.
This addresses the following vulnerability:
290068742 Security fixes for apigee-udca.
This addresses the following vulnerability:
290065830 Security fixes for apigee-udca.
This addresses the following vulnerability:

June 30, 2023

v1.10.0

hybrid v1.10

On June 30, 2023 we released an updated version of the Apigee hybrid software, v1.10.0.

Pre-install Cluster Check Kubernetes job

Starting in version 1.10, Apigee hybrid offers a new tool that examines the hybrid cluster before you install the hybrid runtime. See Step 8: Check cluster readiness .

Automated Issue Surfacing (AIS)

Starting with Apigee hybrid 1.10, Apigee hybrid offers a new tool that examines the hybrid runtime and surfaces issues by running a kubectl command. If the tool detects errors in the cluster, it returns a detailed error message. The error message contains a link to the troubleshooting guide for that specific error. See Automated issue surfacing and Configuration property reference, watcher.

Support for AppGroups (preview)

Starting in version 1.10, Apigee hybrid supports AppGroups, which represent a relationship between one or more apps that are managed by the same set of people. For information, see Using AppGroups to organize app ownership.

AppGroups is in preview as of the Apigee hybrid 1.10 release. See the AppGroups preview launch announcement for details.

Support for environment-level scaling

Starting in version 1.9.3, Apigee hybrid added the following environment configuration properties that enable you to specify environment-specific scaling in the overrides.yaml file:

Documentation: Environment-based scaling

Bug ID Description
181569522 You can now create a new environment with the same name as a deleted environment without needing to perform manual clean-up tasks first. (Fixed in Apigee hybrid v1.8.5 and v1.7.6)
209509030 Apigee Ingressgateway cannot access K8s secret from another namespace.
218567150 The ingress gateway is now configured to consistently preserve UUID in the x-request-id header.
Note: This setting does have some impact on tracing in the ingress gateway. For more information, see pack_trace_reason in "UUID (proto)" in the envoy documentation. (Fixed in Apigee hybrid v1.7.6 and v1.8.3)
223320630 mTLS-related client variables are now set by the Apigee runtime. (Fixed in Apigee hybrid v1.8.6)
245619397 In Apigee hybrid, fluentbit support now includes the NO_PROXY environment variable. (Fixed in Apigee hybrid v1.8.5, v1.8.6, and v1.9.1)
259264961 Added support for ASM v1.15. Please see Known issue 266452840 (Fixed in Apigee hybrid v1.7.6)
260342163 Fixed a narrow scenario where threads in runtime pods ended up consuming 100% CPU. (Fixed in Apigee hybrid v1.9.1)
260372012 Requests failed with 500 response and keyvaluemap.service.ErrorDuringDecryption error after upgrade to Hybrid 1.8. Note: Fixed in Apigee hybrid 1.8.4 and newer. (Fixed in Apigee hybrid v1.8.5)
262699558 The watcher component no longer fails when using Kubernetes Secret to store hybrid service account secret. (Fixed in Apigee hybrid v1.7.6)
263840644 Fixed a conflict with an existing ASM on the cluster. (Fixed in Apigee hybrid v1.8.6)
265374889 Fixed an issue where in some circumstances the Java Callout would to fail due with the following error: Failed to execute JavaCallout. Could not initialize class org.jose4j.jwa.AlgorithmFactoryFactory2. (Fixed in Apigee hybrid v1.9.1)
266411394 Add support for Azure Front Door request headers to /healthz health check. (Fixed in Apigee hybrid v1.8.5 and v1.9.1)
266594584 Websocket was failing in asm 1.15. This was due to incompatible capitalization in variable names between the Anthos Service Mesh overlay.yaml file and the and the Envoy filter apigee-envoyfilter.yaml file. (Fixed in Apigee hybrid v1.8.5 and v1.9.1)
266814873 In certain circumstances, retrieving encrypted KVM entries could fail with an error. This fix ensures that MART will be able to successfully function for environment-scoped KVM entries, even if the encryption key is used in the Org Env configuration or when the keys contain non-UTF8 characters. There is no change to KVM data. (Fixed in Apigee hybrid v1.8.6 and v1.9.1)
266989915
266919136		 In some circumstances, Apigee could return incorrect developer credentials for an app, unless the specific app was selected when requesting the credentials. (Fixed in Apigee hybrid v1.9.1)
267666187 When using a custom Kubernetes service for the Apigee ingress gateway, you can disable the creation of a default load balancer. See Managing Apigee ingress gateway. (Fixed in Apigee hybrid v1.8.6 and v1.9.1)
267691299
265295406		 The Apigee controller uses a dedicated apigee-manager Kubernetes service account, instead of using the default SA. (Fixed in Apigee hybrid v1.8.6 and v1.9.1)
268445095 The validateOrg flag can be set to false to bypass upgrade validation errors when configuration includes HTTP Forward proxy. You can use this to avoid upgrade errors caused by HTTP proxy settings. (Fixed in Apigee hybrid v1.7.6)
268696297 Providing a Kubernetes secret for Cassandra and Redis components is now supported. See cassandra.auth.secret and redis.auth.secret in the Configuration properties reference. (Fixed in Apigee hybrid v1.9.1)
269451743 In certain circumstances, upgrading from Apigee hybrid v1.8.3 to v1.9.0 could fail with an error message when creating the virtual hosts. (Fixed in Apigee hybrid v1.9.1)
269738951 The example network policies are now included in the apigeectl/examples/network-policies directory. see Configuring Kubernetes network policies. (Fixed in Apigee hybrid v1.9.1)
270371160 In Apigee hybrid v1.8.7, we removed certain insecure TLS ciphers. Apigee hybrid supports the TLS cipher suites supported by the Boring FIPS build of Envoy. You can now specify specific cipher suites with the virtualhosts.cipherSuites configuration property in your overrides. (Fixed in Apigee hybrid v1.8.7)

Note: Apigee hybrid only supports the RSA ciphers listed. ECDSA ciphers are not supported.
270371160 In Apigee hybrid v1.9.0, we removed certain insecure TLS ciphers. Apigee hybrid supports the TLS cipher suites supported by the Boring FIPS build of Envoy. You can now specify specific cipher suites with the virtualhosts.cipherSuites configuration property in your overrides. (Fixed in Apigee hybrid v1.9.2)

Note: Apigee hybrid only supports the RSA ciphers listed. ECDSA ciphers are not supported.
271266079 Removed port 80 from the default Kubernetes service of Apigee Ingress Gateway. (Fixed in Apigee hybrid v1.8.6 and v1.9.1)
272212164 Cassandra CSI backup could clash with Azure default configuration. The CSI backup script has been fixed to prevent a resource naming issue that could cause backups to fail. (Fixed in Apigee hybrid v1.9.4 and v1.10.0)
273561434 Some projects were unable to run debug sessions. (Fixed in Apigee hybrid v1.8.8 and v1.9.3)
274292101 In certain circumstances, environment-scoped KVMs in hybrid could cause rollback issues for MART. (Fixed in Apigee hybrid v1.8.6)
274999014 Restrict watcher RBAC to a single K8s namespace
278646149 In certain circumstances, the logger.livenessProbe.timeoutSeconds configuration property was not working as expected. See logger.livenessProbe.timeoutSeconds in the Configuration property reference. (Fixed in Apigee hybrid v1.8.7 and v1.9.2)
279053612 x-forwarded-client-cert (XFCC) HTTP headers handled with the istiod.forwardClientCertDetails configuration property. (Fixed in Apigee hybrid v1.8.7 and v1.9.2)
See the Configuration properties reference for details:
279193831 Envoy has been updated to v1.25.6.. (Fixed in Apigee hybrid v1.8.8)
279712107 Added the ability to annotate apigee-ingressgateway-manager pods through overrides.yaml file. (Fixed in Apigee hybrid v1.8.8)
280544499 Request headers were not seen in debug sessions. (Fixed in Apigee hybrid v1.8.8)
284488296 Removed an unneeded Workload Identity on the Cassandra Schema Validation cron job. (Fixed in Apigee hybrid v1.8.8 and v1.9.3)
Bug ID Description
270371160 In Apigee hybrid v1.9.0, we removed certain insecure TLS ciphers. Apigee hybrid supports the TLS cipher suites supported by the Boring FIPS build of Envoy.

Note: Apigee hybrid only supports the RSA ciphers listed. ECDSA ciphers are not supported.
271266079 Removed port 80 from the default Kubernetes service of Apigee Ingress Gateway. Port 80 is not supported by Apigee ingress gateway. If you are migrating from ASM to Apigee ingress gateway, and followed the instructions in the community post to enable Port 80, it will not work with Apigee Ingress gateway. (Fixed in Apigee hybrid v1.8.6 and v1.9.1)
Bug ID Description
262576079 Security fix for for apigee-envoy. (Fixed in Apigee hybrid v1.10)
This addresses the following vulnerability:
273797045 Security fix for for apigee-diagnostics-collector apigee-synchronizer apigee-udca. (Fixed in Apigee hybrid v1.8.8)
This addresses the following vulnerability:
273800345, 281572616 Security fixes for apigee-diagnostics-collector, apigee-mart-server, apigee-mint-task-scheduler, apigee-runtime, apigee-synchronizer, and apigee-udca. (Fixed in Apigee hybrid v1.8.8 and v1.9.3
This addresses the following vulnerabilities:
273800717 Security fixes for apigee-emulator, apigee-diagnostics-collector, apigee-mart-server, apigee-mint-task-scheduler, apigee-mock-server, apigee-runtime, and apigee-synchronizer. (Fixed in Apigee hybrid v1.8.7 and v1.9.2)
This addresses the following vulnerabilities:
273800965 Security fix for apigee-diagnostics-collector, apigee-mart-server, apigee-mint-task-scheduler, apigee-runtime, and apigee-synchronizer. (Fixed in Apigee hybrid v1.8.7, v1.9.2, and v1.9.3)
This addresses the following vulnerability:
273801301 Security fixes for apigee-mart-server and apigee-runtime.(Fixed in Apigee hybrid v1.8.8 and v1.9.3)
This addresses the following vulnerability:
274112103 Security fixes to the Apigee Controller and Apigee Watcher. (Fixed in Apigee hybrid v1.8.6 and v1.9.1)
This addresses the following vulnerabilities:
275002360 Security fixes for fluent-bit. (Fixed in Apigee hybrid v1.8.6 and v1.9.1)
This addresses the following vulnerabilities:
277367440 Security fixes for Apigee Controller, Watcher, and apigeectl. (Fixed in Apigee hybrid v1.8.7 and v1.9.2)
This addresses the following vulnerabilities:
278313047 Security fixes for apigee-stackdriver-logging-agent. (Fixed in Apigee hybrid v1.9.2)
This addresses the following vulnerabilities:
279194142 Fixes build issues to achieve FIPS compliance. (Fixed in Apigee hybrid v1.8.7 and v1.9.2)
281561243 Security fix for apigee-diagnostics-collector, apigee-mint-task-scheduler, apigee-runtime, and apigee-synchronizer. (Fixed in Apigee hybrid v1.8.8 and v1.9.3)
This addresses the following vulnerability:
283826216 Security fixes for apigee-ingressgateway. (Fixed in Apigee hybrid v1.9.3)
This addresses the following vulnerabilities:
283826785 Security fixes for istiod. (Fixed in Apigee hybrid v1.9.3)
This addresses the following vulnerabilities:

June 14, 2023

v1.8.8

hybrid v1.8.8

On June 14, 2023 we released an updated version of the Apigee hybrid software, v1.8.8.

Bug ID Description
273561434 Some projects were unable to run debug sessions..
279193831 Envoy has been updated to v1.25.6..
279712107 Added the ability to annotate apigee-ingressgateway-manager pods through overrides.yaml file.
280544499 Request headers were not seen in debug sessions.
284488296 Removed an unneeded Workload Identity on the Cassandra Schema Validation cron job.
Bug ID Description
281561243 Security fix for apigee-diagnostics-collector, apigee-mint-task-scheduler, apigee-runtime, and apigee-synchronizer.
This addresses the following vulnerability:
273797045 Security fix for for apigee-diagnostics-collector apigee-synchronizer apigee-udca.
This addresses the following vulnerability:
273800345, 281572616 Security fixes for apigee-diagnostics-collector, apigee-mart-server, apigee-mint-task-scheduler, apigee-runtime, apigee-synchronizer, and apigee-udca.
This addresses the following vulnerabilities:
273801301 Security fixes for apigee-mart-server and apigee-runtime.
This addresses the following vulnerability:

June 05, 2023

v1.9.3

ANNOUNCEMENT

hybrid v1.9.3

On June 5, 2023 we released an updated version of the Apigee hybrid software, v1.9.3.

Bug ID Description
284488296 Removed an unneeded Workload Identity on the Cassandra Schema Validation cron job.
273561434 Some projects were unable to run debug sessions.
Bug ID Description
273800965 Security fix for apigee-diagnostics-collector, apigee-mart-server, apigee-mint-task-scheduler, apigee-runtime, and apigee-synchronizer.
This addresses the following vulnerability:
273800345, 281572616 Security fixes for apigee-diagnostics-collector, apigee-mart-server, apigee-mint-task-scheduler, apigee-runtime, apigee-synchronizer, and apigee-udca.
This addresses the following vulnerabilities:
273801301 Security fixes for apigee-mart-server and apigee-runtime.
This addresses the following vulnerability:
283826216 Security fixes for apigee-ingressgateway.
This addresses the following vulnerabilities:
283826785 Security fixes for istiod.
This addresses the following vulnerabilities:
281561243 Security fix for apigee-diagnostics-collector, apigee-mint-task-scheduler, apigee-runtime, and apigee-synchronizer.
This addresses the following vulnerability:

May 08, 2023

v1.8.7

hybrid v1.8.7

On May 8, 2023 we released an updated version of the Apigee hybrid software, v1.8.7.

Bug ID Description
279053612 x-forwarded-client-cert (XFCC) HTTP headers handled with the istiod.forwardClientCertDetails configuration property. See istiod.forwardClientCertDetails in the Configuration properties reference for details.
278646149 In certain circumstances, the logger.livenessProbe.timeoutSeconds configuration property was not working as expected. See logger.livenessProbe.timeoutSeconds in the Configuration property reference.
270371160 In Apigee hybrid v1.8.7, we removed certain insecure TLS ciphers. Apigee hybrid supports the TLS cipher suites supported by the Boring FIPS build of Envoy. You can now specify specific cipher suites with the virtualhosts.cipherSuites configuration property in your overrides.

Note: Apigee hybrid only supports the RSA ciphers listed. ECDSA ciphers are not supported.
Bug ID Description
279194142 Fixes build issues to achieve FIPS compliance.
277367440 Security fixes for Apigee Controller, Watcher, and apigeectl.
This addresses the following vulnerabilities:
273800965 Security fixes for apigee-diagnostics-collector, apigee-mart-server, apigee-runtime, and synchronizer.
This addresses the following vulnerabilities:
273800717 Security fixes for apigee-emulator, apigee-diagnostics-collector, apigee-mart-serve, apigee-mint-task-scheduler, apigee-mock-server, apigee-runtime, and apigee-synchronizer.
This addresses the following vulnerabilities:

May 02, 2023

v1.9.2

hybrid v1.9.2

On May 2, 2023 we released an updated version of the Apigee hybrid software, v1.9.2.

Bug ID Description
279053612 x-forwarded-client-cert (XFCC) HTTP headers handled with the istiod.forwardClientCertDetails configuration property. See istiod.forwardClientCertDetails in the Configuration properties reference for details.
278646149 In certain circumstances, the logger.livenessProbe.timeoutSeconds configuration property was not working as expected. See logger.livenessProbe.timeoutSeconds in the Configuration property reference.
270371160 In Apigee hybrid v1.9.0, we removed certain insecure TLS ciphers. Apigee hybrid supports the TLS cipher suites supported by the Boring FIPS build of Envoy. You can now specify specific cipher suites with the virtualhosts.cipherSuites configuration property in your overrides.

Note: Apigee hybrid only supports the RSA ciphers listed. ECDSA ciphers are not supported.
Bug ID Description
279194142 Fixes build issues to achieve FIPS compliance.
278313047 Security fixes for apigee-stackdriver-logging-agent.
This addresses the following vulnerabilities:
277367440 Security fixes for Apigee Controller, Watcher, and apigeectl.
This addresses the following vulnerabilities:
273800965 Security fixes for apigee-diagnostics-collector, apigee-mart-server, apigee-runtime, and synchronizer.
This addresses the following vulnerabilities:
273800717 Security fixes for apigee-emulator, apigee-diagnostics-collector, apigee-mart-server, apigee-mint-task-scheduler, apigee-mock-server, apigee-runtime, and apigee-synchronizer.
This addresses the following vulnerabilities:

April 03, 2023

v1.8.6

hybrid v1.8.6

On April 3, 2023 we released an updated version of the Apigee hybrid software, v1.8.6.

Bug ID Description
274292101 In certain circumstances, environment-scoped KVMs in hybrid could cause rollback issues for MART.
271266079 Removed port 80 from the default Kubernetes service of Apigee Ingress Gateway.
267691299 The Apigee controller uses a dedicated apigee-manager Kubernetes service account, instead of using the default SA.
267666187 When using a custom Kubernetes service for the Apigee ingress gateway, you can disable the creation of a default load balancer. See Managing Apigee ingress gateway.
266814873 In certain circumstances, retrieving encrypted KVM entries could fail with an error. This fix ensures that MART will be able to successfully function for environment-scoped KVM entries, even if the encryption key is used in the Org Env configuration or when the keys contain non-UTF8 characters. There is no change to KVM data.
263840644 Fixed a conflict with an existing ASM on the cluster.
245619397 In Apigee hybrid, fluentbit support now includes the NO_PROXY environment variable.
223320630 mTLS-related client variables are now set by the Apigee runtime.
Bug ID Description
275002360 Security fixes for fluent-bit.
This addresses the following vulnerabilities:
274112103 Security fixes to the Apigee Controller and Apigee Watcher.
This addresses the following vulnerabilities:
Bug ID Description
271266079 Removed port 80 from the default Kubernetes service of Apigee Ingress Gateway. Port 80 is not supported by Apigee ingress gateway. If you are migrating from ASM to Apigee ingress gateway, and followed the instructions in the community post to enable Port 80, it will not work with Apigee Ingress gateway.

March 27, 2023

v1.9.1

hybrid v1.9.1

On March 27, 2023 we released an updated version of the Apigee hybrid software, v1.9.1.

Bug ID Description
269738951 The example network policies are now included in the apigeectl/examples/network-policies directory. see Configuring Kubernetes network policies.
271266079 Removed port 80 from the default Kubernetes service of Apigee Ingress Gateway.
269451743 In certain circumstances, upgrading from Apigee hybrid v1.8.3 to v1.9.0 could fail with an error message when creating the virtual hosts.
268696297 Providing a Kubernetes secret for Cassandra and Redis components is now supported. See cassandra.auth.secret and redis.auth.secret in the Configuration properties reference.
267691299 The Apigee controller uses a dedicated apigee-manager Kubernetes service account, instead of using the default SA.
267666187 When using a custom Kubernetes service for the Apigee ingress gateway, you can disable the creation of a default load balancer. See Managing Apigee ingress gateway.
266989915
266919136		 In some circumstances, Apigee could return incorrect developer credentials for an app, unless the specific app was selected when requesting the credentials.
266814873 In certain circumstances, retrieving encrypted KVM entries could fail with an error. This fix ensures that MART will be able to successfully function for environment-scoped KVM entries, even if the encryption key is used in the Org Env configuration or when the keys contain non-UTF8 characters. There is no change to KVM data.
266594584 Websocket was failing in asm 1.15. This was due to incompatible capitalization in variable names between the Anthos Service Mesh overlay.yaml file and the and the Envoy filter apigee-envoyfilter.yaml file.
266411394 Added support for Azure Front Door request headers to /healthz health check.
265374889 Fixed an issue where in some circumstances the Java Callout would to fail due with the following error: Failed to execute JavaCallout. Could not initialize class org.jose4j.jwa.AlgorithmFactoryFactory2.
260342163 Fixed a narrow scenario where threads in runtime pods ended up consuming 100% CPU.
245619397 In Apigee hybrid, fluentbit support now includes the NO_PROXY environment variable.
Bug ID Description
275002360 Security fixes for fluent-bit.
This addresses the following vulnerabilities:
274112103 **Security fixes to the Apigee Controller and Apigee Watcher. This addresses the following vulnerabilities:
Bug ID Description
271266079 Removed port 80 from the default Kubernetes service of Apigee Ingress Gateway. Port 80 is not supported by Apigee ingress gateway. If you are migrating from ASM to Apigee ingress gateway, and followed the instructions in the community post to enable Port 80, it will not work with Apigee Ingress gateway.

February 24, 2023

v1.8.5

hybrid v1.8.5

On February 24, 2023 we released an updated version of the Apigee hybrid software, v1.8.5.

For information on upgrading, see Upgrading Apigee hybrid to version 1.8.

Bug ID Description
266594584 Websocket was failing in asm 1.15. This was due to incompatible capitalization in variable names between the Anthos Service Mesh overlay.yaml file and the and the Envoy filter apigee-envoyfilter.yaml file.
266411394 Add support for Azure Front Door request headers to /healthz health check.
260372012 Requests failed with 500 response and keyvaluemap.service.ErrorDuringDecryption error after upgrade to Hybrid 1.8. Note: Fixed in Apigee hybrid 1.8.4 and newer.
245619397 In Apigee hybrid, fluentbit support now includes the NO_PROXY environment variable.
181569522 You can now create a new environment with the same name as a deleted environment without needing to perform manual clean-up tasks first.

February 14, 2023

v1.7.6

hybrid v1.7.6

On February 14, 2023 we released an updated version of the Apigee hybrid software, v1.7.6.

For information on upgrading, see Upgrading Apigee hybrid to version 1.7.

Bug ID Description
268445095 The validateOrg flag can be set to false to bypass upgrade validation errors when configuration includes HTTP Forward proxy. You can use this to avoid upgrade errors caused by HTTP proxy settings.
262699558 The watcher component no longer fails when using Kubernetes Secret to store hybrid service account secret.
181569522 You can now create a new environment with the same name as a deleted environment without needing to perform manual clean-up tasks first.
218567150 The ingress gateway is now configured to consistently preserve UUID in the x-request-id header.
Note: This setting does have some impact on tracing in the ingress gateway. For more information, see pack_trace_reason in "UUID (proto)" in the envoy documentation. (Also fixed in Apigee hybrid v1.8.3)
259264961 Added support for ASM v1.15. Please see Known issue 266452840

February 01, 2023

v1.9.0

hybrid v1.9.0

On February 1, 2023 we released an updated version of the Apigee hybrid software, v1.9.0.

Kubernetes network policies

Starting in version 1.9, Apigee hybrid offers new Kubernetes network policies to secure Cassandra and Redis pods within an Apigee Hybrid cluster. See Configuring Kubernetes network policies.

CSI Backup and Restore

Starting with Apigee hybrid 1.9, you can back up and restore your hybrid data using CSI (Container Storage Interface) snapshots. CSI backup generates disk snapshots and stores them as encrypted data in cloud storage. See Cassandra CSI backup and restore.

Custom ingress access logs

Starting in version 1.9, Apigee hybrid offers custom log formats for the Apigee Ingress gateway. See Customize Ingress access logs.

Target separate ingress gateways to virtual hosts

Starting in version 1.9, Apigee hybrid you can control how separate Apigee Ingress gateways map to specific virtual hosts. See Targeting an Apigee ingress to a virtual host.

Support for customer-installed Anthos Service Mesh deprecated

Starting in version 1.9, Apigee hybrid only supports the Apigee Ingress gateway for ingress, and no longer supports customer-installed Anthos Service Mesh. See:

Bug ID Description
266356206 A jackson-databind library error was causing OAS Validation errors when loading applications. The jackson-databind libraries have been updated to correctly parse null vs NullMode values. (Fixed in Apigee hybrid v1.9.0)
262616276 Apigee hybrid v1.9 adds support for Kubernetes in AKS v1.24+. Also fixed in Apigee hybrid v1.8.4 and newer (Fixed in Apigee hybrid v1.9.0 and v1.8.4)
260372012 Requests failed with 500 response and keyvaluemap.service.ErrorDuringDecryption error after upgrade to Hybrid 1.8. Also fixed in Apigee hybrid v1.8.4 and newer (Fixed in Apigee hybrid v1.9.0 and v1.8.4)
260324159 Solved up to 30 minute delay proxy deployment due to socket closed error in synchronizer. (Fixed in Apigee hybrid v1.9.0)
259738092 Intermittent 404's were seen at the Apigee Ingress Gateway due to an inconsistent configuration delivery mechanism. (Fixed in Apigee hybrid v1.8.3)
258699204 The default memory requests and limits for metrics pods that were inadvertently changed in 1.8.x. have been fixed. Also fixed in Apigee hybrid v1.8.4 and newer (Fixed in Apigee hybrid v1.9.0 and v1.8.4)
255677576 In fresh installations with Apigee Ingress (instead of user-installed Anthos Service Mesh), the Apigee UI would sometimes show red (not working) status. This is because in certain circumstances Watcher did not send the correct deployment status due to using the wrong selector for the Apigee Ingress Gateway. (Fixed in Apigee hybrid v1.8.3)
251435916 Fixed an issue where in certain circumstances, MP pods would scale without traffic. (Fixed in Apigee hybrid v1.7.5)
249144084 Reuse existing target IPs if DNS resolution fail on DNS cache refresh. (Fixed in Apigee hybrid v1.8.1)
245664917 During the upgrade to Apigee hybrid 1.8.x, after running apigeectl init and confirming that check-ready succeeded, the Cassandra schema validation job was in an error state. Also fixed in Apigee hybrid v1.8.4 and newer (Fixed in Apigee hybrid v1.9.0 and v1.8.4)
243880171 Upgrade from Apigee hybrid v1.7 to v1.8 could fail when http_proxy was configured to DENY internal network traffic. (Fixed in Apigee hybrid v1.8.1)
243717191 Container restart no longer conflicts with the existing certificates.
243599452 Fixed indentation issue with ingress gateway annotations. (Fixed in Apigee hybrid v1.8.2)
243167389 Apigee now validates the length of ingressGateways[].name in overrides.yaml.
TThe value of ingressGateways:name must meet the following requirements:
  • Have a maximum length of 17 characters
  • Contain only lowercase alphanumeric characters, '-' or '.'
  • Start with an alphanumeric character
  • End with an alphanumeric character
See ingressGateways[].name in the Configuration property reference. (Fixed in Apigee hybrid v1.8.1)
243158304 'ApigeeRouteConfig' no longer looks for a cert in the istio-system namespace. (Fixed in Apigee hybrid v1.9.0)
241959053 Fixed apigeectl parsing error for serviceaccountRef. (Fixed in Apigee hybrid v1.8.2)
232529030 Replaced the Logging fluentbit container environment variable http_proxy with HTTP__PROXY to maintain compatibility with fluentbit 1.8. (Fixed in Apigee hybrid v1.8.1)
227212728 Cassandra scripts now avoid writing data to the Pod Filesystem. (Fixed in Apigee hybrid v1.8.1)
218567150 The ingress gateway is now configured to consistently preserve UUID in the x-request-id header. This applies to both Apigee Ingress gateway and to the Istio ingress gateway when are using customer-installed Anthos Service Mesh.
Note: This setting does have some impact on tracing in the ingress gateway. For more information, see pack_trace_reason in the "UUID (proto)" page in the envoy documentation. (Fixed in Apigee hybrid v1.8.3)
Bug ID Description
270371160 In Apigee hybrid v1.9.0, we removed certain insecure TLS ciphers. Apigee hybrid supports the TLS cipher suites supported by the Boring FIPS build of Envoy.

Note: Apigee hybrid only supports the RSA ciphers listed. ECDSA ciphers are not supported.
Bug ID Description
N/A Upgraded to ASM 1.12.9 to address Istio and Go language vulnerabilities in an earlier version (CVE-2022-39278). For more information, see the Service Mesh security bulletin. (Fixed in Apigee hybrid v1.8.2)
N/A Upgraded to ASM 1.12.9 to address Istio and Go language vulnerabilities in an earlier version (CVE-2022-39278). For more information, see the Service Mesh security bulletin. (Fixed in Apigee hybrid v1.7.5)
N/A Miscellaneous Security updates and fixes. (Fixed in Apigee hybrid v1.7.4)
262576073 Security fix for apigee-watcher. Also fixed in Apigee hybrid v1.8.4 and newer (Fixed in Apigee hybrid v1.9.0 and v1.8.4)
This addresses the following vulnerabilities:
262574571 Security fix for apigee-operators. Also fixed in Apigee hybrid v1.8.4 and newer (Fixed in Apigee hybrid v1.9.0 and v1.8.4)
This addresses the following vulnerabilities:
259290668 Update Kubectl in the backup utility. (Fixed in Apigee hybrid v1.8.3)
This addresses the following vulnerabilities:
256019598 Security fix for Cassandra. Also fixed in Apigee hybrid v1.8.4 and newer (Fixed in Apigee hybrid v1.9.0 and v1.8.4)
This addresses the following vulnerabilities:
254862745, 249630685 Security fix for apigee-diagnostics-runner and apigee-envoy. (Fixed in Apigee hybrid v1.8.3)
This addresses the following vulnerabilities:
254774193, 254773110 Fix for vulnerability in apigee-diagnostics-collector, apigee-mart-server, and apigee-mint-task-scheduler. (Fixed in Apigee hybrid v1.8.3)
This addresses the following vulnerability:
254774167 Fix for apigee-mart-server, apigee-mint-task-scheduler, apigee-runtime, and apigee-synchronizer. (Fixed in Apigee hybrid v1.8.3)
This addresses the following vulnerabilities:
254773838, 254773636, 254772551, 254771693 Fix for vulnerability in apigee-diagnostics-collector, apigee-mart-server, apigee-mint-task-scheduler, apigee-runtime, and apigee-synchronizer. (Fixed in Apigee hybrid v1.8.3)
This addresses the following vulnerabilities:
254770883, 249633275, 249629782 Security fixes for apigee-diagnostics-collector, apigee-mart-server, apigee-runtime, and apigee-synchronizer. (Fixed in Apigee hybrid v1.8.3)
This addresses the following vulnerabilities:
253693906 Upgraded Prometheus to 2.39.1 to address vulnerabilities in an earlier version. (Fixed in Apigee hybrid v1.8.2)
This addresses the following vulnerabilities:
253498057 Upgraded Fluent Bit to 1.9.9 to address vulnerabilities in an earlier version. (Fixed in Apigee hybrid v1.8.2)
This addresses the following vulnerabilities:
249635718, 249629771 Security fixes for Apigee Connect and apigee-redis. (Fixed in Apigee hybrid v1.8.3)
This addresses the following vulnerabilities:
249633289 Fix for apigee-prometheus-adapter. (Fixed in Apigee hybrid v1.8.3)
This addresses the following vulnerabilities:
248288668 Fixes to address apigee-installer vulnerabilities. (Fixed in Apigee hybrid v1.8.2)
247864229 upgraded kube-rbac-proxy to v0.13.0 to address vulnerabilities in an earlier version. (Fixed in Apigee hybrid v1.8.2)
240833499 Security fix for gopkg.in/yaml.v3. (Fixed in Apigee hybrid v1.7.4)
230369447 Security fix for commons-codec (Fixed in Apigee hybrid v1.7.4)
230368838 Security fix for CVE-2018-10237, auto-value:guava. (Fixed in Apigee hybrid v1.7.4)
230366823 Security fix for jackson-databind. (Fixed in Apigee hybrid v1.7.4)
230366589 Security fix for CVE-2021-22696-cxf in cxf. (Fixed in Apigee hybrid v1.7.4)
230366276 Security fix for CVE-2021-22569. (Fixed in Apigee hybrid v1.7.4)
229804717 Security fix for apigee-envoy. (Fixed in Apigee hybrid v1.7.4)
222772470, 220169963, 210116413 Security fix for CVE-2021-38297: Updated Go language version. (Fixed in Apigee hybrid v1.8.1)
222772341, 222772333, 222772261, 222771839 Security fix for CVE-2022-23806: Updated Go language version. (Fixed in Apigee hybrid v1.8.2)
202174499 Fixed Vulnerability for protobuf-java and protobuf-java-util packages. (Fixed in Apigee hybrid v1.8.2)

January 26, 2023

v1.8.4

hybrid v1.8.4

On January 26, 2023 we released an updated version of the Apigee hybrid software, v1.8.4. For information on upgrading, see Upgrading Apigee hybrid to version 1.8.

Bug ID Description
262699558 The watcher component failed when using Kubernetes Secret to store hybrid service account secret.
262616276 Added support for Kubernetes v1.24 and v1.25
260372012 Requests failed with 500 response and keyvaluemap.service.ErrorDuringDecryption error after upgrade to Hybrid 1.8.
258699204 The default memory requests and limits for metrics pods that were inadvertently changed in 1.8.x. have been fixed.
245664917 During the upgrade to Apigee hybrid 1.8.x, after running apigeectl init and confirming that check-ready succeeded, the Cassandra schema validation job was in an error state.
Bug ID Description
262576073 Security fix for apigee-watcher. This addresses the following vulnerabilities:
∙ CVE-2022-41716
∙ CVE-2022-41715
∙ CVE-2022-2880
∙ CVE-2022-2879
262574571 Security fix for apigee-operators. This addresses the following vulnerabilities:
∙ CVE-2022-41716

December 05, 2022

v1.8.3

hybrid v1.8.3

On December 5, 2022 we released an updated version of the Apigee hybrid software, v1.8.3.

For information on upgrading, see Upgrading Apigee hybrid to version 1.8.

Bug ID Description
259738092 Intermittent 404's were seen at the Apigee Ingress Gateway due to an inconsistent configuration delivery mechanism.
255677576 In fresh installations with Apigee Ingress (instead of user-installed Anthos Service Mesh), the Apigee UI would sometimes show red (not working) status. This is because in certain circumstances Watcher did not send the correct deployment status due to using the wrong selector for the Apigee Ingress Gateway.
218567150 The ingress gateway is now configured to consistently preserve UUID in the x-request-id header. This applies to both Apigee Ingress gateway and to the Istio ingress gateway when are using customer-installed Anthos Service Mesh.
Note: This setting does have some impact on tracing in the ingress gateway. For more information, see pack_trace_reason in the "UUID (proto)" page in the envoy documentation.
Bug ID Description
259290668 Update Kubectl in the backup utility. This addresses the following vulnerabilities:
∙ CVE-2022-42004
∙ CVE-2022-42003
∙ CVE-2022-32189
∙ CVE-2022-30635
∙ CVE-2022-30633
∙ CVE-2022-30632
∙ CVE-2022-30631
∙ CVE-2022-30630
∙ CVE-2022-30580
∙ CVE-2022-28327
∙ CVE-2022-28131
∙ CVE-2022-27664
∙ CVE-2022-25857
∙ CVE-2022-24921
∙ CVE-2022-24675
∙ CVE-2022-23773
∙ CVE-2022-23772
∙ CVE-2022-23635
∙ CVE-2022-0391
∙ CVE-2021-39156
∙ CVE-2021-39155
∙ CVE-2021-37137
∙ CVE-2021-37136
∙ CVE-2021-29482
∙ CVE-2021-20190
∙ CVE-2021-3121
∙ CVE-2020-36518
∙ CVE-2020-36189
∙ CVE-2020-36188
∙ CVE-2020-36187
∙ CVE-2020-36186
∙ CVE-2020-36185
∙ CVE-2020-36184
∙ CVE-2020-36183
∙ CVE-2020-36182
∙ CVE-2020-36181
∙ CVE-2020-36180
∙ CVE-2020-36179
∙ CVE-2020-35728
∙ CVE-2020-35491
∙ CVE-2020-35490
∙ CVE-2020-29652
∙ CVE-2020-25649
∙ CVE-2020-24750
∙ CVE-2020-24616
∙ CVE-2020-17516
∙ CVE-2020-16845
∙ CVE-2020-14195
∙ CVE-2020-14062
∙ CVE-2020-14061
∙ CVE-2020-14060
∙ CVE-2019-16869
∙ CVE-2019-9674
∙ CVE-2019-0205
∙ CVE-2018-1320
∙ CVE-2017-18640
∙ CVE-2016-5397
254862745, 249630685 Security fix for apigee-diagnostics-runner and apigee-envoy. This addresses the following vulnerabilities:
∙ CVE-2021-3999
∙ CVE-2022-2068
254774167 Fix for apigee-mart-server, apigee-mint-task-scheduler, apigee-runtime, and apigee-synchronizer. This addresses the following vulnerabilities:
∙ CVE-2022-42004
∙ CVE-2022-42003
254773838, 254773636, 254772551, 254771693 Fix for vulnerability in apigee-diagnostics-collector, apigee-mart-server, apigee-mint-task-scheduler, apigee-runtime, and apigee-synchronizer. This addresses the following vulnerabilities:
∙ CVE-2022-25857
∙ CVE-2022-0239
∙ CVE-2021-37136
∙ CVE-2021-30468
∙ CVE-2017-18640
254774193, 254773110 Fix for vulnerability in apigee-diagnostics-collector, apigee-mart-server, and apigee-mint-task-scheduler. This addresses the following vulnerability:
∙ CVE-2022-2048
254770883, 249633275, 249629782 Security fixes for apigee-diagnostics-collector, apigee-mart-server, apigee-runtime, and apigee-synchronizer. This addresses the following vulnerabilities:
∙ CVE-2022-42889
∙ CVE-2022-42004
∙ CVE-2022-42003
∙ CVE-2022-40150
∙ CVE-2022-40149
∙ CVE-2022-38752
∙ CVE-2022-38751
∙ CVE-2022-38750
∙ CVE-2022-38749
∙ CVE-2022-25857
∙ CVE-2022-3171
∙ CVE-2022-2097
∙ CVE-2022-2068
∙ CVE-2022-2048
∙ CVE-2022-2047
∙ CVE-2022-1292
∙ CVE-2021-31684
∙ CVE-2021-29425
∙ CVE-2021-27568
∙ CVE-2020-13936
∙ CVE-2020-8908
∙ CVE-2019-18276
∙ CVE-2018-10237
∙ CVE-2017-18640
249635718, 249629771 Security fixes for Apigee Connect and apigee-redis. This addresses the following vulnerabilities:
∙ CVE-2022-32189
∙ CVE-2022-30635
∙ CVE-2022-30633
∙ CVE-2022-30632
∙ CVE-2022-30631
∙ CVE-2022-30630
∙ CVE-2022-30580
∙ CVE-2022-28327
∙ CVE-2022-28131
∙ CVE-2022-27664
∙ CVE-2022-24921
∙ CVE-2022-24675
∙ CVE-2022-23773
∙ CVE-2022-23772
249633289 Fix for apigee-prometheus-adapter. This addresses the following vulnerabilities:
∙ CVE-2022-21698
∙ CVE-2022-1996

November 02, 2022

v1.7.5

hybrid v1.7.5

On November 2, 2022 we released an updated version of the Apigee hybrid software, v1.7.5.

For information on upgrading, see Upgrading Apigee hybrid to version 1.7.

Bug ID Description
251435916 Fixed an issue where in certain circumstances, MP pods would scale without traffic.
241959053 Fixed apigeectl parsing error for serviceaccountRef.
Bug ID Description
253693906 Upgraded Prometheus to 2.39.1 to address vulnerabilities in an earlier version. This change addresses the following vulnerabilities:
∙ CVE-2022-24675
∙ CVE-2022-27664
∙ CVE-2022-28131
∙ CVE-2022-28327
∙ CVE-2022-30580
∙ CVE-2022-30630
∙ CVE-2022-30631
∙ CVE-2022-30632
∙ CVE-2022-30633
∙ CVE-2022-30635
∙ CVE-2022-32189
253498057 Upgraded Fluent Bit to 1.9.9 to address vulnerabilities in an earlier version. This change addresses the following vulnerabilities:
∙ CVE-2022-1292
∙ CVE-2022-2068
∙ CVE-2021-3999
∙ CVE-2022-23218
∙ CVE-2022-23219
∙ CVE-2022-25013
∙ CVE-2021-33574
∙ CVE-2018-12886
∙ CVE-2022-0778
248288668 Fixes to address apigee-installer vulnerabilities.
247864229 upgraded kube-rbac-proxy to v0.13.0 to address vulnerabilities in an earlier version.
N/A Upgraded to ASM 1.12.9 to address Istio and Go language vulnerabilities in an earlier version (CVE-2022-39278). For more information, see the Service Mesh security bulletin.

October 27, 2022

v1.8.2

hybrid v1.8.2

On October 27, 2022 we released an updated version of the Apigee hybrid software, v1.8.2.

For information on upgrading, see Upgrading Apigee hybrid to version 1.8.

Bug ID Description
253693906 Upgraded Prometheus to 2.39.1 to address vulnerabilities in an earlier version. This change addresses the following vulnerabilities:
∙ CVE-2022-24675
∙ CVE-2022-27664
∙ CVE-2022-28131
∙ CVE-2022-28327
∙ CVE-2022-30580
∙ CVE-2022-30630
∙ CVE-2022-30631
∙ CVE-2022-30632
∙ CVE-2022-30633
∙ CVE-2022-30635
∙ CVE-2022-32189
253498057 Upgraded Fluent Bit to 1.9.9 to address vulnerabilities in an earlier version. This change addresses the following vulnerabilities:
∙ CVE-2022-1292
∙ CVE-2022-2068
∙ CVE-2021-3999
∙ CVE-2022-23218
∙ CVE-2022-23219
∙ CVE-2022-25013
∙ CVE-2021-33574
∙ CVE-2018-12886
∙ CVE-2022-0778
248288668 Fixes to address apigee-installer vulnerabilities.
247864229 upgraded kube-rbac-proxy to v0.13.0 to address vulnerabilities in an earlier version.
N/A Upgraded to ASM 1.12.9 to address Istio and Go language vulnerabilities in an earlier version (CVE-2022-39278). For more information, see the Service Mesh security bulletin.

October 01, 2022

preview

hybrid new installation experience public preview

On September 30, 2022 we released the public preview of the new Apigee hybrid installation experience.

Overview

The new Apigee hybrid installation experience provides a Kubernetes way of installing Apigee components using kubectl. The enhanced validations and visibility of the components being installed provides better debuggability and improves the overall install process.

An install script, apigee-hybrid-setup.sh, provides an easy tool for basic installation. You can use that to create your hybrid installation and then modify it to fit your needs with kubectl, or you can create your hybrid installation from scratch using kubectl. All Apigee hybrid configuration properties are stored in yaml files, one for each major component. This allows much more granular control of your hybrid installation on your Kubernetes environment.

Preview release

The new Apigee hybrid installation experience and its documentation are part of a preview release. Apigee does not warranty the software or the procedures in this preview. The software and documentation are subject to change without notice.

September 29, 2022

v1.8.1

hybrid v1.8.1

On September 29, 2022 we released an updated version of the Apigee hybrid software, v1.8.1.

For information on upgrading, see Upgrading Apigee hybrid to version 1.8.

Bug ID Description
249144084 Reuse existing target IPs if DNS resolution fail on DNS cache refresh.
243880171 Upgrade from Apigee hybrid v1.7 to v1.8 could fail when http_proxy was configured to DENY internal network traffic.
243717191 Container restart no longer conflicts with the existing certificates.
243599452 Fixed indentation issue with ingress gateway annotations.
243167389 Apigee now validates the length of ingressGateways[].name in overrides.yaml.
TThe value of ingressGateways:name must meet the following requirements:
  • Have a maximum length of 17 characters
  • Contain only lowercase alphanumeric characters, '-' or '.'
  • Start with an alphanumeric character
  • End with an alphanumeric character
See ingressGateways[].name in the Configuration property reference.
241959053 Fixed apigeectl parsing error for serviceaccountRef.
232529030 Replaced the Logging fluentbit container environment variable http_proxy with HTTP__PROXY to maintain compatibility with fluentbit 1.8.
227212728 Cassandra scripts now avoid writing data to the Pod Filesystem.
Bug ID Description
222772470, 220169963, 210116413 Security fix for CVE-2021-38297: Updated Go language version.
222772341, 222772333, 222772261, 222771839 Security fix for CVE-2022-23806: Updated Go language version.
202174499 Fixed Vulnerability for protobuf-java and protobuf-java-util packages.

September 02, 2022

v1.7.4

hybrid v1.7.4

On September 2, 2022 we released an updated version of the Apigee hybrid software, v1.7.4.

For information on upgrading, see Upgrading Apigee hybrid to version 1.7.

Bug ID Description
232529030 Replaced the Logging fluentbit container environment variable http_proxy with HTTP_PROXY to maintain compatibility with fluentbit 1.8.
Bug ID Description
240833499 Security fix for gopkg.in/yaml.v3.
230369447 Security fix for commons-codec
230368838 Security fix for CVE-2018-10237, auto-value:guava.
230366823 Security fix for jackson-databind.
230366589 Security fix for CVE-2021-22696-cxf in cxf.
230366276 Security fix for CVE-2021-22569.
229804717 Security fix for apigee-envoy.
N/A Miscellaneous Security updates and fixes.

August 22, 2022

v1.8.0

hybrid v1.8.0

On August 22, 2022 we released an updated version of the Apigee hybrid software, v1.8.0.

For information on upgrading, see Upgrading Apigee hybrid to version 1.8.

Apigee Ingress gateway

Starting in version 1.8, Apigee hybrid offers a new feature to manage the ingress gateway for your hybrid installation, Apigee ingress gateway. Anthos Service Mesh is no longer a prerequisite for hybrid installation. With Apigee ingress gateway, Apigee will stop supplying routing configuration to Anthos Service Mesh. See Managing Apigee ingress.

ORG-level UDCA

Apigee hybrid now supports setting UDCA at the org level instead of at the environment level. See orgScopedUDCA in the Configuration property reference.

Support for newer versions of Anthos, Anthos Service Mesh, and Kubernetes

Starting in version 1.8, Apigee hybrid supports Anthos version 1.12, Anthos Service Mesh version 1.13, and Kubernetes version 1.23 on specific platforms. See Apigee hybrid supported platforms and versions for details.

KVM pagination

The Key Value Map (KVM) CRUD APIs with pagination support are now available in Hybrid (introduced in Apigee X on March 10, 2022). See REST Resource: organizations.keyvaluemaps and REST Resource: v1.organizations.environments.keyvaluemaps.

apigeectl now supports the --v option to set the log verbosity level

Starting in version 1.8, apigeectl includes a --v option to set log verbosity levels in the format --v=int, for example apigeectl apply --v=5. This option replaces the --verbose option (now deprecated). This is the same as the kubectl --v option. See apigeectl for details.

tools/apigee-pull-push.sh includes a –list option to list all images

Starting in version 1.8, The tools/apigee-pull-push.sh utility has a --list or -l option that will list all images in the gcr repo. See apigee-pull-push.sh for details.

Bug ID Description
239854141 apigee-pull-push.sh now uses gcr source repositories. (Fixed in Apigee hybrid v1.8.0)
238370197 Fixed an issue where the timeTaken variable's value could sometimes be calculated incorrectly. (Fixed in Apigee hybrid v1.7.3)
236399482 Added support for ASM v1.13. (Fixed in Apigee hybrid v1.7.2)
236129944 Fixed the controller crashloopbackoff due to null pointer issue. (Fixed in Apigee hybrid v1.6.9)
236129944 Fixed the controller crashloopbackoff resulting from null pointer. (Fixed in Apigee hybrid v1.7.2)
234620567 Fix logger issue in Anthos BareMetal with CentOS. (Fixed in Apigee hybrid v1.6.9)
234355351 Fixed issue with message processor pods restarting frequently. Added backoff polling task for Cloud KMS key listener. The listener is paused only when the flush policy is met. (Fixed in Apigee X, June 21, 2022)
233349518 Fixed "Invalid Resource" error generated for job/apigee-resources-install. (Fixed in Apigee hybrid v1.7.1)
233094108 Fixed Stacktrace truncation in runtime containers to support proxy diagnosis. (Fixed in Apigee hybrid v1.7.2)
232977937 Fixed an issue where deployment would become stuck on "Applying routing changes on" for multiple ingress gateways. (Fixed in Apigee hybrid v1.7.1)
231313050 Fixed issue causing Apigee logger pod to remain in crashloopbackoff state. (Fixed in Apigee hybrid v1.7.2)
229824389 Fixed an issue in hybrid 1.7.0 where the output apigeectl init could be generated in the wrong order. (Fixed in Apigee hybrid v1.7.1)
229804717 Fixed upgrade envoy to use distroless v1.22.0. (Fixed in Apigee hybrid v1.7.1)
229639530 Fixed an error harmonizing the container process ID to use Apigee ID for Hybrid on OpenShift. (Fixed in Apigee hybrid v1.7.1)
228855520 Upgraded support for ASM version 1.13. (Fixed in Apigee X, May 9, 2022)
227600373 Fixed an installation issue with Cassandra. (Fixed in Apigee hybrid v1.6.7)
227538469 Fixed an issue where configuration actions would write logs to the pod file system. (Fixed in Apigee hybrid v1.7.1)
226964206 MART, runtime and synchronizer would write to the pod file system. (Fixed in Apigee hybrid v1.7.1)
226464960 Apigee hybrid fresh installations on OpenShift 4.6 and 4.8 would fail. (Fixed in Apigee hybrid v1.6.7)
225939342 Fixed an error where deployment status would show as "Applying routing changes on {env}". (Fixed in Apigee hybrid v1.7.1)
225198475 Fixed an issue where resource reference changes could not be detected. (Fixed in Apigee hybrid v1.7.1)
225169066 Cassandra database backup and restore was not working when http_proxy is enabled under certain circumstances. (Fixed in Apigee hybrid v1.5.10)
225081332 Fixed allow privileged pods issue. (Fixed in Apigee hybrid v1.7.1)
224620542 On some Kubernetes platforms, logging would fail without adding an empty directory for the logs. (Fixed in Apigee hybrid v1.6.7)
223081301 Fixed organization-level UDCA incorrect http-proxy secret name. (Fixed in Apigee hybrid v1.6.7)
222649295 Organization-level UDCA would hang. (Fixed in Apigee hybrid v1.6.7)
221885751 Hybrid logging functionality has been reworked. This should resolve issues with excessive log volume generation, frequent logger restarts, and ensure correct logger functionality with both docker and containerd runtimes. (Fixed in Apigee hybrid v1.5.8)
221292104 Fix to address failure to capture requests in Debug sessions involving PostClientFlow ServiceCallouts. (Fixed in Apigee X, May 9, 2022)
221266789 Hybrid logging functionality has been reworked. This should resolve issues with excessive log volume generation, frequent logger restarts, and ensure correct logger functionality with both docker and containerd runtimes. (Fixed in Apigee hybrid v1.6.7)
219622478 Fixed the CPS property token so that when set to true, it will allow the instance to shutdown and reboot when cassandra connection failures occur. (Fixed in Apigee hybrid v1.7.3)
216018530 Fixed an issue where the apigee-logger-apigee-telemetry DaemonSet could still be left running after turning off logger. (Fixed in Apigee hybrid v1.7.1)
213261445 Fixed reliance on keystore generated by cert manager for metrics endpoint and removed the need for a custom generate_cert script. (Fixed in Apigee hybrid v1.6.7)
211716827 Fixed an issue where a non-default gateway could cause routing errors in certain circumstances. (Fixed in Apigee hybrid v1.7.1)
205616792 Fixed core dump on running user schema setup. (Fixed in Apigee hybrid v1.7.1)
202950533 Remove the server header from all responses. This will be consistent with CG SaaS and OPDK. (Fixed in Apigee hybrid v1.8.0)
202403896 Upgrade to Apigee hybrid v1.6 from v1.5 could fail due to annotation size. (Fixed in Apigee hybrid v1.5.8)
Bug ID Description
230369447 Security fix for commons-codec (Fixed in Apigee hybrid v1.8)
230368838 Security fix for CVE-2018-10237, auto-value:guava (Fixed in Apigee hybrid v1.8)
230366823 Security fix for jackson-databind (Fixed in Apigee hybrid v1.8)
230366589 Security fix for CVE-2021-22696-cxf in cxf (Fixed in Apigee hybrid v1.8)
229804717 Security fix for CVE-2021-3711 (Fixed in Apigee hybrid v1.8, v1.7, v1.6)
229804942 Security fix for CVE-2022-0778 (Fixed in Apigee hybrid v1.8)
N/A Miscellaneous Security updates and fixes.

The apigeectl --verbose option has been deprecated. It is replaced with the --v option which lets you specify the log verbosity level in the format --v=int, for example apigeectl apply --v=5. See apigeectl for details.

July 29, 2022

v1.7.3

hybrid v1.7.3

On July 29, 2022 we released an updated version of the Apigee hybrid software, v1.7.3.

For information on upgrading, see Upgrading Apigee hybrid to version 1.7.

Bug ID Description
219622478 Fixed the CPS property token so that when set to true, it will allow the instance to shutdown and reboot when cassandra connection failures occur.
238370197 Fixed an issue where the timeTaken variable's value could sometimes be calculated incorrectly.
Bug ID Description
N/A Security fix for CVE-2022-22963.

July 08, 2022

v1.6.9

hybrid v1.6.9

On July 8, 2022 we released an updated version of the Apigee hybrid software, v1.6.9.

For information on upgrading, see Upgrading Apigee hybrid to version 1.6.

Bug ID Description
236129944 Fixed the controller crashloopbackoff due to null pointer issue.
234620567 Fix logger issue in Anthos BareMetal with CentOS.
231313050 Fixed issue causing Apigee logger pod to remain in crashloopbackoff state.
233094108 Fixed Stacktrace truncation in runtime containers to support proxy diagnosis.

June 24, 2022

v1.7.2

hybrid v1.7.2

On June 24, 2022 we released an updated version of the Apigee hybrid software, v1.7.2.

For information on upgrading, see Upgrading Apigee hybrid to version 1.7.

Bug ID Description
233094108 Fixed Stacktrace truncation in runtime containers to support proxy diagnosis.
236129944 Fixed the controller crashloopbackoff resulting from null pointer.
231313050 Fixed issue causing Apigee logger pod to remain in crashloopbackoff state.
236399482 Added support for ASM v1.13.

June 02, 2022

v1.7.1

hybrid v1.7.1

On June 2, 2022 we released an updated version of the Apigee hybrid software, v1.7.1.

For information on upgrading, see Upgrading Apigee hybrid to version 1.7.

Bug ID Description
233349518 Fixed "Invalid Resource" error generated for job/apigee-resources-install.
232977937 Fixed an issue where deployment would become stuck on "Applying routing changes on" for multiple ingress gateways.
216018530 Fixed an issue where the apigee-logger-apigee-telemetry DaemonSet could still be left running after turning off logger.
226964206 MART, runtime and synchronizer would write to the pod file system.
211716827 Fixed an issue where a non-default gateway could cause routing errors in certain circumstances.
225198475 Fixed an issue where resource reference changes could not be detected.
225939342 Fixed an error where deployment status would show as "Applying routing changes on {env}".
229824389 Fixed an issue in hybrid 1.7.0 where the output apigeectl init could be generated in the wrong order.
229639530 Fixed an error harmonizing the container process ID to use Apigee ID for Hybrid on OpenShift
229804717 Fixed upgrade envoy to use distroless v1.22.0.
227538469 Fixed an issue where configuration actions would write logs to the pod file system.
205616792 Fixed core dump on running user schema setup.
225081332 Fixed allow privileged pods issue.

May 31, 2022

v1.6.8

hybrid v1.6.8

On May 31, 2022 we released an updated version of the Apigee hybrid software, v1.6.8.

For information on upgrading, see Upgrading Apigee hybrid to version 1.6.

Bug ID Description
233349518 Fixed "Invalid Resource" error generated for job/apigee-resources-install.
225939342 Fixed an error where deployment status would show as "Applying routing changes on {env}".
225198475 Fixed an issue where resource reference changes could not be detected.
232977937 Fixed an issue where deployment would become stuck on "Applying routing changes on" for multiple ingress gateways.
229804717 Fixed upgrade envoy to use distroless v1.22.0.

May 11, 2022

v1.6.7

hybrid v1.6.7

On May 11, 2022 we released an updated version of the Apigee hybrid v1.6.7 software.

For information on upgrading, see Upgrading Apigee hybrid to version 1.6.

Bug ID Description
227600373 Fixed an installation issue with Cassandra.
227538469 Configuration actions would write logs to the pod file system.
226964206 MART, runtime and synchronizer would write to the pod file system.
226464960 Apigee hybrid fresh installations on OpenShift 4.6 and 4.8 would fail.
225081332 Allow privileged pods issue.
224620542 On some Kubernetes platforms, logging would fail without adding an empty directory for the logs.
223081301 Fixed organization-level UDCA incorrect http-proxy secret name.
222649295 Organization-level UDCA would hang.
221266789 Hybrid logging functionality has been reworked. This should resolve issues with excessive log volume generation, frequent logger restarts, and ensure correct logger functionality with both docker and containerd runtimes.
213261445 Fixed reliance on keystore generated by cert manager for metrics endpoint and removed the need for a custom generate_cert script.
205616792 Fixed core dump on running user schema setup.

May 10, 2022

v1.7.0 & v1.5.10 & v1.6.7

April 22, 2022

v1.5.10

hybrid v1.5.10

On April 22, 2022 we released an updated version of the Apigee hybrid v1.5.10 software.

For information on upgrading, see Upgrading Apigee hybrid to version 1.5.

Bug ID Description
225169066 Cassandra database backup and restore was not working when http_proxy is enabled under certain circumstances.
221885751 Hybrid logging functionality has been reworked. This should resolve issues with excessive log volume generation, frequent logger restarts, and ensure correct logger functionality with both docker and containerd runtimes.
202403896 Upgrade to Apigee hybrid v1.6 from v1.5 could fail due to annotation size.
221885751 Multiple issues with the Hybrid logging functionality have been addressed.

April 18, 2022

v1.7.0

Apigee hybrid v1.7.0

On April 18, 2022 we released Apigee hybrid v1.7.0.

For information on upgrading, see Upgrading Apigee hybrid to version 1.7.

mTLS communication between Cassandra clients and Cassandra nodes

Apigee hybrid now supports mTLS communication between Cassandra clients (MART, Sync, and MP) and Cassandra nodes. For related ports used, see feedbackSecure ports usage. (Implemented in Apigee hybrid v1.7.0)

Custom metrics scaling

Apigee hybrid v1.7.0 now supports custom metrics scaling using the metrics:appStackdriverExporter and metrics:proxyStackdriverExporter configuration properties. See metrics in the Configuration properties reference. (Implemented in Apigee hybrid v1.7.0)

OAuth JWT access tokens

Apigee hybrid v1.7.0 now supports JWT operations that allow the OAuthV2 policy to generate, verify, and refresh access tokens that conform to the JWT token standard. See Using JWT OAuth tokens. (Implemented in Apigee hybrid v1.7.0)

Cloud Logging

Apigee hybrid v1.7.0 now supports the <CloudLogging> element in the MessageLogging policy that lets you log messages to Cloud Logging. (Implemented in Apigee hybrid v1.7.0)

PublishMessage policy

Apigee hybrid v1.7.0 now supports the PublishMessage policy that lets you publish your API proxy flow information to a Google Cloud Pub/Sub topic.

GraphQL policy now supports JSON-encoded payloads. (Implemented in Apigee X, March 15, 2022)

Bug ID Description
224577096 Support Added for Anthos Service Mesh 1.12 (Fixed in Apigee hybrid v1.6.6)
219523719 Fix to address CPU and memory consumption when debug-session is enabled with response-status as the filtering criteria. (Fixed in Apigee X, March 15, 2022)
217386412 Change the property set logging level to fine when property is not found. (Fixed in Apigee hybrid v1.6.5)
215773113 Setting the securityPolicy appeared to have no effect for specific configurations. (Fixed in Apigee hybrid v1.6.5)
214960081 HTTPS endpoints could be called as HTTP resulting in an "EOF unexpected" error. (Fixed in Apigee hybrid v1.6.4)
211787541 Errors displayed in synchronizer logs for stale contracts. (Fixed in Apigee hybrid v1.6.5)
210590135 Invalid protocol in proxy.url flow variable in Apigee X. The proxy.url flow variable could show as http even when the request is https. (Fixed in Apigee hybrid v1.6.4)
210314786 The backup utility did not work with workload identity. (Fixed in Apigee hybrid v1.6.4)
209622008 Dynamic updates to rate in spike arrest are now reflected immediately. (Fixed in Apigee X, March 15, 2022)
209484701 Invalid client IP sent to analytics. (Fixed in Apigee hybrid v1.6.5)
209097822 Fixed and issue where SpikeArrest was not reflecting updated rate (Fixed in Apigee hybrid v1.6.3)
208474799 Apigee hybrid now supports ASM version 1.12. See Supported platforms for ASM version support for each supported version of Apigee Hybrid. (Fixed in Apigee hybrid v1.7.0)
208322185 Apigee hybrid Cassandra backup and restore can now use either a user-provided custom secret or a generated secret (Fixed in Apigee hybrid v1.6.3)
207762842 Hybrid logging functionality has been reworked. This should resolve issues with excessive log volume generation, frequent logger restarts, and ensure correct logger functionality with both docker and containerd runtimes.(Fixed in Apigee hybrid v1.7.0)
207618262 Fixed an issue where SpikeArrest opened too many connections to redis-envoy. (Fixed in Apigee hybrid v1.6.3)
207400645 Allow direct reads from API server to API client when enabled. (Fixed in Apigee hybrid v1.6.3)
205820658 Fixed an issue where Apigee X/hybrid Debug could show the authorization header. (Fixed in Apigee hybrid v1.6.3)
205810988 Resolve suspension dialog displayed "user not authorized" message for regions other than US This has been fixed. Location information is now added into the suspension URL from Apigee. (Fixed in Apigee X, December 02,2021)
205732137 Handle Quota correctly when the Operation group is set with empty string params (Fixed in Apigee hybrid v1.6.3)
205148816 Product Level Quota Info now available in Proxy. (Fixed in Apigee hybrid v1.6.3)
204943895 Quota Policy in Shared Flow now working properly. (Fixed in Apigee hybrid v1.6.3)
204943880 Fixed issue where SpikeArrest in Shared Flow did not have context of API Proxy. (Fixed in Apigee hybrid v1.6.3)
204905727 GenerateResponse was hanging on response flow when enabled=true. (Fixed in Apigee hybrid v1.6.5)
204368970 TLS variables are now set by Apigee Runtime. (Fixed in Apigee hybrid v1.6.3)
204146857 Fixed an issue where new environments were not created in Apigee hybrid deployment. (Fixed in Apigee hybrid v1.6.3)
203785814 A transient error could occur when calling conversion webhook for Apigee Telemetry. The error would occur when Apigee CRD is installed too early in the sequence. The installer job now checks for the correct sequence. (Fixed in Apigee hybrid v1.6.4)
203468593 Corrected the storageclass property name. (Fixed in Apigee hybrid v1.5.5)
203462573 The StorageClass set in overrides was not honored. (Fixed in Apigee hybrid v1.6.1)
202560276 AKS - containerd broke apigee-logger. (Fixed in Apigee hybrid v1.6.1)
202309278 Monetization: Eliminated a race condition that could make a prepaid developer's balance appear incorrect. (Fixed in Apigee hybrid v1.6.2)
202299966 Added new remote-address-related headers and modified the headers to be RFC compliant. (Fixed in Apigee hybrid v1.6.3)
200918549 There was an issue when using forward proxy with the ApigeeConnect agent. (Fixed in Apigee hybrid v1.6.1)
200700375 Fixed API products sorting issue in UI. Previously, sorting was disabled on the API products page. Sorting is now enabled. (Fixed in Apigee X, September 23, 2021)
200648523 Trace Variable, is_request_blocked, was showing incorrect information. (Fixed in Apigee hybrid v1.6.2)
199952038 The apigeectl command uses the new --restore flag to restore Cassandra to a previously saved snapshot. For more information, see Restoring in a single region. (Fixed in Apigee hybrid v1.6.6)
199807323 Updating Developer would reset the Developer billing type attribute (Fixed in Apigee hybrid v1.6.2)
199541025 Transaction ID is now required to be unique when used with prepaid developer balance credit API. (Fixed in Apigee hybrid v1.6.2)
198549304, 197730687, 196937143, 188370635, 187890034 Error state for conflicting dates is now correct. (Fixed in Apigee hybrid v1.6.2)
198036824 The securityContext was empty when it should have been populated. (Fixed in Apigee hybrid v1.6.1)
197945951 Stale DNS record in MP memory could cause an outage for a proxy. (Fixed in Apigee hybrid v1.6.4)
197910247 SetDialogflowResponse Policy - JSONPath expressions were not working. (Fixed in Apigee hybrid v1.6.1)
197711066 Cluster upgrade failed due to PDB (PodDisruptionBudget) policy not being met. (Fixed in Apigee hybrid v1.6.1)
196095557 Fixed proxy high response times. (Fixed in Apigee hybrid v1.5.4)
196024622 Hybrid images contained keys. (Fixed in Apigee hybrid v1.6.1)
196024483 Hybrid images did not set USER instruction when building the container. (Fixed in Apigee hybrid v1.6.1)
193799009 Fixed wrong status code shown on trace with ServiceCallout in PostClientflow. (Fixed in Apigee hybrid v1.5.4)
193520269 Fixed Apigee UI not showing the trace UI. (Fixed in Apigee hybrid v1.5.4)
193041253 Cassandra upgraded to v3.11.9 The Cassandra database in Apigee hybrid v1.7.0 has been upgraded to version 3.11.9. (Fixed in Apigee hybrid v1.7.0)
192987085 Fixed the ApiProductNotFound exception, which occurred when you deleted an API product but the deletion of associated rate plans was pending. (Fixed in Apigee X Monetization, November 3, 2021)
191853747 Apigee Workload Identities not working for specific configurations. (Fixed in Apigee hybrid v1.6.5)
190679584 There was an Incorrect error message on deploying AssertCondition policy with invalid condition. (Fixed in Apigee hybrid v1.6.1)
189341334 Fixed an issue to eliminate the potential for connection leaks for the watcher component. (Fixed in Apigee hybrid v1.6.3)
188407113 Invalid value in the ConsumptionPricingType during rate plan creation displayed the 500 status code. Now the status code for an invalid value is 4xx. (Fixed in Apigee X Monetization, November 3, 2021)
181259284 Fixed unresolved flow variables system.region.name and system.pod.name. (Fixed in Apigee hybrid v1.5.4)
180672249 FlowCallout succeeded, although SharedFlow had errors in deployment. (Fixed in Apigee hybrid v1.6.1)
173738907 Fixed support resource request/limit in override.yaml in apigee-metrics. (Fixed in Apigee hybrid v1.5.4)
173566787 Reuse existing target IPs if DNS resolution fail on DNS cache refresh. (Fixed in Apigee hybrid v1.6.5)
111777025 LookupCache: cachehit was shown false in trace when the actual value was true. (Fixed in Apigee hybrid v1.6.5)N/A
N/A If there is more than one SpikeArrest policy in a bundle, 502 errors will occur. (Fixed in Apigee X, December 12, 2021)
N/A If ServiceCallout is "fire and forget" (no tag), a race condition can occur if there is another policy that occurs after it. (Fixed in Apigee X, December 12, 2021)
N/A Fix bug delete dialog does not open Previously, on click of delete dialog, the dialog was not appearing, this fixes it (Fixed in Apigee X, October 05, 2021)
N/A Dynamic updates to rate in SpikeArrest may not reflect immediately. (Fixed in Apigee X, December 12, 2021)
Bug ID Description
217743790 ** Cassandra backup would run as privileged.**
204994504 Container Vulnerability fixed: CVE-2018-12934. (Fixed in Apigee hybrid v1.6.5)
N/A Multiple security fixes including CVE-2019-5021. (Fixed in Apigee hybrid v1.6.5)
N/A Miscellaneous Security updates and fixes. (Fixed in Apigee X, December 12, 2021)
205820658 A security issue was addressed. (Fixed in Apigee hybrid v1.6.2-hotfix.1)

"Apigee Deployer" role deprecated and replaced by "Apigee Environment Admin" The environment role "Apigee Deployer" has been deprecated, and replaced by "Apigee Environment Admin". (Implemented in Apigee X, December 2, 2022)

March 24, 2022

v1.6.6

hybrid v1.6.6

On March 24, 2022 we released an updated version of the Apigee hybrid v1.6.6 software.

For information on upgrading, see Upgrading Apigee hybrid to version 1.6.

Bug ID Description
224577096 Support Added for Anthos Service Mesh 1.12
204368970 Fixed a bug in TLS variables population
199952038 The apigeectl command uses the new --restore flag to restore Cassandra to a previously saved snapshot. For more information, see Restoring in a single region.
v1.5.9

hybrid v1.5.9

On March 24, 2022 we released an updated version of the Apigee hybrid v1.5.9 software.

For information on upgrading, see Upgrading Apigee hybrid to version 1.5.

Bug ID Description
224577096 Support Added for Anthos Service Mesh 1.12
204368970 Fixed the bug in TLS variables population

March 01, 2022

v1.5.8

hybrid v1.5.8

On March 1, 2022 we released an updated version of the Apigee hybrid v1.5.8 software.

For information on upgrading, see Upgrading Apigee hybrid to version 1.5.

Bug ID Description
219523719 Fix to address the CPU and memory consumption when debug-session is enabled with response-status as the filtering criteria.
217386412 Change the property set logging level to fine when property is not found.
215773113 Setting the securityPolicy appeared to have no effect for specific configurations.
209484701 Invalid client IP sent to analytics.
189233354 Distributed tracing with Jaeger would error out.
Bug ID Description
N/A Multiple security fixes including CVE-2019-5021.

February 28, 2022

v1.6.5

hybrid v1.6.5

On February 28, 2022 we released an updated version of the Apigee hybrid v1.6.5 software.

For information on upgrading, see Upgrading Apigee hybrid to version 1.6.

Bug ID Description
217386412 Change the property set logging level to fine when property is not found.
215773113 Setting the securityPolicy appeared to have no effect for specific configurations.
211787541 Errors displayed in synchronizer logs for stale contracts.
209484701 Invalid client IP sent to analytics.
204905727 GenerateResponse was hanging on response flow when enabled=true.
191853747 Apigee Workload Identities not working for specific configurations.
173566787 Reuse existing target IPs if DNS resolution fail on DNS cache refresh.
111777025 LookupCache: cachehit was shown false in trace when the actual value was true.
Bug ID Description
217743790 ** Cassandra backup would run as privileged.**
204994504 Container Vulnerability fixed: CVE-2018-12934.
N/A Multiple security fixes including CVE-2019-5021.

January 31, 2022

v1.6.4

hybrid v1.6.4

On January 31, 2022 we released an updated version of the Apigee hybrid v1.6.4 software

For information on upgrading, see Upgrading Apigee hybrid to version 1.6.

Bug ID Description
214960081 HTTPS endpoints could be called as HTTP resulting in an "EOF unexpected" error.
210590135 Invalid protocol in proxy.url flow variable in Apigee X. The proxy.url flow variable could show as http even when the request is https.
203785814 A transient error could occur when calling conversion webhook for Apigee Telemetry. The error would occur when Apigee CRD is installed too early in the sequence. The installer job now checks for the correct sequence.
197945951 Stale DNS record in MP memory could cause an outage for a proxy.
210314786 The backup utility did not work with workload identity.

January 28, 2022

v1.5.7

hybrid v1.5.7

On January 28, 2022 we released an updated version of the Apigee hybrid v1.5.7 software

For information on upgrading, see Upgrading Apigee hybrid to version 1.5.

Bug ID Description
214960081 HTTPS endpoints could be called as HTTP resulting in an EOF unexpected error.
204368970 TLS variables were not being set by Apigee Runtime.
189341334 Fixed potential connection leaks for watcher component.