Apigee hybrid release notes

This page documents production updates to Apigee hybrid in 2022 and later. We recommend that users periodically check this list for any new announcements.

See 2021 and prior for older release notes.

See also: Known issues.

You can see the latest product updates for all of Google Cloud on the Google Cloud page, browse and filter all release notes in the Google Cloud console, or you can programmatically access release notes in BigQuery.

To get the latest product updates delivered to you, add the URL of this page to your feed reader, or add the feed URL directly: https://cloud.google.com/feeds/apigee-hybrid-release-notes.xml

June 24, 2022

v1.7.2

On June 24, 2022 we released an updated version of the Apigee hybrid software, v1.7.2.

For information on upgrading, see Upgrading Apigee hybrid to version 1.7.

Bug ID Description
233094108 Fixed Stacktrace truncation in runtime containers to support proxy diagnosis.
236129944 Fixed the controller crashloopbackoff resulting from null pointer.
231313050 Fixed issue causing Apigee logger pod to remain in crashloopbackoff state.
236399482 Added support for ASM v1.13.

June 02, 2022

v1.7.1

On June 2, 2022 we released an updated version of the Apigee hybrid software, v1.7.1.

For information on upgrading, see Upgrading Apigee hybrid to version 1.7.

Bug ID Description
233349518 Fixed "Invalid Resource" error generated for job/apigee-resources-install.
232977937 Fixed an issue where deployment would become stuck on "Applying routing changes on" for multiple ingress gateways.
216018530 Fixed an issue where the apigee-logger-apigee-telemetry DaemonSet could still be left running after turning off logger.
226964206 MART, runtime and synchronizer would write to the pod file system.
211716827 Fixed an issue where a non-default gateway could cause routing errors in certain circumstances.
225198475 Fixed an issue where resource reference changes could not be detected.
225939342 Fixed an error where deployment status would show as "Applying routing changes on {env}".
229824389 Fixed an issue in hybrid 1.7.0 where the output apigeectl init could be generated in the wrong order.
229639530 Fixed an error harmonizing the container process ID to use Apigee ID for Hybrid on OpenShift
229804717 Fixed upgrade envoy to use distroless v1.22.0.
227538469 Fixed an issue where configuration actions would write logs to the pod file system.
205616792 Fixed core dump on running user schema setup.
225081332 Fixed allow privileged pods issue.

May 31, 2022

v1.6.8

hybrid v1.6.8

On May 31, 2022 we released an updated version of the Apigee hybrid software, v1.6.8.

For information on upgrading, see Upgrading Apigee hybrid to version 1.6.

Bug ID Description
233349518 Fixed "Invalid Resource" error generated for job/apigee-resources-install.
225939342 Fixed an error where deployment status would show as "Applying routing changes on {env}".
225198475 Fixed an issue where resource reference changes could not be detected.
232977937 Fixed an issue where deployment would become stuck on "Applying routing changes on" for multiple ingress gateways.
229804717 Fixed upgrade envoy to use distroless v1.22.0.

May 11, 2022

v1.6.7

hybrid v1.6.7

On May 11, 2022 we released an updated version of the Apigee hybrid v1.6.7 software.

For information on upgrading, see Upgrading Apigee hybrid to version 1.6.

Bug ID Description
227600373 Fixed an installation issue with Cassandra.
227538469 Configuration actions would write logs to the pod file system.
226964206 MART, runtime and synchronizer would write to the pod file system.
226464960 Apigee hybrid fresh installations on OpenShift 4.6 and 4.8 would fail.
225081332 Allow privileged pods issue.
224620542 On some Kubernetes platforms, logging would fail without adding an empty directory for the logs.
223081301 Fixed organization-level UDCA incorrect http-proxy secret name.
222649295 Organization-level UDCA would hang.
221266789 Hybrid logging functionality has been reworked. This should resolve issues with excessive log volume generation, frequent logger restarts, and ensure correct logger functionality with both docker and containerd runtimes.
213261445 Fixed reliance on keystore generated by cert manager for metrics endpoint and removed the need for a custom generate_cert script.
205616792 Fixed core dump on running user schema setup.

May 10, 2022

v1.7.0 & v1.5.10 & v1.6.7
Issue ID Affects Status Description
231758700
231976420
Apigee hybrid 1.7.x
Apigee hybrid 1.6.x
Apigee hybrid 1.5.x
OPEN Apigee Hybrid Dockerhub customers unable to pull images with Docker Content Trust enabled. Users are encountering the following error when pulling images for Apigee Hybrid from Docker Hub: ERRO[0001] Metadata for targets expired. This applies to the following hybrid components:
- google/apigee-authn-authz
- google/apigee-mart-server
- google/apigee-runtime
- google/apigee-synchronizer

Workaround
If you encounter this error, you can use one of the two following workarounds:
- Switch to using gcr.io/apigee-release to pull hybrid images.
- Disable docker content trust by setting the DOCKER_CONTENT_TRUST environment variable to 0

April 22, 2022

v1.5.10

hybrid v1.5.10

On April 22, 2022 we released an updated version of the Apigee hybrid v1.5.10 software.

For information on upgrading, see Upgrading Apigee hybrid to version 1.5.

Bug ID Description
225169066 Cassandra database backup and restore was not working when http_proxy is enabled under certain circumstances.
221885751 Hybrid logging functionality has been reworked. This should resolve issues with excessive log volume generation, frequent logger restarts, and ensure correct logger functionality with both docker and containerd runtimes.
202403896 Upgrade to Apigee hybrid v1.6 from v1.5 could fail due to annotation size.
221885751 Multiple issues with the Hybrid logging functionality have been addressed.

April 18, 2022

v1.7.0

Apigee hybrid v1.7.0

On April 18, 2022 we released Apigee hybrid v1.7.0.

For information on upgrading, see Upgrading Apigee hybrid to version 1.7.

mTLS communication between Cassandra clients and Cassandra nodes

Apigee hybrid now supports mTLS communication between Cassandra clients (MART, Sync, and MP) and Cassandra nodes. For related ports used, see feedbackSecure ports usage. (Implemented in Apigee hybrid v1.7.0)

Custom metrics scaling

Apigee hybrid v1.7.0 now supports custom metrics scaling using the metrics:appStackdriverExporter and metrics:proxyStackdriverExporter configuration properties. See metrics in the Configuration properties reference. (Implemented in Apigee hybrid v1.7.0)

OAuth JWT access tokens

Apigee hybrid v1.7.0 now supports JWT operations that allow the OAuthV2 policy to generate, verify, and refresh access tokens that conform to the JWT token standard. See Using JWT OAuth tokens. (Implemented in Apigee hybrid v1.7.0)

Cloud Logging

Apigee hybrid v1.7.0 now supports the <CloudLogging> element in the MessageLogging policy that lets you log messages to Cloud Logging. (Implemented in Apigee hybrid v1.7.0)

** PublishMessage policy**

Apigee hybrid v1.7.0 now supports the PublishMessage policy that lets you publish your API proxy flow information to a Google Cloud Pub/Sub topic.

GraphQL policy now supports JSON-encoded payloads. (Implemented in Apigee X, March 15, 2022)

Bug ID Description
224577096 Support Added for Anthos Service Mesh 1.12 (Fixed in Apigee hybrid v1.6.6)
219523719 Fix to address CPU and memory consumption when debug-session is enabled with response-status as the filtering criteria. (Fixed in Apigee X, March 15, 2022)
217386412 Change the property set logging level to fine when property is not found. (Fixed in Apigee hybrid v1.6.5)
215773113 Setting the securityPolicy appeared to have no effect for specific configurations. (Fixed in Apigee hybrid v1.6.5)
214960081 HTTPS endpoints could be called as HTTP resulting in an "EOF unexpected" error. (Fixed in Apigee hybrid v1.6.4)
211787541 Errors displayed in synchronizer logs for stale contracts. (Fixed in Apigee hybrid v1.6.5)
210590135 Invalid protocol in proxy.url flow variable in Apigee X. The proxy.url flow variable could show as http even when the request is https. (Fixed in Apigee hybrid v1.6.4)
210314786 The backup utility did not work with workload identity. (Fixed in Apigee hybrid v1.6.4)
209622008 Dynamic updates to rate in spike arrest are now reflected immediately. (Fixed in Apigee X, March 15, 2022)
209484701 Invalid client IP sent to analytics. (Fixed in Apigee hybrid v1.6.5)
209097822 Fixed and issue where SpikeArrest was not reflecting updated rate (Fixed in Apigee hybrid v1.6.3)
208474799 Apigee hybrid now supports ASM version 1.12. See Supported platforms for ASM version support for each supported version of Apigee Hybrid. (Fixed in Apigee hybrid v1.7.0)
208322185 Apigee hybrid Cassandra backup and restore can now use either a user-provided custom secret or a generated secret (Fixed in Apigee hybrid v1.6.3)
207762842 Hybrid logging functionality has been reworked. This should resolve issues with excessive log volume generation, frequent logger restarts, and ensure correct logger functionality with both docker and containerd runtimes.(Fixed in Apigee hybrid v1.7.0)
207618262 Fixed an issue where SpikeArrest opened too many connections to redis-envoy. (Fixed in Apigee hybrid v1.6.3)
207400645 Allow direct reads from API server to API client when enabled. (Fixed in Apigee hybrid v1.6.3)
205820658 Fixed an issue where Apigee X/hybrid Debug could show the authorization header. (Fixed in Apigee hybrid v1.6.3)
205810988 Resolve suspension dialog displayed "user not authorized" message for regions other than US This has been fixed. Location information is now added into the suspension URL from Apigee. (Fixed in Apigee X, December 02,2021)
205732137 Handle Quota correctly when the Operation group is set with empty string params (Fixed in Apigee hybrid v1.6.3)
205148816 Product Level Quota Info now available in Proxy. (Fixed in Apigee hybrid v1.6.3)
204943895 Quota Policy in Shared Flow now working properly. (Fixed in Apigee hybrid v1.6.3)
204943880 Fixed issue where SpikeArrest in Shared Flow did not have context of API Proxy. (Fixed in Apigee hybrid v1.6.3)
204905727 GenerateResponse was hanging on response flow when enabled=true. (Fixed in Apigee hybrid v1.6.5)
204368970 TLS variables are now set by Apigee Runtime. (Fixed in Apigee hybrid v1.6.3)
204146857 Fixed an issue where new environments were not created in Apigee hybrid deployment. (Fixed in Apigee hybrid v1.6.3)
203785814 A transient error could occur when calling conversion webhook for Apigee Telemetry. The error would occur when Apigee CRD is installed too early in the sequence. The installer job now checks for the correct sequence. (Fixed in Apigee hybrid v1.6.4)
203468593 Corrected the storageclass property name. (Fixed in Apigee hybrid v1.5.5)
203462573 The StorageClass set in overrides was not honored. (Fixed in Apigee hybrid v1.6.1)
202560276 AKS - containerd broke apigee-logger. (Fixed in Apigee hybrid v1.6.1)
202309278 Monetization: Eliminated a race condition that could make a prepaid developer's balance appear incorrect. (Fixed in Apigee hybrid v1.6.2)
202299966 Added new remote-address-related headers and modified the headers to be RFC compliant. (Fixed in Apigee hybrid v1.6.3)
200918549 There was an issue when using forward proxy with the ApigeeConnect agent. (Fixed in Apigee hybrid v1.6.1)
200700375 Fixed API products sorting issue in UI. Previously, sorting was disabled on the API products page. Sorting is now enabled. (Fixed in Apigee X, September 23, 2021)
200648523 Trace Variable, is_request_blocked, was showing incorrect information. (Fixed in Apigee hybrid v1.6.2)
199952038 The apigeectl command uses the new --restore flag to restore Cassandra to a previously saved snapshot. For more information, see Restoring in a single region. (Fixed in Apigee hybrid v1.6.6)
199807323 Updating Developer would reset the Developer billing type attribute (Fixed in Apigee hybrid v1.6.2)
199541025 Transaction ID is now required to be unique when used with prepaid developer balance credit API. (Fixed in Apigee hybrid v1.6.2)
198549304, 197730687, 196937143, 188370635, 187890034 Error state for conflicting dates is now correct. (Fixed in Apigee hybrid v1.6.2)
198036824 The securityContext was empty when it should have been populated. (Fixed in Apigee hybrid v1.6.1)
197945951 Stale DNS record in MP memory could cause an outage for a proxy. (Fixed in Apigee hybrid v1.6.4)
197910247 SetDialogflowResponse Policy - JSONPath expressions were not working. (Fixed in Apigee hybrid v1.6.1)
197711066 Cluster upgrade failed due to PDB (PodDisruptionBudget) policy not being met. (Fixed in Apigee hybrid v1.6.1)
196095557 Fixed proxy high response times. (Fixed in Apigee hybrid v1.5.4)
196024622 Hybrid images contained keys. (Fixed in Apigee hybrid v1.6.1)
196024483 Hybrid images did not set USER instruction when building the container. (Fixed in Apigee hybrid v1.6.1)
193799009 Fixed wrong status code shown on trace with ServiceCallout in PostClientflow. (Fixed in Apigee hybrid v1.5.4)
193520269 Fixed Apigee UI not showing the trace UI. (Fixed in Apigee hybrid v1.5.4)
193041253 Cassandra upgraded to v3.11.9 The Cassandra database in Apigee hybrid v1.7.0 has been upgraded to version 3.11.9. (Fixed in Apigee hybrid v1.7.0)
192987085 Fixed the ApiProductNotFound exception, which occurred when you deleted an API product but the deletion of associated rate plans was pending. (Fixed in Apigee X Monetization, November 3, 2021)
191853747 Apigee Workload Identities not working for specific configurations. (Fixed in Apigee hybrid v1.6.5)
190679584 There was an Incorrect error message on deploying AssertCondition policy with invalid condition. (Fixed in Apigee hybrid v1.6.1)
189341334 Fixed an issue to eliminate the potential for connection leaks for the watcher component. (Fixed in Apigee hybrid v1.6.3)
188407113 Invalid value in the ConsumptionPricingType during rate plan creation displayed the 500 status code. Now the status code for an invalid value is 4xx. (Fixed in Apigee X Monetization, November 3, 2021)
181259284 Fixed unresolved flow variables system.region.name and system.pod.name. (Fixed in Apigee hybrid v1.5.4)
180672249 FlowCallout succeeded, although SharedFlow had errors in deployment. (Fixed in Apigee hybrid v1.6.1)
173738907 Fixed support resource request/limit in override.yaml in apigee-metrics. (Fixed in Apigee hybrid v1.5.4)
173566787 Reuse existing target IPs if DNS resolution fail on DNS cache refresh. (Fixed in Apigee hybrid v1.6.5)
111777025 LookupCache: cachehit was shown false in trace when the actual value was true. (Fixed in Apigee hybrid v1.6.5)N/A
N/A If there is more than one SpikeArrest policy in a bundle, 502 errors will occur. (Fixed in Apigee X, December 12, 2021)
N/A If ServiceCallout is "fire and forget" (no tag), a race condition can occur if there is another policy that occurs after it. (Fixed in Apigee X, December 12, 2021)
N/A Fix bug delete dialog does not open Previously, on click of delete dialog, the dialog was not appearing, this fixes it (Fixed in Apigee X, October 05, 2021)
N/A Dynamic updates to rate in SpikeArrest may not reflect immediately. (Fixed in Apigee X, December 12, 2021)
Bug ID Description
217743790 ** Cassandra backup would run as privileged.**
204994504 Container Vulnerability fixed: CVE-2018-12934. (Fixed in Apigee hybrid v1.6.5)
N/A Multiple security fixes including CVE-2019-5021. (Fixed in Apigee hybrid v1.6.5)
N/A Miscellaneous Security updates and fixes. (Fixed in Apigee X, December 12, 2021)
205820658 A security issue was addressed. (Fixed in Apigee hybrid v1.6.2-hotfix.1)

"Apigee Deployer" role deprecated and replaced by "Apigee Environment Admin" The environment role "Apigee Deployer" has been deprecated, and replaced by "Apigee Environment Admin". (Implemented in Apigee X, December 2, 2022)

March 24, 2022

v1.6.6

hybrid v1.6.6

On March 24, 2022 we released an updated version of the Apigee hybrid v1.6.6 software.

For information on upgrading, see Upgrading Apigee hybrid to version 1.6.

Bug ID Description
224577096 Support Added for Anthos Service Mesh 1.12
204368970 Fixed a bug in TLS variables population
199952038 The apigeectl command uses the new --restore flag to restore Cassandra to a previously saved snapshot. For more information, see Restoring in a single region.
v1.5.9

hybrid v1.5.9

On March 24, 2022 we released an updated version of the Apigee hybrid v1.5.9 software.

For information on upgrading, see Upgrading Apigee hybrid to version 1.5.

Bug ID Description
224577096 Support Added for Anthos Service Mesh 1.12
204368970 Fixed the bug in TLS variables population

March 01, 2022

v1.5.8

hybrid v1.5.8

On March 1, 2022 we released an updated version of the Apigee hybrid v1.5.8 software.

For information on upgrading, see Upgrading Apigee hybrid to version 1.5.

Bug ID Description
219523719 Fix to address the CPU and memory consumption when debug-session is enabled with response-status as the filtering criteria.
217386412 Change the property set logging level to fine when property is not found.
215773113 Setting the securityPolicy appeared to have no effect for specific configurations.
209484701 Invalid client IP sent to analytics.
189233354 Distributed tracing with Jaeger would error out.
Bug ID Description
N/A Multiple security fixes including CVE-2019-5021.

February 28, 2022

v1.6.5

hybrid v1.6.5

On February 28, 2022 we released an updated version of the Apigee hybrid v1.6.5 software.

For information on upgrading, see Upgrading Apigee hybrid to version 1.6.

Bug ID Description
217386412 Change the property set logging level to fine when property is not found.
215773113 Setting the securityPolicy appeared to have no effect for specific configurations.
211787541 Errors displayed in synchronizer logs for stale contracts.
209484701 Invalid client IP sent to analytics.
204905727 GenerateResponse was hanging on response flow when enabled=true.
191853747 Apigee Workload Identities not working for specific configurations.
173566787 Reuse existing target IPs if DNS resolution fail on DNS cache refresh.
111777025 LookupCache: cachehit was shown false in trace when the actual value was true.
Bug ID Description
217743790 ** Cassandra backup would run as privileged.**
204994504 Container Vulnerability fixed: CVE-2018-12934.
N/A Multiple security fixes including CVE-2019-5021.

January 31, 2022

v1.6.4

hybrid v1.6.4

On January 31, 2022 we released an updated version of the Apigee hybrid v1.6.4 software

For information on upgrading, see Upgrading Apigee hybrid to version 1.6.

Bug ID Description
214960081 HTTPS endpoints could be called as HTTP resulting in an "EOF unexpected" error.
210590135 Invalid protocol in proxy.url flow variable in Apigee X. The proxy.url flow variable could show as http even when the request is https.
203785814 A transient error could occur when calling conversion webhook for Apigee Telemetry. The error would occur when Apigee CRD is installed too early in the sequence. The installer job now checks for the correct sequence.
197945951 Stale DNS record in MP memory could cause an outage for a proxy.
210314786 The backup utility did not work with workload identity.

January 28, 2022

v1.5.7

hybrid v1.5.7

On January 28, 2022 we released an updated version of the Apigee hybrid v1.5.7 software

For information on upgrading, see Upgrading Apigee hybrid to version 1.5.

Bug ID Description
214960081 HTTPS endpoints could be called as HTTP resulting in an EOF unexpected error.
204368970 TLS variables were not being set by Apigee Runtime.
189341334 Fixed potential connection leaks for watcher component.