1.3.0 - Apigee hybrid runtime release notes

On July 30, 2020, we released Apigee hybrid runtime version 1.3.0.


New features and updates

This section describes the new features and enhancements in this release.

New installation prerequisites


Apigee hybrid requires cert-manager v0.14.2 to manage and verify certificates. You can use kubectl to install and configure cert-manager directly from github > jetstack > cert-manager. See Download and install cert-manager.

Anthos Service Mesh

Apigee hybrid uses the Istio distribution provided with Anthos Service Mesh (ASM) version 1.5.x to create and manage the runtime ingress gateway. See Download and install ASM.

Environment groups

Environment groups allow you to logically group environments together. Environments within each group share the same hostnames. You can group environments by function, by hostname address, by region if you are implementing a multi-region hybrid installation, or by any other metric you choose.

Environment groups provide the same routing features provided by the virtualHosts property in Apigee hybrid version 1.2.

See About environment groups.

New CLI command

The following new command was added to the apigeectl CLI. For more information about each command, see apigeectl.

Command Description
encode Returns a list of encoded names of all the ApigeeDeployments for the specified organization or the specified environment within the specified organization.

CLI flag changes

The following new flags were added to the apigeectl CLI. For more information about each command, see apigeectl.

Flag Status Description
--all-envs New Applies the apigeectl command to all environments under the organization specified in your overrides config file.
--env New Applies the configuration to the specified environment under the organization specified in your overrides config file.
--datastore New Applies the configuration to the datastore scope (Cassandra).
--dry-run Changed Arguments depend on the version of kubectl you are using:
  • kubectl version 1.7.x or older: ‑‑dry‑run=true
  • kubectl version 1.8.x or newer: ‑‑dry‑run=client
--org New Applies the configuration to the specified organization.
--telemetry New Applies the configuration for telemetry components: apigee-logger and apigee-metrics.
-c, --components Removed Instead, apply changes by env, org, datastore, or telemetry scopes.

Unique hashed values for organization and environment names

Apigee hybrid now uses a unique hashed value to keep track of organization and environment names. This avoids errors when org and env names are too long to concatenate. You can see the hashed values with the apigeectl encode command. For example:

$ ./apigeectl encode --org hybrid-example
List of ApigeeDeployments are:

$ ./apigeectl encode --org hybrid-example --env example-env
List of ApigeeDeployments are:

See apigeectl.

New axHashSalt configuration property

A new org-level configuration property, axHashSalt, was added. This property lets you specify the name of a Kubernetes secret that contains a hashing salt value used to encrypt obfuscated user data sent to Apigee analytics. For more information, see the Configuration property reference.

Added support for mTLS on the Istio ingress

You can configure mTLS. on the Istio ingress. For more information, see Configuring mTLS.


Apigee Watcher pulls virtual hosts related changes for an org from synchronizer and makes necessary changes to configure istio ingress.

Watcher introduces:

  • New service account: apigee-watcher
  • New role: Apigee Runtime Agent
  • New configuration property: watcher

See Create service accounts and the Configuration properties reference.

GA of the OASValidation policy

The OASValidation policy is now GA.

GA of the WebSockets feature

The WebSockets feature is now GA. See Using WebSockets.

Apigee Connect enabled by default

Apigee Connect is now GA and is enabled by default for all new Apigee installs. See Apigee Connect.

Metrics enabled by default

Metrics are enabled by default for all new Apigee hybrid installations. See Configure metrics collection.

Changes to the virtualHosts configuration property

Use environment groups to configure routing rules. This replaces the virtualHosts:routingRules property. See Configuring virtual hosts and About environment groups.

Changes to product limits

The following default limits have been changed:

Limit New value
Message logging payload 11 MB
Target connection timeout 600 seconds

Bugs fixed

The following bugs are fixed in this release. This list is primarily for users checking to see if their support tickets have been fixed. It's not designed to provide detailed information for all users.

Issue ID Description
153755536 An issue was fixed where a long environment and/or organization name caused an error. Org and env names are now truncated to prevent this problem from occurring. A 32 character limit is also now enforced for org and env names.
155913227 An issue was fixed where the runtime was unable to connect to the UDCA (fluentd) endpoint even though UDCA is healthy.
155913227 Unable to connect to UDCA (fluentd) pod
153755536 Error received when environment length is long

Known issues

The following table describes the known issues for this release:

Issue Description
172332786 Double slashes (//) in a request can cause the request not to resolve. You can solve this by applying a configuration to your Istio ingress that filters for double slashes. See Remove double slashes from requests for instructions.
162759110 Base paths consisting of only "/" will fail. You must include a path after the "/".

For example:

  • /123
  • /v1/customers/123
161658025 Inaccurate deployment status for Shared Flows

For Shared Flows, the deployment status is not being reported correctly to the UI, resulting in an indefinitely spinning deployment icon. If you have deployed a Shared Flow and see the spinning deployment icon, you must assume that the deployment has in fact failed.

To obtain the correct deployment status, use the v1.organizations.environments.sharedflows.revisions.deployments API. For example:

curl -s -H "$AUTH" \


146222881 Invalid HTTP Header error: The Istio ingress switches all incoming target responses to the HTTP2 protocol. Because the hybrid message processor only supports HTTP1, you may see the following error when an API proxy is called:

http2 error: Invalid HTTP header field was received: frame type: 1, stream: 1,

name: [:authority], value: [domain_name]

If you see this error, you can take either of the following actions to correct the problem:

  • Modify the target service to omit the Host header in the response.
  • Remove the Host header using the AssignMessage policy in your API proxy if necessary.
143659917 The PopulateCache policy's expiration setting must be set to an explicit value between 1 and 30. For example: