Apigee known issues

This page applies to Apigee and Apigee hybrid.

View Apigee Edge documentation.

Select one or more of the following to filter this page:

Select product API hub API Monitoring/Analytics Apigee Apigee in VS Code Archive deployments Connectors Documentation hybrid Integration Integrated portal Monetization UI Select status FIXED OPEN Apigee 1-10 Apigee 1-9 hybrid 1.13 hybrid 1.12 hybrid 1.11 hybrid 1.10 hybrid 1.9 hybrid 1.8 hybrid 1.7 hybrid 1.6 hybrid 1.5 hybrid 1.4 hybrid 1.3

This section lists known issues for Apigee components. For a list of bugs, new features, and other release information, see the release notes.

Issue ID	Affects	Status	Description
384937220	hybrid 1.14.0	OPEN	Helm release creation may fail when there are multiple virtual hosts. When there are multiple virtual hosts, the Helm release creation may fail due to conflicting ApigeeRoute names. The workaround is to run the following commands for every virtual host when creating: kubectl annotate ar apigee-ingressgateway-internal-chaining-PROJECT_ID_SUFFIX -n APIGEE_NAMESPACE meta.helm.sh/release-name=NEW_ENV_GROUP_NAME --overwrite kubectl annotate cert apigee-ingressgateway-internal-chaining-PROJECT_ID_SUFFIX -n APIGEE_NAMESPACE meta.helm.sh/release-name=NEW_ENV_GROUP_NAME --overwrite where: PROJECT_ID_SUFFIX is a unique suffix for internal chaining for your project in Kubernetes. You can find this suffix with the following command: kubectl get svc -n apigee -l app=apigee-ingressgateway | grep internal-chaining Your output will look something like: kubectl get svc -n apigee -l app=apigee-ingressgateway | grep internal-chaining apigee-ingressgateway-internal-chaining-my-project--1234567 ClusterIP 34.118.226.140 <none> 15021/TCP,443/TCP 5d6h In the example output, my-project--1234567 is the PROJECT_ID_SUFFIX. APIGEE_NAMESPACE is your Apigee namespace. NEW_ENV_GROUP_NAME is the name the additional environment group. Update this value for each virtual host.
N/A	Apigee hybrid	OPEN	Invalid service account can send the UpdateControlPlaneAccess api into a retry loop. If a user provides an invalid service account to the UpdateControlPlaneAccess api, the operation goes on a retry loop effectively locking the organization from invoking the API until the operation times out.
N/A	Apigee hybrid	OPEN	The apigee-logger component does not support Workload Identity Federation. apigee-logger utilizes Google IAM service accounts for shipping logs to Cloud Logging. This is due to FluentBit's lack of support for Workload Identity Federation, which prevents the apigee-logger from utilizing this feature.
N/A	Apigee hybrid	OPEN	Distributed trace in Apigee hybrid is not supported for orgs using data residency.
N/A	Apigee	OPEN	Apigee does not support key re-encryption, which means even after rotation, the old key version will still be used and you cannot change the CMEK key after org creation.
354716994	Apigee	OPEN	Apigee does not support Cloud External Key Manager.
381129705	Apigee Apigee hybrid	OPEN	In the OASValidation policy, when you specify an array type in header, the MP will throw an error, even if the input is valid.
364872027	Apigee Apigee hybrid	OPEN	PEM parsing error in JWT/JWS policies due to non-standard format For Apigee and Apigee hybrid versions 1.13 and higher, any deviations in the required PEM format of keys used in Apigee JWS or JWT policies may result in a parsing error. For example, placing any character other an a newline (/n) immediately before the "-----END" line (post-encapsulation boundary) is not allowed and will result in an error. To prevent this error, make sure that no characters other than a newline, such as trailing spaces or slashes, immediately precede the post-encapsulation boundary. For more information about the encoding used for public or private keys, see IETF RFC 7468.
310191899	Apigee Apigee hybrid	OPEN	Timeouts when deploying API proxies and shared flows The following endpoints may experience timeouts when used with a high volume of queries per second (QPS): organizations.environments.apis.revisions. deployments.deploy organizations.environments.apis.revisions. deployments.undeploy organizations.environments.sharedflows.revisions. deployments.deploy organizations.environments.sharedflows.revisions. deployments.undeploy To reduce the likelihood of timeouts, we recommend a target of three QPS when using these endpoints.
329304975	Apigee	FIXED	Limit on number of basepaths per environment Apigee is enforcing a temporary limit of 1000 basepaths per environment to avoid potential failures when deploying API proxy revisions. While this limit is in place, you can deploy up to 1000 API proxy revisions (each containing a single basepath) per environment. If your API proxies or revisions contain more than one basepath, the total number of basepaths per environment must not exceed 1000.
333791378	hybrid 1.12.0	OPEN	Cassandra backup and restore features not supported by Helm migration tool For the steps required to install a patch for the workaround, see Troubleshooting.
310384001	hybrid 1.11.0	OPEN	Cert validation failures may return a 502 instead of a 503 error response when users add the tag <Enforce>true</Enforce> in the target <SSLInfo> block for default validation of TLS target endpoint certificates.
289583112	Apigee	OPEN	OASValidation policy does not work with global security requirements in OpenAPI specifications If the OASValidation policy specifies an <OASResource> with security requirements set at a global level, the security requirements are not enforced. Workaround: To ensure enforcement, all security requirements must be set at the operation level in the OpenAPI specification passed in the <OASResource> element of the OASValidation policy.
205666368	Apigee hybrid 1.10.2 hybrid 1.10.3	FIXED in Apigee 1-10-0-apigee-6 and Hybrid 1.10.3-hotfix.1	Apigee hybrid does not validate the target certificate by default. See About setting TLS options in a target endpoint or target server.
295929616	hybrid 1.10.0 and later	FIXED in hybrid 1.10.3	Installing Apigee hybrid 1.10 on OpenShift (OSE) can fail with out-of-memory errors. Installing or upgrading to Apigee hybrid 1.10.0 through 1.10.2 could fail on OSE due to out-of-memory issues. Fixed in Apigee hybrid version 1.10.3.
292268968	hybrid 1.10.1	OPEN	apigee-udca may not honor the http proxy settings. If the firewall forces all traffic through a forward-proxy, apigee-udca may go into a crash-loop backoff state.
269573358	Apigee hybrid 1.8.0 and later	OPEN	OASValidation policy fails with Unable to parse JSON error. The OASValidation Policy fails when the JSON content does not match the anticipated pattern. For example, if a header is expecting a value in the format <text>@<text> and is populated with text missing the @ symbol, the policy will fail with an Unable to parse JSON error. If the OASValidation policy specifies an <OASResource> containing a path parameter that utilizes a $ref schema, the policy will fail with an Unable to parse JSON - Unrecognized token error. Workaround: Do not use $ref in the path parameters of the OpenAPI spec specified in the <OASResource> element.
299953958	Apigee hybrid 1.8.0 and later	OPEN	Deployment Issues with OAS Validation while using circular reference. Apigee deployment will fail for OAS Validation policy when using circular references for OpenAPI 3.0.0 specification as it gets into an infinite loop. Workaround: Use an OpenAPI specification yaml without circular references.
289254725	Apigee Apigee 1-10-0-apigee-3 hybrid 1.8.8 hybrid 1.9.3	FIXED in Apigee 1-10-0-apigee-5 OPEN in hybrid	Proxy deployments that include the OASValidation policy may fail. Proxy deployments that include the OASValidation policy may fail if: The OpenAPI specification used for validation in the OASValidation policy is in YAML format, and The YAML-formatted OpenAPI specification contains a floating number. For example: schema: type: number example: 2.345
284500460	Apigee Apigee 1-10-0-apigee-1	FIXED	Increase in latency for Message Logging policy when used with Cloud Logging. To avoid increasing latency in responses to the client, the Message Logging policy should be attached to the PostClientFlow. For more information on using policies in PostClientFlows, see Controlling API proxies with flows.
282997216	hybrid 1.8.0 and later hybrid 1.9.0 and later	OPEN	Special characters not allowed in Cassandra Jolokia password Use only alphanumeric characters for the Cassandra Jolokia password. Using special characters (including but not limited to "!", "@", "#", "$", "%", "^", "&", & "*") can cause Cassandra startup to fail.
271415351	Apigee	OPEN	Avoid concurrent API proxy or SharedFlow deployments Concurrent deployment requests for a SharedFlow or API proxy can result in an inconsistent state in the Management Server where multiple revisions are shown as deployed. This can happen, for example, when concurrent runs of a CI/CD deployment pipeline occur using different revisions. To avoid this problem, avoid deploying API proxies or SharedFlows before the current deployment is complete.
271689008	hybrid 1.9.0 and later	OPEN	cert-manager pods on OpenShift version 4.7 to 4.10 do not start as expected With cert-manager v1.10.1 on OpenShift versions 4.7 to 4.10, the cert-manager pods do not start as expected. To resolve the problem, modify the security config constraint as described in the cert-manager 1.10 release notes.
270371160	hybrid 1.9.0 and later	FIXED	Apigee Ingress gateway supports TLS1.2+ protocol/ciphers only Apigee Ingress gateway only supports TLS1.2+, and not earlier versions of TLS.
269139342	hybrid 1.7.0 and later	OPEN	apigeectl getOrg does not follow HTTP_PROXY settings in overrides.yaml Apigee organization validation does not follow HTTP Forward proxy rules set in overrides.yaml. Set validateOrg: false to skip this validation.
266452840	Apigee hybrid 1.7.0 and later hybrid 1.8.0 and later hybrid 1.9.0 and later	OPEN	Web sockets not working with Anthos Service Mesh 1.15.3 in Apigee X and Apigee Hybrid In certain circumstances, web sockets are not working for Apigee X and Apigee Hybrid when using Anthos Service Mesh 1.15.3-asm.6.
242213234	Apigee	OPEN	API product fails to load with a "no connections available" error This error might be returned when attempting to load API products: "Products were not loaded successfully. Error: no connections available from the Apigee connect agent(s)." The problem occurs after enabling VPC service control in the Google Cloud project and adding iamcredentials.googleapis.com as one of the restricted services in the service perimeter. Workaround: Manually create an egress rule, such as the following: -egressTo: operations: -serviceName: "iamcredentials.googleapis.com" methodSelectors: -method: resources: -projects/608305225983 egressFrom: identityType: ANY_IDENTITY
247540503	hybrid 1.7.0 and later hybrid 1.8.0 and later	OPEN	A race condition with encryption key lookup can cause KVM lookup failures. In certain circumstances at very high throughput a race condition with encryption key lookup can cause KVM lookup failures.
258699204	hybrid 1.8.0 and later	OPEN	The default memory requests and limits for metrics pods have inadvertently changed in 1.8.x. If you see problems with the apigee-telemetry-app or apigee-telemetry-proxy pods not running, change the metrics resources requests and resources limits properties to match the following defaults in Configuration property reference: metrics. Configuration property Default value metrics.aggregator.resources.requests.memory: 512Mi metrics.aggregator.resources.limits.memory: 3Gi metrics.app.resources.requests.memory: 512Mi metrics.app.resources.limits.memory: 1Gi metrics.appStackdriverExporter.resources.requests.memory: 512Mi metrics.appStackdriverExporter.resources.limits.memory: 1Gi metrics.proxy.resources.requests.memory: 512Mi metrics.proxy.resources.limits.memory: 1Gi metrics.proxyStackdriverExporter.resources.requests.memory: 512Mi metrics.proxyStackdriverExporter.resources.limits.memory: 1Gi Apply the changes with apigeectl apply with the ‑‑telemetry flag: apigeectl apply --telemetry -f overrides.yaml
260324159	Apigee 1-9-0-apigee-16	OPEN	API proxy and shared flow deployments taking up to 30 minutes. API proxies and shared flows could take around 20 to 30 minutes to deploy in the runtime plane in certain circumstances due to a 'socket closed' error in synchronizer.
254505866	API hub	FIXED	New regions cause provisioning failure Provisioning API hub using the UI fails if you select a region other than the following: asia-east1 asia-southeast1 europe-west1 europe-west4 us-central1 us-east1 us-west1 us-west4
251897633	Documentation	OPEN	Apigee hybrid version selector The Apigee hybrid version selector works only if you select or click directly on the text.
250875730	All	OPEN	The message "Precondition check failed" is displayed in the audit log. This is expected to occur every minute and does not affect your billing cost.
260772383	hybrid 1.8.0 and later	OPEN	Socket bind error on the AKS platform If installing hybrid on AKS, you may see this error: envoy config listener '0.0.0.0_443' failed to bind or apply socket options: cannot bind '0.0.0.0:443': Permission denied Workaround: Add the following svcAnnotations stanza to the overrides file: ingressGateways: - name: INGRESS_NAME ... svcAnnotations: service.beta.kubernetes.io/azure-load-balancer-internal: "true" See Configure the hybrid runtime. See also Use an internal load balancer with AKS.
241786534	hybrid 1.8.0 and later	OPEN	MART sometimes unable to connect to FluentD. When using Org-scoped UDCA, MART is sometimes unable to connect to FluentD. Org-scoped UDCA is the default in Apigee hybrid version 1.8. See orgScopedUDCA in the Configuration property reference.
N/A	hybrid 1.6.0 and later	OPEN	apigee-logger not working on Anthos BareMetal with CentOS or RHEL. After migration of apigee-logger from fluend to fluent-bit in Apigee hybrid version 1.6.6, the logger stopped working on Anthos BareMetal with CentOS or RHEL.
231758700 231976420	hybrid 1.5.0 and later	OPEN	Apigee Hybrid Dockerhub customers unable to pull images with Docker Content Trust enabled. Users are encountering the following error when pulling images for Apigee Hybrid from Docker Hub: ERRO[0001] Metadata for targets expired. This applies to the following hybrid components: google/apigee-authn-authz google/apigee-mart-server google/apigee-runtime google/apigee-synchronizer Workaround If you encounter this error, you can use one of the two following workarounds: Switch to using gcr.io/apigee-release to pull hybrid images. Disable docker content trust by setting the DOCKER_CONTENT_TRUST environment variable to 0
207762842	hybrid 1.5.0 and later	OPEN	Logs not shipped to Cloud Logging by apigee-logger. Current apigee-logger configurations, including liveness probes, are incompatible with the Kubernetes runtime, resulting in logs not being shipped to Cloud Logging as expected. This problem also causes the apigee-logger pods to crash regularly. This issue affects Apigee hybrid installations on AKS, Anthos Bare Metal, and other platforms. Note that in some cases this issue results in excessive log volume.
203827738	Archive deployments	OPEN	Configurable API proxy without operations fails. Proxies that do not contain operations, or contain operations without HTTP matches, will return a 404 error code if the request path does not contain exactly one segment in addition to the basepath. For example, requests to /basepath will fail, but requests to /basepath/user may succeed. To prevent failure, add an Operations section with at least one http_match to your configurable API proxy.
201429104	Apigee	OPEN	Wildcard in proxy basepath results in incorrect request path. Usage of a wildcard (*) in the proxy basepath of a configurable API proxy results in forwarding incorrect request paths to the backend target. To prevent incorrect request path forwarding, avoid using * in the configurable API proxy basepath until the issue is fixed.
191291501, 191000617	Apigee	OPEN	Changing the email address of a developer entity will fail in the UI.
191002224	hybrid 1.5.0 and later	OPEN	Changing an email address fails while using the PUT /organizations/{org_name}/developers/{developer_email} API.
184555974	hybrid 1.5.0 and later	OPEN	The apigee-logger Fluentd can't parse logs in the OpenShift cluster.
N/A	Archive deployments	OPEN	Managing and debugging Apigee archive deployments in the UI is not supported In the Apigee UI, you cannot view, confirm deployment status, or manage your archive deployments, as described Deploying an API proxy, or use the Debug UI as described in Using Debug. As a workaround, you can use gcloud or the API to List all archive deployments in an environment and use the Debug API.
N/A	Archive deployments	OPEN	Rolling back an archive deployment is not supported Rolling back an archive deployment is not currently supported. To remove a version of an archive deployment you need to either redeploy a previous version of an archive or delete the environment.
N/A	Apigee in VS Code	OPEN	Google Authentication in policies is not supported in Apigee in Visual Studio Code (VS Code) Google authentication in ServiceCallout and ExternalCallout policies, as described in Using Google Authentication, is not supported in Apigee in VS Code.
146222881	hybrid 1.3.0 and later	OPEN	Invalid HTTP Header error Invalid HTTP Header error: The Istio ingress switches all incoming target responses to the HTTP2 protocol. Because the hybrid message processor only supports HTTP1, you may see the following error when an API proxy is called: http2 error: Invalid HTTP header field was received: frame type: 1, stream: 1, name: [:authority], value: [domain_name] If you see this error, you can take either of the following actions to correct the problem: Modify the target service to omit the Host header in the response. Remove the Host header using the AssignMessage policy in your API proxy if necessary.
N/A	Integrated portal	OPEN	SmartDocs Apigee supports OpenAPI Specification 3.0 when you publish your APIs using SmartDocs on your portal, though a subset of features are not yet supported. For example, allOf properties for combining and extending schemas. If an unsupported feature is referenced in your OpenAPI Specification, in some cases the tools will ignore the feature but still render the API reference documentation. In other cases, an unsupported feature will cause errors that prevent the successful rendering of the API reference documentation. In either case, you will need to modify your OpenAPI Specification to avoid use of the unsupported feature until it is supported in a future release. When using Try this API in the portal, the Accept header is set to application/json regardless of the value set for consumes in the OpenAPI Specification.
N/A	Integrated portal	OPEN	Portal admin Simultaneous portal updates (such as page, theme, CSS, or script edits) by multiple users is not supported at this time. If you delete an API reference documentation page from the portal, there is no way to recreate it; you'll need to delete and re-add the API product, and regenerate the API reference documentation. When customizing your portal theme, it may take up to 5 minutes for changes to fully apply.
N/A	Integrated portal	OPEN	Portal features Search will be integrated into the integrated portal in a future release.
N/A	Integrated portal	OPEN	SAML identity provider Single logout (SLO) with the SAML identity provider is not supported for custom domains. To enable a custom domain with a SAML identity provider, leave the Sign-out URL field blank when you configure SAML settings.
191815997	hybrid 1.6.0 and later	OPEN	If a hybrid customer configures a forward proxy for the API proxy, Google token will not work unless it has direct access to *.googleapis.com.
N/A	Apigee	FIXED in Apigee 1-12-0-apigee-2 and hybrid 1.12.0	API Monitoring and Cloud Monitoring show abnormal spikes API Proxy request and response counts (for proxy and targets) show abnormal spikes Here is a sample showing such a spike: (view larger image) Due to a bug, the system registers the count incorrectly for a brief period and the count is corrected. This happens where there is a reduction in API traffic (which results in a scale down of API gateways). To distinguish actual spikes in requests vs. this issue, please consult the API Analytics page (specifically the Proxy Performance and Target Performance pages) Affected Metrics: apigee.googleapis.com/proxyv2/request_count apigee.googleapis.com/proxyv2/response_count apigee.googleapis.com/targetv2/request_count apigee.googleapis.com/targetv2/response_count New metrics You can use the new metrics to avoid this issue. Note: These metrics are currently available for Apigee and Apigee hybrid v1.12 and above. For Apigee hybrid v1.10 and v1.11, use the original metrics. Metric Description apigee.googleapis.com/proxy/request_count Number of requests to the Apigee proxy since the last sample was recorded. apigee.googleapis.com/proxy/response_count Number of responses sent by the Apigee API proxy. apigee.googleapis.com/proxy/latencies Distribution of latencies, which are calculated from the time the request was received by the Apigee proxy to the time the response was sent from the Apigee proxy to the client. apigee.googleapis.com/target/request_count Number of requests sent to the Apigee target since the last sample was recorded. apigee.googleapis.com/target/response_count Number of responses received from the Apigee target since the last sample was recorded. apigee.googleapis.com/target/latencies Distribution of latencies, which are calculated from the time the request was sent to the Apigee target to the time the response was received by the Apigee proxy. Time does not include the Apigee API proxy overhead. For Apigee hybrid, see: Metrics collection overview and View metrics.
203778087	hybrid 1.5.3 and later	OPEN	apigee-stackdriver-logging-agent currently runs as root. Workaround: Disable the logging agent on hybrid.
205629443	Apigee	OPEN	If ServiceCallout is fire and forget (no <Response> tag), a race condition can occur if there is another policy that occurs after it. Workaround: To maintain the fire and forget behavior: Add <Response>calloutResponse</Response> to the ServiceCallout. Set continueOnError to true.
207719377	Apigee	FIXED in Apigee 1-11-0-apigee-1	If there is more than one SpikeArrest policy in a bundle, 502 errors will occur. Workaround: Avoid using more than one SpikeArrest policy in the proxy to prevent the issue.
209097822	hybrid 1.5.0 and later Apigee	OPEN	Dynamic updates to rate in spike arrest may not reflect immediately For a particular key, if there is continuous traffic, the key may not be rate limited at the updated rate. If there is five minutes of no traffic for a particular key, the rate will be reflected. Workaround: Redeploy the proxy with a new reference variable if the rate has to take effect immediately. Or use two conditional spike arrests with different flow variables to adjust the rate.
221305498	Apigee	OPEN	API Monitoring may display fault code of '(not set)'. API Monitoring of Configurable API Proxies may display a fault code of '(not set)' for responses with a non-2xx status from the target.
246774745	Apigee	FIXED	Value of io.timeout.millis is not honored when used with multiple dynamic targets. If a proxy sets two or more io.timeout.millis values in two or more flows using the same target host, only one io.timeout.millis value is honored.
245664917	hybrid 1.8.x	OPEN	Apigee hybrid upgrade error can be ignored During upgrade to Apigee hybrid 1.8.x, after running apigeectl init and confirming that check-ready succeeded, you may notice, if you view the pods, that the Cassandra schema validation job is in an error state. This is a harmless condition, and you can safely move to the next step in the upgrade procedure.
300660653	Apigee	OPEN	An error should be, but is not, returned when deploying proxies with the same path to multiple environments that are attached to the same instance and environment group Deploying proxies with the same path to multiple environments that are attached to the same instance and environment group is not allowed and should return a warning message about a base path conflict. Instead, no error is shown and the deployments appear to succeed. Workaround: When deploying and after deployment, verify that there are not base path conflicts with deployed proxies and correct as needed.
301458133	Apigee	OPEN	Some proxy deployment attempts return an error that the revision is immutable When attempting to save a previously deployed proxy, the deployment might fail with an error stating that the revision is immutable. Workaround: Click the dropdown arrow next to the Save button and select Save as new revision. Then reattempt the deployment.
301845257	Apigee	FIXED	Attempting to deploy more than 800 proxies to an environment group fails with an error. The limit on which an error is returned is lower than 800 when the basepaths are longer than 15 characters.
N/A	Apigee 1-9-0-apigee-23	OPEN	TLS version upgrade required for clients experiencing Unsupported protocol errors Updates to the default set of ciphers supported by Apigee servers to enhance security may result in Unsupported protocol errors for some TLS versions. Customers encountering these errors should review the full list of ciphers supported by the FIPS build of Envoy.
315874988	Apigee	OPEN	With gRPC proxy requests, gRPC trailers are removed from the response When a call is made to a gRPC Target Server, the only trailer that's returned is the "grpc-status" trailer. All other trailers are removed from the response.
341157011	Apigee	OPEN	Specs generated with Gemini Code Assist in Cloud Code that also include non-ASCII UTF-8 characters are not parsed correctly when uploaded to API hub. As a result API hub won't extract operations, definitions, and other metadata correctly.
338285095	Apigee	FIXED	Apps associated with an AppGroup do not appear in the Apps list in the Apigee UI in Cloud console. As a result, users cannot access the app's App Detail page in the console. Using search in the console with a partial app name or API key search for the app is not supported until this issue is resolved. Apigee hybrid organizations are not impacted by this problem, as they use the Classic UI to view the app details. Partial workaround: A list of all apps is available using the APIs. See organizations.apps.list.
355714868	Monetization Apigee hybrid	OPEN	Recurring, setup, and top-up fee data not captured or billed for hybrid organizations For hybrid organizations with monetization, the recurring (RECURRING_FEE), setup fee (SETUP_FEE) and top-up fee (TOPUP_FEE) data is not captured or billed for post-paid billing. For pre-paid billing, the wallet deduction occurs for these fees but is not reported.