Apigee known issues

Stay organized with collections Save and categorize content based on your preferences.

You're viewing Apigee and Apigee hybrid documentation.
View Apigee Edge documentation.

This section lists known issues for Apigee components. For a list of bugs, new features, and other release information, see Release notes.

Issue ID Affects Status Description
N/A • Apigee hybrid 1.6.0 and later
 OPEN apigee-logger not working on Anthos BareMetal with CentOS or RHEL.

After migration of apigee-logger from fluend to fluent-bit in Apigee hybrid version 1.6.6, the logger stopped working on Anthos BareMetal with CentOS or RHEL.
N/A • Archive deployments
 OPEN Managing and debugging Apigee archive deployments in the UI is not supported

In the Apigee UI, you cannot view, confirm deployment status, or manage your archive deployments, as described Deploying an API proxy, or use the Debug UI as described in Using Debug. As a workaround, you can use gcloud or the API to List all archive deployments in an environment and use the Debug API.
N/A • Archive deployments
 OPEN Rolling back an archive deployment is not supported

Rolling back an archive deployment is not currently supported. To remove a version of an archive deployment you need to either redeploy a previous version of an archive or delete the environment.
N/A • Apigee in VS Code
 OPEN Google Authentication in policies is not supported in Apigee in Visual Studio Code (VS Code)

Google authentication in ServiceCallout and ExternalCallout policies, as described in Using Google Authentication, is not supported in Apigee in VS Code.
N/A • Integrated portal
 OPEN SmartDocs
  • Apigee supports OpenAPI Specification 3.0 when you publish your APIs using SmartDocs on your portal, though a subset of features are not yet supported. For example, allOf properties for combining and extending schemas.

    If an unsupported feature is referenced in your OpenAPI Specification, in some cases the tools will ignore the feature but still render the API reference documentation. In other cases, an unsupported feature will cause errors that prevent the successful rendering of the API reference documentation. In either case, you will need to modify your OpenAPI Specification to avoid use of the unsupported feature until it is supported in a future release.

  • When using Try this API in the portal, the Accept header is set to application/json regardless of the value set for consumes in the OpenAPI Specification.
N/A • Integrated portal
 OPEN Portal admin
  • Simultaneous portal updates (such as page, theme, CSS, or script edits) by multiple users is not supported at this time.
  • If you delete an API reference documentation page from the portal, there is no way to recreate it; you'll need to delete and re-add the API product, and regenerate the API reference documentation.
  • When customizing your portal theme, it may take up to 5 minutes for changes to fully apply.
N/A • Integrated portal
 OPEN Portal features
Search will be integrated into the integrated portal in a future release.
N/A • Integrated portal
 OPEN SAML identity provider
Single logout (SLO) with the SAML identity provider is not supported for custom domains. To enable a custom domain with a SAML identity provider, leave the Sign-out URL field blank when you configure SAML settings.
N/A • Apigee
 OPEN API Monitoring and Cloud Monitoring show abnormal spikes

  • API Proxy request and response counts (for proxy and targets) show abnormal spikes

    Here is a sample showing such a spike:

    (view larger image)

  • Due to a bug, the system registers the count incorrectly for a brief period and the count is corrected. This happens where there is a reduction in API traffic (which results in a scale down of API gateways).
  • To distinguish actual spikes in requests vs. this issue, please consult the API Analytics page (specifically the Proxy Performance and Target Performance pages)

Affected Metrics:

  • apigee.googleapis.com/proxyv2/request_count
  • apigee.googleapis.com/proxyv2/response_count
  • apigee.googleapis.com/targetv2/request_count
  • apigee.googleapis.com/targetv2/response_count

231758700
231976420

 • Apigee hybrid 1.5.0 and later
 OPEN Apigee Hybrid Dockerhub customers unable to pull images with Docker Content Trust enabled.
Users are encountering the following error when pulling images for Apigee Hybrid from Docker Hub: ERRO[0001] Metadata for targets expired. This applies to the following hybrid components:
  • google/apigee-authn-authz
  • google/apigee-mart-server
  • google/apigee-runtime
  • google/apigee-synchronizer

Workaround

If you encounter this error, you can use one of the two following workarounds:

  • Switch to using gcr.io/apigee-release to pull hybrid images.
  • Disable docker content trust by setting the DOCKER_CONTENT_TRUST environment variable to 0

271689008

 • Apigee hybrid 1.9.0 and later
 OPEN cert-manager pods on OpenShift version 4.7 to 4.10 do not start as expected

With cert-manager v1.10.1 on OpenShift versions 4.7 to 4.10, the cert-manager pods do not start as expected. To resolve the problem, modify the security config constraint as described in the cert-manager 1.10 release notes.

270371160

 • Apigee hybrid 1.9.0 and later
 OPEN Apigee Ingress gateway supports TLS1.2+ protocol/ciphers only

Apigee Ingress gateway only supports TLS1.2+, and not earlier versions of TLS.

269573358

 • Apigee hybrid 1.8.0 and later
• Apigee
 OPEN OASValidation Policy fails with Unable to parse JSON

The OASValidation Policy fails when the JSON content does not match the anticipated pattern. For example if a header is expecting a value in the format <text>@<text>, and is populated with text missing the @ symbol.

269139342

 • Apigee hybrid 1.7.0 and later
 OPEN apigeectl getOrg does not follow HTTP_PROXY settings in overrides.yaml
Apigee organization validation does not follow HTTP Forward proxy rules set in overrides.yaml. Set validateOrg: false to skip this validation.

266957367

 • Apigee hybrid 1.7.0 and later
 FIXED IN Apigee hybrid 1.8.4 Anthos Service Mesh upgrade from 1.12 to 1.15 causes Ingress gateway 503 errors when using TLS/SSL headers

On Apigee Hybrid v1.7 installations, upgrading Anthos Service Mesh to v1.15 can cause 503 errors for proxies using TLS/SSL headers.

266452840

 • Apigee hybrid 1.9.0 and later
• Apigee hybrid 1.8.0 and later
• Apigee hybrid 1.7.0 and later
• Apigee
 OPEN Web sockets not working with Anthos Service Mesh 1.15.3 in Apigee X and Apigee Hybrid

In certain circumstances, web sockets are not working for Apigee X and Apigee Hybrid when using Anthos Service Mesh 1.15.3-asm.6.

260324159

 • Apigee (1-9-0-apigee-16)
 OPEN API proxy and shared flow deployments taking up to 30 minutes.
API proxies and shared flows could take around 20 to 30 minutes to deploy in the runtime plane in certain circumstances due to a 'socket closed' error in synchronizer.

259738092

 • Apigee hybrid 1.8.2
• Apigee hybrid 1.8.1
• Apigee hybrid 1.8.0
 FIXED IN Apigee hybrid 1.8.3 Intermittent 404's at the Apigee Ingress Gateway.
In certain circumstances, intermittent 404's could be seen at the Apigee Ingress Gateway due to an inconsistent configuration delivery mechanism.

258699204

 • Apigee hybrid 1.8.0 and later
 OPEN The default memory requests and limits for metrics pods have inadvertently changed in 1.8.x.

If you see problems with the apigee-telemetry-app or apigee-telemetry-proxy pods not running, change the metrics resources requests and resources limits properties to match the following defaults in Configuration property reference: metrics.

Configuration propertyDefault value
metrics.aggregator.resources.requests.memory:512Mi
metrics.aggregator.resources.limits.memory:3Gi
metrics.app.resources.requests.memory:512Mi
metrics.app.resources.limits.memory:1Gi
metrics.appStackdriverExporter.resources.requests.memory:512Mi
metrics.appStackdriverExporter.resources.limits.memory:1Gi
metrics.proxy.resources.requests.memory:512Mi
metrics.proxy.resources.limits.memory:1Gi
metrics.proxyStackdriverExporter.resources.requests.memory:512Mi
metrics.proxyStackdriverExporter.resources.limits.memory:1Gi

Apply the changes with apigeectl apply with the ‑‑telemetry flag:

apigeectl apply --telemetry -f overrides.yaml

257268790

 • Apigee (1-9-0-apigee-5)
 FIXED IN Apigee (1-9-0-apigee-16) Restart or bootup of new runtime pods
There is an edge case scenario where an invalid resource or bundle configuration resulting in unhandled exception will result in failure that leads to restart of runtime pods or bootup of new runtime pods.

254505866

 • Apigee API hub
 FIXED IN Apigee API hub New regions cause provisioning failure

Provisioning API hub using the UI fails if you select a region other than the following:

  • asia-east1
  • asia-southeast1
  • europe-west1
  • europe-west4
  • us-central1
  • us-east1
  • us-west1
  • us-west4

251897633

 • Documentation
 OPEN Apigee hybrid version selector

The Apigee hybrid version selector works only if you select or click directly on the text.

250875730

 • All
 OPEN The message "Precondition check failed" is displayed in the audit log.

This is expected to occur every minute and does not affect your billing cost.

247540503

 • Apigee hybrid 1.8.0 and later
• Apigee hybrid 1.7.0 and later
 OPEN A race condition with encryption key lookup can cause KVM lookup failures.

In certain circumstances at very high throughput a race condition with encryption key lookup can cause KVM lookup failures.

243880171

 • Apigee hybrid 1.8.0 and later
 FIXED IN Apigee hybrid 1.8.1 Readiness probe fails when http_proxy is configured to DENY internal network traffic.

If http_proxy is configured to DENY internal network traffic, the apigee-cassandra-default-* pod doesn't get into ready state.

Use the following work around:

  1. Update the readinessProbe section in the $APGEECTL_HOME/templates/1_apigee-datastore.yaml to add the line - unset http_proxy as follows:
    2. readinessProbe:
      exec:
        command:
          - /bin/bash
          - -c
          - unset http_proxy # this is the line that needs to be added.
          - /opt/apigee/ready-probe.sh
  2. Apply the changes: 
    $APGEECTL_HOME/apigeectl apply --datastore -f overrides.yaml
  3. Delete the unhealthy cassandra pod: 
    kubectl -n apigee delete pods unhealthy-cassandra-pod

243717191

 • Apigee hybrid 1.8.0 and later
 FIXED IN Apigee hybrid 1.8.1 apigee-cassandra-default-* pod is stuck in crashLoopBackOff state with Caused by: java.io.IOException: keystore password was incorrect exception in the logs.
Workaround: Delete the affected pod with the command:
kubectl -n delete pods <pod name>

243599452

 • Apigee hybrid 1.8.0 and later
 OPEN SvcAnnotations in IngressGateways causing validation failures.

The ingressGateways[].svcAnnotations field in overrides.yaml is not working as expected. If you need to add custom annotations to the Apigee ingress gateway service, create a custom Kubernetes service as documented in Expose Apigee ingress gateway.

243167389

 • Apigee hybrid 1.8.0 and later
 FIXED IN Apigee hybrid 1.8.1 Missing validation for ingress gateway name length.

Because the validating webhook does not check the length of the ingressGateways[].name value, long ingress gateway names can cause deployments to fail. The value of ingressGateways:name must meet the following requirements:

  • Have a maximum length of 17 characters
  • Contain only lowercase alphanumeric characters, '-' or '.'
  • Start with an alphanumeric character
  • End with an alphanumeric character

See ingressGateways[].name in the Configuration property reference.

242213234

 • Apigee
 OPEN API product fails to load with a "no connections available" error

This error might be returned when attempting to load API products: "Products were not loaded successfully. Error: no connections available from the Apigee connect agent(s)."

The problem occurs after enabling VPC service control in the GCP project and adding iamcredentials.googleapis.com as one of the restricted services in the service perimeter.

Workaround

Manually create an egress rule, such as the following:
- egressTo:
    operations:
    - serviceName: "iamcredentials.googleapis.com"
        methodSelectors:
        - method: \"*\"
    resources:
    - projects/608305225983
  egressFrom:
    identityType: ANY_IDENTITY

241786534

 • Apigee hybrid 1.8.0 and later
 OPEN MART sometimes unable to connect to FluentD.

When using Org-scoped UDCA, MART is sometimes unable to connect to FluentD. Org-scoped UDCA is the default in Apigee hybrid version 1.8. See orgScopedUDCA in the Configuration property reference.

230798614

 • Apigee hybrid 1.7.2
 FIXED IN Apigee hybrid 1.7.3 Unresponsive runtime behavior

If you are running Apigee hybrid 1.7.3 or an older version, and you are experiencing unresponsive hybrid runtime behavior, it may be caused by a version incompatibility between the runtime pods and C*. The recommended solution is to upgrade to Apigee hybrid 1.7.4 or later. See Upgrading Apigee hybrid.

218567150

 • Apigee hybrid 1.8.0 and later
 OPEN Apigee hybrid changes the UUID in the x-request-id header.

Apigee hybrid sometimes modifies x-request-id header. For example, in this header the 14th nibble has been changed from a 4 to a 9: 123e4567-e89b-12d3-a456-426614174000 is changed to 123e4567-e89b-12d3-a456-926614174000.

213798518

 • Apigee hybrid 1.7.0 and later
 FIXED IN Apigee hybrid 1.8.3 Deleting and recreating an environment with the same name can cause an error: "Instance x is not reporting status for this environment"

181569522

 • Apigee hybrid 1.7.0 and later
 FIXED IN Apigee hybrid 1.8.4 Deleting an environment when there are runtime instances catering to it can syncronization failure with the management plane

Before deleting an environment, make sure there is a wipeout plan for the runtime plane to make sure all the environment scoped data is cleared off from the runtime plane, such as caches, key value maps, and contract data.

181569522

 • Apigee hybrid 1.5.0 and later
 OPEN Deleting an environment and recreating it with the same name may lead to proxies not being deployed.

When creating new environments do not re-use deleted envrionment names. If you delete an environment and recreate it with the same environment name, proxies can fail to deploy to the new environment.
246774745 • Apigee
 OPEN Value of io.timeout.millis is not honored when used with multiple dynamic targets.

If a proxy sets two or more io.timeout.millis values in two or more flows using the same target host, only one io.timeout.millis value is honored.
245664917 • Apigee hybrid 1.8.x
 OPEN Apigee hybrid upgrade error can be ignored

During upgrade to Apigee hybrid 1.8.x, after running apigeectl init and confirming that check-ready succeeded, you may notice, if you view the pods, that the Cassandra schema validation job is in an error state. This is a harmless condition, and you can safely move to the next step in the upgrade procedure.
232946715 • Apigee hybrid 1.6.9
 FIXED IN Apigee hybrid 1.7.1 apigee-logger-apigee-telemetry DaemonSet is still running after turning off logger.
To fix this issue please [upgrade to Hybrid 1.7.x.](/apigee/docs/hybrid/v1.7/upgrade)
229639530 • Apigee hybrid 1.5.0 and later
 FIXED IN Apigee hybrid 1.7.1 When installing Apigee hybrid on OpenShift, under certain circumstances the runtime pod can become stuck in initialization mode.

On OpenShift, after running apigeectl init, the runtime pod can become stuck in initialization mode. To avoid this, try the following workaround:

  1. Edit the file plugins/open-shift/scc.yaml and insert the line 
    - system:serviceaccount:{{ .Namespace }}:default

    under the line system:serviceaccount:{{ .Namespace }}:{{ .Logger.PodServiceAccountName }} in the specifications for apigee-logger scc.

  2. Run apigeectl init again.
  3. Verify that the users section has both default and apigee-logger-apigee-telemetry services accounts with the following oc command: 
    oc get scc apigee-apigee-logger -o yaml

    The output should resemble:

    users: 
     - system:serviceaccount:apigee:default 
     - system:serviceaccount:apigee:apigee-logger-apigee-telemetry
225169066 • Apigee hybrid 1.5.9
• Apigee hybrid 1.5.8
• Apigee hybrid 1.5.7
 FIXED IN Apigee hybrid 1.5.10 Cassandra backup and restore not working when http_proxy is enabled.

Cassandra backup and restore will work when http_proxy is enabled. This has been fixed in Apigee hybrid v1.5.10
221305498 • Apigee
 OPEN API Monitoring may display fault code of '(not set)'.

API Monitoring of Configurable API Proxies may display a fault code of '(not set)' for responses with a non-2xx status from the target.
216018530 • Apigee hybrid 1.5.0 and later
 FIXED IN Apigee hybrid 1.7.1 Disabling the Hybrid logger by setting logger.enabled to false may not result in the deletion of existing logging agents.

The customer may need to also execute 

kubectl -n <namespace> delete ds apigee-logger-apigee-telemetry

where <namespace> is the namespace where Hybrid was installed, by default apigee.
209097822 • Apigee hybrid 1.5.0 and later
• Apigee
 OPEN Dynamic updates to rate in spike arrest may not reflect immediately

For a particular key, if there is continuous traffic, the key may not be rate limited at the updated rate. If there is five minutes of no traffic for a particular key, the rate will be reflected.

Workaround: Redeploy the proxy with a new reference variable if the rate has to take effect immediately. Or use two conditional spike arrests with different flow variables to adjust the rate.
207762842 • Apigee hybrid 1.5.0 and later
 OPEN Logs not shipped to Cloud Logging by apigee-logger.

Current apigee-logger configurations, including liveness probes, are incompatible with the Kubernetes runtime, resulting in logs not being shipped to Cloud Logging as expected. This problem also causes the apigee-logger pods to crash regularly. This issue affects Apigee hybrid installations on AKS, Anthos Bare Metal, and other platforms. Note that in some cases this issue results in excessive log volume.
207719377 • Apigee
 OPEN If there is more than one SpikeArrest policy in a bundle, 502 errors will occur.
Workaround: Avoid using more than one SpikeArrest policy in the proxy to prevent the issue.
205629443 • Apigee
 OPEN If ServiceCallout is fire and forget (no <Response> tag), a race condition can occur if there is another policy that occurs after it.

Workaround: To maintain the fire and forget behavior:

  1. Add <Response>calloutResponse</Response> to the ServiceCallout.
  2. Set continueOnError to true.
204943880 • Apigee hybrid 1.5.0 and later
• Apigee
 FIXED IN Apigee hybrid 1.5.6 SpikeArrest and Quota policies in a shared flow

If you are using the SpikeArrest policy in a shared flow, then the Identifier has the shared flow name appended, and the throttling limit is enforced for all the APIs that use the shared flow. Similarly, if you are using the Quota policy in a shared flow, the policy counters are updated for all the APIs that use the shared flow.
203827738 • Archive deployments
 OPEN Configurable API proxy without operations fails.

Proxies that do not contain operations, or contain operations without HTTP matches, will return a 404 error code if the request path does not contain exactly one segment in addition to the basepath. For example, requests to /basepath will fail, but requests to /basepath/user may succeed. To prevent failure, add an Operations section with at least one http_match to your configurable API proxy.
203778087 • Apigee hybrid 1.5.3 and later
 OPEN apigee-stackdriver-logging-agent currently runs as root.
Workaround: Disable the logging agent on hybrid.
201429104 • Apigee
 OPEN Wildcard in proxy basepath results in incorrect request path.

Usage of a wildcard (*) in the proxy basepath of a configurable API proxy results in forwarding incorrect request paths to the backend target.

To prevent incorrect request path forwarding, avoid using * in the configurable API proxy basepath until the issue is fixed.
200918549 • Apigee hybrid 1.6.0
 FIXED IN Apigee hybrid 1.6.1 There is a known issue with using forward proxy with the ApigeeConnect agent.
To work around this issue, use the 1.5.x version of the ApigeeConnect agent by setting the 1.5.x tag in the overrides.yaml file. For example: 
connectAgent:
  image:
    url: "gcr.io/apigee-release/hybrid/apigee-connect-agent"
    tag: "1.5.4"
.
191815997 • Apigee hybrid 1.6.0 and later
 OPEN If a hybrid customer configures a forward proxy for the API proxy, Google token will not work unless it has direct access to *.googleapis.com.
191745621 • Apigee hybrid 1.5.1
• Apigee hybrid 1.5.0
 FIXED IN Apigee hybrid 1.5.2 MART is not able to communicate with *.googleapis.com via a Forward Proxy.
191339147 • Apigee hybrid 1.5.0
 FIXED IN Apigee hybrid 1.5.1 Updated env-scoped JavaScript resource file fails to update in runtime instance.
191291501, 191000617 • Apigee
 OPEN Changing the email address of a developer entity will fail in the UI.
191002224 • Apigee hybrid 1.5.0 and later
 OPEN Changing an email address fails while using the PUT /organizations/{org_name}/developers/{developer_email} API.
184555974 • Apigee hybrid 1.5.0 and later
 OPEN The apigee-logger Fluentd can't parse logs in the OpenShift cluster.
178079779 • Apigee hybrid 1.4.0
 FIXED IN Apigee hybrid 1.4.1 UDCA prevents file uploads
UDCA encounters token generator issues with HTTP / HTTPS forward proxy, which prevent UDCA from uploading files.
175881688 • Apigee hybrid 1.4.0
 FIXED IN Apigee hybrid 1.4.1 Return codes
The Quota policy and the SpikeArrest policy return 500 instead of 429.
175771199 • Apigee hybrid 1.4.0
• Apigee hybrid 1.3.5
 FIXED IN Apigee hybrid 1.4.1 Generic DNS service endpoint
Prior to Apigee hybrid version 1.4.1 Cassandra used a node-specific DNS service instead of a generic DNS service endpoint.
174166751 • Apigee hybrid 1.3.3
 FIXED IN Apigee hybrid 1.3.4 Cassandra vertical scale-up using nodepools
Cassandra vertical scale-up using nodepools does not work with Apigee hybrid version 1.3.3.
172653617 • Apigee hybrid 1.3.6
• Apigee hybrid 1.3.5
• Apigee hybrid 1.3.4
• Apigee hybrid 1.3.3
 FIXED IN Apigee hybrid 1.4.0 API traffic interruption
When a new proxy revision is deployed there can be an interruption to the API traffic.
172332786 • Apigee hybrid 1.3.6
• Apigee hybrid 1.3.5
• Apigee hybrid 1.3.4
• Apigee hybrid 1.3.3
• Apigee hybrid 1.3.2
• Apigee hybrid 1.3.1
• Apigee hybrid 1.3.0
 FIXED IN Apigee hybrid 1.4.0 Unresolved request
Double slashes (//) in a request can cause the request not to resolve. You can solve this by applying a configuration to your Istio ingress that filters for double slashes. See Remove double slashes from requests for instructions.
162759110 • Apigee hybrid 1.3.0
 FIXED IN Apigee hybrid 1.3.1 Base path failure

Base paths consisting of only / will fail. You must include a path after the /.

For example:

  • /123
  • /v1/customers/123
146222881 • Apigee hybrid 1.3.0 and later
 OPEN Invalid HTTP Header error

Invalid HTTP Header error: The Istio ingress switches all incoming target responses to the HTTP2 protocol. Because the hybrid message processor only supports HTTP1, you may see the following error when an API proxy is called:

http2 error: Invalid HTTP header field was received: frame type: 1, stream: 1,

name: [:authority], value: [domain_name]

If you see this error, you can take either of the following actions to correct the problem:

  • Modify the target service to omit the Host header in the response.
  • Remove the Host header using the AssignMessage policy in your API proxy if necessary.
143659917 • Apigee hybrid 1.3.1
• Apigee hybrid 1.3.0
 FIXED IN Apigee hybrid 1.3.2 PopulateCache policy's expiration setting

The PopulateCache policy's expiration setting must be set to an explicit value between 1 and 30. For example:

<ExpirySettings>

  <TimeoutInSec>30</TimeoutInSec>

</ExpirySettings>