The client applications get timeout errors as a response for API calls on Apigee hybrid.
You observe errors such as Error from server (invalid) or The Job
"apigee-resources-install" is invalid while applying configuration
(overrides.yaml) to the cluster during hybrid installation.
Error messages
You may observe one of the following errors:
Error response to API calls
The API requests on Apigee hybrid may fail with the following error message:
* Connection failed
* connect to 34.84.67.39 port 443 failed: Operation timed out
* Failed to connect to example.apis.com port 443: Operation timed out
* Closing connection 0
curl: (7) Failed to connect to example.apis.com port 443: Operation timed out
Errors observed while applying configuration (overrides.yaml) to clusters
You may observe one of the following errors while applying configuration
(overrides.yaml file) to clusters during the installation:
The incomplete/erroneous jobs in the istio-system namespace could
cause the istio-ingressgateway service to be in pending state forever
and unable to bind to an external IP address.
The incomplete jobs in the apigee-system namespace could cause the
istio- ingressgateway service to be in pending state forever and unable
to bind to an external IP address.
An incorrect IP address range may be configured in the istio-operator.yaml file
causing the istio-ingressgateway service to get into pending state
forever and unable to bind to an external IP address during the installation.
Cause: Jobs in istio-system namespace in erroneous/pending state
Diagnosis
Check the status of the jobs in the istio-system namespace using the following
command:
kubectl get jobs -n istio-system
The status of the jobs must be complete. If the status of the jobs is
in an erroneous/pending state, then that's the cause of this problem.
Resolution
If any of the jobs are in the pending or erroneous state, delete
them using the following command:
Cause: apigee-resources-install job in the apigee-system namespace may be in erroneous state
Diagnosis
Check the status of the jobs in the apigee-system namespace using the
following command:
kubectl get jobs -n apigee-system
The status of the jobs must be complete. If the status of the jobs is in an
erroneous/pending state, then that's the cause of this problem. The following
sample output shows that the job apigee-resources-install is successfully
completed.
kubectl get jobs -n apigee-system
NAME COMPLETIONS DURATION AGE
apigee-resources-install 1/1 23s 16d
Resolution
If the jobs are in the pending or erroneous state, delete them
using the following command:
Re-run the installation by applying the overrides.yaml file:
apigeectl apply -f overrides.yaml
Cause: Incorrect IP address range assigned to external load balancer
Diagnosis
Check the IP address configured for the load balancer in the istio-
operator.yaml file. For example, the following snippet shows the location in the
istio-operator.yaml file where the IP address is configured:
The istio-ingressgateway service is configured as a load balancer (indicated by
type) in the istio-operator.yaml file. During the ASM installation, a load balancer
is created with the configured IP address and wired to communicate with the istio-
ingressgateway service. Therefore, the IP address configured should be correct and
reserved for the load balancer.
Engage your network team and verify that the IP address configured for
loadBalancerIP is correct. If it is incorrect, then the load balancer service will
not be able to bind to the IP address. This causes the istio-ingressgateway service
to be in the pending state forever.
Resolution
Work with your network team and configure the correct IP address in the istio-
operator.yaml file.
If the problem persists even after following the above instructions, gather the following
diagnostic information and then contact Google Cloud Customer Care:
The Google Cloud Project ID
The name of the Apigee hybrid organization
Kubernetes Cluster name
Google Cloud project name if kubernetes cluster resides in different Google Cloud project
The overrides.yaml file
The Istio-operator .yaml file used during the ASM installation.
Collect the logs from each istio-ingressgateway pod in the
istio-system namespace:
[[["Easy to understand","easyToUnderstand","thumb-up"],["Solved my problem","solvedMyProblem","thumb-up"],["Other","otherUp","thumb-up"]],[["Hard to understand","hardToUnderstand","thumb-down"],["Incorrect information or sample code","incorrectInformationOrSampleCode","thumb-down"],["Missing the information/samples I need","missingTheInformationSamplesINeed","thumb-down"],["Other","otherDown","thumb-down"]],["Last updated 2025-08-26 UTC."],[[["\u003cp\u003eThis document addresses issues where Apigee hybrid API calls time out or configuration applications to the cluster fail.\u003c/p\u003e\n"],["\u003cp\u003eThe primary symptoms of these issues include timeout errors for client applications and specific errors during the application of \u003ccode\u003eoverrides.yaml\u003c/code\u003e.\u003c/p\u003e\n"],["\u003cp\u003eThe root cause of these symptoms is often the \u003ccode\u003eistio-ingressgateway\u003c/code\u003e service being in a \u003ccode\u003epending\u003c/code\u003e state and unable to bind to an external IP address.\u003c/p\u003e\n"],["\u003cp\u003eSolutions involve checking and resolving incomplete or erroneous jobs in the \u003ccode\u003eistio-system\u003c/code\u003e or \u003ccode\u003eapigee-system\u003c/code\u003e namespaces and ensuring the correct IP address range is configured for the external load balancer.\u003c/p\u003e\n"],["\u003cp\u003eIf the problem persists, diagnostic information such as logs, service lists, project IDs, and configuration files should be collected for Google Cloud Customer Care assistance.\u003c/p\u003e\n"]]],[],null,["# API calls fail with timeout errors\n\n*You're viewing **Apigee** and **Apigee hybrid** documentation.\nView [Apigee Edge](https://docs.apigee.com/api-platform/troubleshoot/404-support-d) documentation.*\n| **Note:** This document is applicable for Apigee hybrid users only.\n\nSymptom\n-------\n\n\nYou may observe one of the following symptoms:\n\n1. The client applications get timeout errors as a response for API calls on Apigee hybrid.\n2. You observe errors such as `Error from server (invalid)` or `The Job\n \"apigee-resources-install\" is invalid` while applying configuration (`overrides.yaml`) to the cluster during hybrid installation.\n\nError messages\n--------------\n\n\nYou may observe one of the following errors:\n\n### Error response to API calls\n\n\nThe API requests on Apigee hybrid may fail with the following error message: \n\n```scdoc\n* Connection failed\n* connect to 34.84.67.39 port 443 failed: Operation timed out\n* Failed to connect to example.apis.com port 443: Operation timed out\n* Closing connection 0\ncurl: (7) Failed to connect to example.apis.com port 443: Operation timed out\n```\n\n### Errors observed while applying configuration (overrides.yaml) to clusters\n\n\nYou may observe one of the following errors while applying configuration\n(`overrides.yaml` file) to clusters during the installation: \n\n### Error #1\n\n```\nhelm upgrade operator apigee-operator/ \\\n --install \\\n --create-namespace \\\n --namespace APIGEE_NAMESPACE \\\n --atomic \\\n -f OVERRIDES_FILE\n```\n\n\u003cbr /\u003e\n\n```text\n...\n...\nError from server (Invalid): error when applying patch:\nto:\nResource: \"batch/v1, Resource=jobs\", GroupVersionKind: \"batch/v1, Kind=Job\"\nName: \"istio-init-crd-10-1.4.6\", Namespace: \"istio-system\"\nto:\nResource: \"batch/v1, Resource=jobs\", GroupVersionKind: \"batch/v1, Kind=Job\"\nName: \"istio-init-crd-11-1.4.6\", Namespace: \"istio-system\"\nto:\nResource: \"batch/v1, Resource=jobs\", GroupVersionKind: \"batch/v1, Kind=Job\"\nName: \"istio-init-crd-14-1.4.6\", Namespace: \"istio-system\"\n```\n\n### Error #2\n\n```\nhelm upgrade operator apigee-operator/ \\\n--install \\\n--create-namespace \\\n--namespace APIGEE_NAMESPACE \\\n--atomic \\\n-f OVERRIDES_FILE\n```\n\n\u003cbr /\u003e\n\n```carbon\n...\n...\nThe Job \"apigee-resources-install\" is invalid: spec.template: Invalid value:\ncore.PodTemplateSpec{ObjectMeta:v1.ObjectMeta{Name:\"apigee-resources-install\",\nGenerateName:\"\", Namespace:\"\", SelfLink:\"\", UID:\"\", ResourceVersion:\"\",\nGeneration:0,\n```\n\nPossible causes\n---------------\n\n\nThese errors can happen if the `istio-ingressgateway` service is in a\n`pending` state and unable to bind to an external IP address as shown below: \n\n```\nkubectl get services -n istio-system\nNAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE\nistio-ingressgateway LoadBalancer 10.198.5.104 \u003cpending\u003e 15020:31927/TCP, 12h\n 80:31381/TCP,\n 443:31391/TCP,\n 31400:31401/TCP,\n 15443:32623/TCP\n```\n\n\nThe possible causes for the `istio-ingressgateway` service to be in a\n`pending` state are as follows:\n\nCause: Jobs in istio-system namespace in erroneous/pending state\n----------------------------------------------------------------\n\n### Diagnosis\n\n1. Check the status of the jobs in the `istio-system` namespace using the following command: \n\n ```\n kubectl get jobs -n istio-system\n ```\n2. The status of the jobs must be `complete`. If the status of the jobs is in an `erroneous/pending` state, then that's the cause of this problem.\n\n### Resolution\n\n1. If any of the jobs are in the `pending` or `erroneous` state, delete them using the following command: \n\n ```\n kubectl -n istio-system delete job JOB_NAME_FROM_STEP_1\n ```\n2. Re-run the installation by applying the `overrides.yaml` file:\n\n\n Update the `apigee-serving-cert` using\n [Helm](/apigee/docs/hybrid/latest/helm-reference): \n\n ```\n helm install operator apigee-operator/\n --namespace APIGEE_NAMESPACE \\\n --atomic \\\n -f OVERRIDES_FILE \\\n --dry-run=server\n ```\n\n Make sure to include all of the settings shown, including `--atomic`\n so that the action rolls back on failure.\n\n Install the chart: \n\n ```\n helm upgrade operator apigee-operator/\n --namespace APIGEE_NAMESPACE \\\n --atomic \\\n -f OVERRIDES_FILE\n ```\n\nCause: apigee-resources-install job in the apigee-system namespace may be in erroneous state\n--------------------------------------------------------------------------------------------\n\n| **Note:** The `apigee-resources-install` job is not part of hybrid versions 1.13 and later.\n\n### Diagnosis\n\n1. Check the status of the jobs in the `apigee-system` namespace using the following command: \n\n ```\n kubectl get jobs -n apigee-system\n ```\n2. The status of the jobs must be `complete`. If the status of the jobs is in an `erroneous/pending` state, then that's the cause of this problem. The following sample output shows that the job `apigee-resources-install `is successfully completed. \n\n ```\n kubectl get jobs -n apigee-system\n NAME COMPLETIONS DURATION AGE\n apigee-resources-install 1/1 23s 16d\n ```\n\n### Resolution\n\n1. If the jobs are in the `pending` or `erroneous` state, delete them using the following command: \n\n ```\n kubectl -n apigee-system delete job JOB_NAME_FROM_STEP_1\n ```\n2. Re-run the installation by applying the `overrides.yaml` file: \n\n ```\n apigeectl apply -f overrides.yaml\n ```\n\nCause: Incorrect IP address range assigned to external load balancer\n--------------------------------------------------------------------\n\n### Diagnosis\n\n1. Check the IP address configured for the load balancer in the `istio-\n operator.yaml` file. For example, the following snippet shows the location in the `istio-operator.yaml` file where the IP address is configured: \n\n ```gdscript\n -name: istio-ingressgateway\n enabled: true\n k8s:\n service:\n type: LoadBalancer\n loadBalancerIP: 10.195.24.23\n ```\n2. The `istio-ingressgateway` service is configured as a load balancer (indicated by type) in the `istio-operator.yaml` file. During the ASM installation, a load balancer is created with the configured IP address and wired to communicate with the `istio-\n ingressgateway` service. Therefore, the IP address configured should be correct and reserved for the load balancer.\n3. Engage your network team and verify that the IP address configured for `loadBalancerIP` is correct. If it is incorrect, then the load balancer service will not be able to bind to the IP address. This causes the `istio-ingressgateway` service to be in the `pending` state forever.\n\n### Resolution\n\n1. Work with your network team and configure the correct IP address in the `istio-\n operator.yaml` file.\n2. Re-run the installation for the [Apigee ingress gateway](/apigee/docs/hybrid/latest/managing-ingress) and apply the `overrides.yaml` file: \n\n ```\n helm upgrade $ORG_NAME apigee-org/ \\\n --install \\\n --namespace APIGEE_NAMESPACE \\\n --atomic \\\n -f OVERRIDES_FILE\n ```\n\nMust gather diagnostic information\n----------------------------------\n\n\nIf the problem persists even after following the above instructions, gather the following\ndiagnostic information and then contact [Google Cloud Customer Care](https://cloud.google.com/support-hub/):\n\n1. The Google Cloud Project ID\n2. The name of the Apigee hybrid organization\n3. Kubernetes Cluster name\n4. Google Cloud project name if kubernetes cluster resides in different Google Cloud project\n5. The `overrides.yaml` file\n6. The `Istio-operator .yaml` file used during the ASM installation.\n7. Collect the logs from each `istio-ingressgateway` pod in the `istio-system` namespace: \n\n ```\n kubectl logs NAME_OF_ISTIO_INGRESSGATEWAY_POD -n istio-system \u003e /tmp/NAME_OF_ISTIO_INGRESSGATEWAY_POD.log\n ```\n8. Collect the description of the each pod in the `istio-system` namespace: \n\n ```\n kubectl describe pod NAME_OF_ISTIO_INGRESSGATEWAY_POD -n istio-system \u003e /tmp/NAME_OF_ISTIO_INGRESSGATEWAY_POD.yaml\n ```\n9. Collect the list of services in the `istio-system` namespace: \n\n ```\n kubectl get svc -n istio-system\n ```"]]
