API calls fail with timeout errors

You're viewing Apigee and Apigee hybrid documentation.
View Apigee Edge documentation.

Symptom

You may observe one of the following symptoms:

The client applications get timeout errors as a response for API calls on Apigee hybrid.
You observe errors such as Error from server (invalid) or The Job "apigee-resources-install" is invalid while applying configuration (overrides.yaml) to the cluster during hybrid installation.

Error messages

You may observe one of the following errors:

Error response to API calls

The API requests on Apigee hybrid may fail with the following error message:

* Connection failed
* connect to 34.84.67.39 port 443 failed: Operation timed out
* Failed to connect to example.apis.com port 443: Operation timed out
* Closing connection 0
curl: (7) Failed to connect to example.apis.com port 443: Operation timed out

Errors observed while applying configuration (overrides.yaml) to clusters

You may observe one of the following errors while applying configuration (overrides.yaml file) to clusters during the installation:

Error #1

helm upgrade operator apigee-operator/ \
  --install \
  --create-namespace \
  --namespace APIGEE_NAMESPACE \
  --atomic \
  -f OVERRIDES_FILE

...
...
Error from server (Invalid): error when applying patch:
to:
Resource: "batch/v1, Resource=jobs", GroupVersionKind: "batch/v1, Kind=Job"
Name: "istio-init-crd-10-1.4.6", Namespace: "istio-system"
to:
Resource: "batch/v1, Resource=jobs", GroupVersionKind: "batch/v1, Kind=Job"
Name: "istio-init-crd-11-1.4.6", Namespace: "istio-system"
to:
Resource: "batch/v1, Resource=jobs", GroupVersionKind: "batch/v1, Kind=Job"
Name: "istio-init-crd-14-1.4.6", Namespace: "istio-system"

Error #2

helm upgrade operator apigee-operator/ \
--install \
--create-namespace \
--namespace APIGEE_NAMESPACE \
--atomic \
-f OVERRIDES_FILE

...
...
The Job "apigee-resources-install" is invalid: spec.template: Invalid value:
core.PodTemplateSpec{ObjectMeta:v1.ObjectMeta{Name:"apigee-resources-install",
GenerateName:"", Namespace:"", SelfLink:"", UID:"", ResourceVersion:"",
Generation:0,

Possible causes

These errors can happen if the istio-ingressgateway service is in a pending state and unable to bind to an external IP address as shown below:

kubectl get services -n istio-system
NAME                      TYPE         CLUSTER-IP   EXTERNAL-IP  PORT(S)             AGE
istio-ingressgateway      LoadBalancer 10.198.5.104 <pending>    15020:31927/TCP,    12h
                                                                 80:31381/TCP,
                                                                 443:31391/TCP,
                                                                 31400:31401/TCP,
                                                                 15443:32623/TCP

The possible causes for the istio-ingressgateway service to be in a pending state are as follows:

Cause	Description
Jobs in erroneous/pending state in istio-system namespace	The `incomplete/erroneous` jobs in the `istio-system` namespace could cause the `istio-ingressgateway` service to be in `pending` state forever and unable to bind to an external IP address.
apigee-resources-install job in erroneous/pending state in apigee-system namespace	The `incomplete` jobs in the `apigee-system` namespace could cause the `istio- ingressgateway` service to be in `pending` state forever and unable to bind to an external IP address.
Incorrect IP address range assigned to external load balancer	An incorrect IP address range may be configured in the `istio-operator.yaml` file causing the `istio-ingressgateway` service to get into `pending` state forever and unable to bind to an external IP address during the installation.

Cause: Jobs in istio-system namespace in erroneous/pending state

Diagnosis

Check the status of the jobs in the istio-system namespace using the following command:
```
kubectl get jobs -n istio-system
```
The status of the jobs must be complete. If the status of the jobs is in an erroneous/pending state, then that's the cause of this problem.

Resolution

If any of the jobs are in the pending or erroneous state, delete them using the following command:
```
kubectl -n istio-system delete job JOB_NAME_FROM_STEP_1
```

Re-run the installation by applying the overrides.yaml file:

Update the apigee-serving-cert using Helm:

helm install operator apigee-operator/
--namespace APIGEE_NAMESPACE \
--atomic \
-f OVERRIDES_FILE \
--dry-run=server

Make sure to include all of the settings shown, including --atomic so that the action rolls back on failure.

Install the chart:

helm upgrade operator apigee-operator/
--namespace APIGEE_NAMESPACE \
--atomic \
-f OVERRIDES_FILE

Cause: apigee-resources-install job in the apigee-system namespace may be in erroneous state

Note: The apigee-resources-install job is not part of hybrid versions 1.13 and later.

Diagnosis

Check the status of the jobs in the apigee-system namespace using the following command:
```
kubectl get jobs -n apigee-system
```
The status of the jobs must be complete. If the status of the jobs is in an erroneous/pending state, then that's the cause of this problem. The following sample output shows that the job apigee-resources-install is successfully completed.
```
kubectl get jobs  -n apigee-system
NAME                       COMPLETIONS   DURATION   AGE
apigee-resources-install   1/1           23s        16d
```

Resolution

If the jobs are in the pending or erroneous state, delete them using the following command:
```
kubectl -n apigee-system delete job JOB_NAME_FROM_STEP_1
```
Re-run the installation by applying the overrides.yaml file:
```
apigeectl apply -f overrides.yaml
```

Cause: Incorrect IP address range assigned to external load balancer

Diagnosis

Check the IP address configured for the load balancer in the istio- operator.yaml file. For example, the following snippet shows the location in the istio-operator.yaml file where the IP address is configured:
```
-name: istio-ingressgateway
  enabled: true
  k8s:
    service:
      type: LoadBalancer
     loadBalancerIP: 10.195.24.23
```
The istio-ingressgateway service is configured as a load balancer (indicated by type) in the istio-operator.yaml file. During the ASM installation, a load balancer is created with the configured IP address and wired to communicate with the istio- ingressgateway service. Therefore, the IP address configured should be correct and reserved for the load balancer.
Engage your network team and verify that the IP address configured for loadBalancerIP is correct. If it is incorrect, then the load balancer service will not be able to bind to the IP address. This causes the istio-ingressgateway service to be in the pending state forever.

Resolution

Work with your network team and configure the correct IP address in the istio- operator.yaml file.

Re-run the installation for the Apigee ingress gateway and apply the overrides.yaml file:

helm upgrade $ORG_NAME apigee-org/ \
  --install \
  --namespace APIGEE_NAMESPACE \
  --atomic \
  -f OVERRIDES_FILE

Must gather diagnostic information

If the problem persists even after following the above instructions, gather the following diagnostic information and then contact Google Cloud Customer Care:

The Google Cloud Project ID
The name of the Apigee hybrid organization
Kubernetes Cluster name
Google Cloud project name if kubernetes cluster resides in different Google Cloud project
The overrides.yaml file
The Istio-operator .yaml file used during the ASM installation.

Collect the logs from each istio-ingressgateway pod in the istio-system namespace:

kubectl logs NAME_OF_ISTIO_INGRESSGATEWAY_POD -n istio-system > /tmp/NAME_OF_ISTIO_INGRESSGATEWAY_POD.log

Collect the description of the each pod in the istio-system namespace:

kubectl describe pod NAME_OF_ISTIO_INGRESSGATEWAY_POD -n istio-system > /tmp/NAME_OF_ISTIO_INGRESSGATEWAY_POD.yaml

Collect the list of services in the istio-system namespace:
```
kubectl get svc -n istio-system
```