API Calls failing with TLS errors

You're viewing Apigee and Apigee hybrid documentation.
View Apigee Edge documentation.

Symptom

You may observe one or more of the following symptoms on Apigee hybrid:

The runtime traffic stopped working
The API requests fail with TLS connection errors
The API clients don't receive the expected TLS certificate from the istio-ingressgateway

Error messages

You will observe that the API calls will fail with TLS errors similar to the error shown below:

* TLSv1.2 (OUT), TLS handshake, Client hello (1):
* LibreSSL SSL_connect: SSL_ERROR_SYSCALL in connection to example.apis.com:443
* Closing connection 0
curl: (35) LibreSSL SSL_connect: SSL_ERROR_SYSCALL in connection to example.apis.com:443

Note: The above sample output is from a curl command.

Possible causes

Cause	Description
Missing K8S secret in Ingress	K8S secret for the `org-envgroup` is missing from the `istio-system` namespace

K8S secrets

In Apigee hybrid, the virtual hosts for ingress traffic are configured using the environment groups. The TLS certificate and key files are configured for each environment group in the overrides.yaml file.

Sample TLS certificate and key files configuration in overrides.yaml:

org: my-hybrid-project
virtualhosts:
  - name: uat-group
    sslCertPath: ./certs/uat-group-keystore.pem
    sslKeyPath: ./certs/uat-group-keystore.key
  - name: prod-group
    sslCertPath: ./certs/prod-group-keystore.pem
    sslKeyPath: ./certs/prod-group-keystore.key

During the installation of Apigee hybrid runtime, these TLS keys and certificates are stored in the K8S secrets in the istio-system namespace.

For example, the above configuration in the overrides.yaml file will create the following K8S secrets:

kubectl -n istio-system get secrets

NAME                            TYPE      DATA   AGE
my-hybrid-project-prod-group    Opaque    2      15h
my-hybrid-project-uat-group     Opaque    2      38m

Since these K8S secrets store the required TLS keys and certificates, it is important to ensure they are kept intact. However, these secrets could be deleted in the following situations:

Re-installation of ASM
Deletion of the istio-system namespace
Inadvertent deletion

Cause: Missing secret in Ingress

Diagnosis

Analyze the logs of istio-ingressgateway pods in the istio-system namespace and see if there are any errors.
You may observe an error such as the following:
```
2021-01-04T23:37:14.542520Z     error   secretfetcher   cannot find secret MY_HYBRID_PROJECT-UAT_GROUP and cannot find fallback secret gateway-fallback
```
Where:

MY_HYBRID_PROJECT is the name of the Apigee Hybrid organization

UAT_GROUP is the name of the environment group

The above error indicates that secretfetcher could not find the K8S secret for the above-mentioned environment group of the Apigee hybrid organization.
Verify that the K8S secrets are indeed missing by using the following command:
```
kubectl -n istio-system get secrets
<no output>
```
The above example output shows that there are no K8S secrets. They must have been deleted due to one of the situations mentioned in the K8S secrets section above.

Resolution

You can recreate the missing K8S secrets using the TLS Certificates and key files information from the overrides.yaml file using the following steps:

Run the following command to recreate the missing secrets:

apigeectl apply -f overrides/overrides.yaml --settings virtualhosts

The output of the above command should show that the secret is created:
```
my-hybrid-project-prod-group created
```

Verify that the K8S secrets are successfully created using the following command:

kubectl -n istio-system get secrets

NAME                            TYPE      DATA   AGE
my-hybrid-project-prod-group    Opaque    2      15h
my-hybrid-project-uat-group     Opaque    2      38m

Must gather diagnostic information

If the problem persists even after following the above instructions, gather the following diagnostic information and then contact Apigee Support:

The Google Cloud Project ID
The name of the Apigee hybrid organization
Kubernetes Cluster name
Google Cloud project name if kubernetes cluster resides in different Google Cloud project
The overrides.yaml file

Collect the logs from each istio-ingressgateway pod in the istio-system namespace:

kubectl logs NAME_OF_ISTIO_INGRESSGATEWAY_POD -n istio-system > /tmp/NAME_OF_ISTIO_INGRESSGATEWAY_POD.log

Collect the list of the secrets in the istio-system namespace:
```
kubectl -n istio-system get secrets
```
Collect the list of services in the istio-system namespace:
```
kubectl get svc -n istio-system
```

Collect the description of the each pod in the istio-system namespace:

kubectl describe pod NAME_OF_ISTIO_INGRESSGATEWAY_POD -n istio-system > /tmp/NAME_OF_ISTIO_INGRESSGATEWAY_POD.yaml