API Calls failing with TLS errors

You're viewing Apigee and Apigee hybrid documentation.
View Apigee Edge documentation.

Symptom

You may observe one or more of the following symptoms on Apigee hybrid:

  1. The runtime traffic stopped working
  2. The API requests fail with TLS connection errors
  3. The API clients don't receive the expected TLS certificate from the istio-ingressgateway

Error messages

You will observe that the API calls will fail with TLS errors similar to the error shown below: 

* TLSv1.2 (OUT), TLS handshake, Client hello (1):
* LibreSSL SSL_connect: SSL_ERROR_SYSCALL in connection to example.apis.com:443
* Closing connection 0
curl: (35) LibreSSL SSL_connect: SSL_ERROR_SYSCALL in connection to example.apis.com:443

Possible causes

Cause Description
Missing K8S secret in Ingress K8S secret for the org-envgroup is missing from the istio-system namespace

K8S secrets

In Apigee hybrid, the virtual hosts for ingress traffic are configured using the environment groups. The TLS certificate and key files are configured for each environment group in the overrides.yaml file.

Sample TLS certificate and key files configuration in overrides.yaml:

org: my-hybrid-project
virtualhosts:
  - name: uat-group
    sslCertPath: ./certs/uat-group-keystore.pem
    sslKeyPath: ./certs/uat-group-keystore.key
  - name: prod-group
    sslCertPath: ./certs/prod-group-keystore.pem
    sslKeyPath: ./certs/prod-group-keystore.key

During the installation of Apigee hybrid runtime, these TLS keys and certificates are stored in the K8S secrets in the istio-system namespace.

For example, the above configuration in the overrides.yaml file will create the following K8S secrets: 

kubectl -n istio-system get secrets


NAME                            TYPE      DATA   AGE
my-hybrid-project-prod-group    Opaque    2      15h
my-hybrid-project-uat-group     Opaque    2      38m

Since these K8S secrets store the required TLS keys and certificates, it is important to ensure they are kept intact. However, these secrets could be deleted in the following situations:

  1. Re-installation of ASM
  2. Deletion of the istio-system namespace
  3. Inadvertent deletion

Cause: Missing secret in Ingress

Diagnosis

  1. Analyze the logs of istio-ingressgateway pods in the istio-system namespace and see if there are any errors.
  2. You may observe an error such as the following: 
    2021-01-04T23:37:14.542520Z     error   secretfetcher   cannot find secret MY_HYBRID_PROJECT-UAT_GROUP and cannot find fallback secret gateway-fallback

    Where:

    MY_HYBRID_PROJECT is the name of the Apigee Hybrid organization

    UAT_GROUP is the name of the environment group

    The above error indicates that secretfetcher could not find the K8S secret for the above-mentioned environment group of the Apigee hybrid organization.

  3. Verify that the K8S secrets are indeed missing by using the following command: 
    kubectl -n istio-system get secrets
<no output>

    The above example output shows that there are no K8S secrets. They must have been deleted due to one of the situations mentioned in the K8S secrets section above.

Resolution

You can recreate the missing K8S secrets using the TLS Certificates and key files information from the overrides.yaml file using the following steps:

  1. Run the following command to recreate the missing secrets: 
    apigeectl apply -f overrides/overrides.yaml --settings virtualhosts
  2. The output of the above command should show that the secret is created: 
    my-hybrid-project-prod-group created
  3. Verify that the K8S secrets are successfully created using the following command: 
    kubectl -n istio-system get secrets


    NAME                            TYPE      DATA   AGE
my-hybrid-project-prod-group    Opaque    2      15h
my-hybrid-project-uat-group     Opaque    2      38m

Must gather diagnostic information

If the problem persists even after following the above instructions, gather the following diagnostic information and then contact Apigee Support:

  1. The Google Cloud Project ID
  2. The name of the Apigee hybrid organization
  3. Kubernetes Cluster name
  4. Google Cloud project name if kubernetes cluster resides in different Google Cloud project
  5. The overrides.yaml file
  6. Collect the logs from each istio-ingressgateway pod in the istio-system namespace: 
    kubectl logs NAME_OF_ISTIO_INGRESSGATEWAY_POD -n istio-system > /tmp/NAME_OF_ISTIO_INGRESSGATEWAY_POD.log
  7. Collect the list of the secrets in the istio-system namespace: 
    kubectl -n istio-system get secrets
  8. Collect the list of services in the istio-system namespace: 
    kubectl get svc -n istio-system
  9. Collect the description of the each pod in the istio-system namespace: 
    kubectl describe pod NAME_OF_ISTIO_INGRESSGATEWAY_POD -n istio-system > /tmp/NAME_OF_ISTIO_INGRESSGATEWAY_POD.yaml