You're viewing Apigee and Apigee hybrid documentation.
View
Apigee Edge documentation.
Symptom
You may observe one or more of the following symptoms on Apigee hybrid:
- The runtime traffic stopped working
- The API requests fail with TLS connection errors
- The API clients don't receive the expected TLS certificate from the
istio-ingressgateway
Error messages
You will observe that the API calls will fail with TLS errors similar to the error shown below:
* TLSv1.2 (OUT), TLS handshake, Client hello (1): * LibreSSL SSL_connect: SSL_ERROR_SYSCALL in connection to example.apis.com:443 * Closing connection 0 curl: (35) LibreSSL SSL_connect: SSL_ERROR_SYSCALL in connection to example.apis.com:443
Possible causes
Cause | Description |
---|---|
Missing K8S secret in Ingress | K8S secret for the org-envgroup is missing from the
istio-system namespace
|
K8S secrets
In Apigee hybrid, the virtual hosts for ingress
traffic are configured using the
environment groups. The TLS certificate and key files are configured for each environment group
in the overrides.yaml
file.
Sample TLS certificate and key files configuration in overrides.yaml
:
org: my-hybrid-project virtualhosts: - name: uat-group sslCertPath: ./certs/uat-group-keystore.pem sslKeyPath: ./certs/uat-group-keystore.key - name: prod-group sslCertPath: ./certs/prod-group-keystore.pem sslKeyPath: ./certs/prod-group-keystore.key
During the installation of Apigee hybrid runtime, these TLS keys and certificates are stored in
the K8S secrets in the istio-system
namespace.
For example, the above configuration in the overrides.yaml
file will create the
following K8S secrets:
kubectl -n istio-system get secrets
NAME TYPE DATA AGE my-hybrid-project-prod-group Opaque 2 15h my-hybrid-project-uat-group Opaque 2 38m
Since these K8S secrets store the required TLS keys and certificates, it is important to ensure they are kept intact. However, these secrets could be deleted in the following situations:
- Re-installation of ASM
- Deletion of the
istio-system
namespace - Inadvertent deletion
Cause: Missing secret in Ingress
Diagnosis
- Analyze the logs of
istio-ingressgateway
pods in theistio-system
namespace and see if there are any errors. - You may observe an error such as the following:
2021-01-04T23:37:14.542520Z error secretfetcher cannot find secret MY_HYBRID_PROJECT-UAT_GROUP and cannot find fallback secret gateway-fallback
Where:
MY_HYBRID_PROJECT is the name of the Apigee Hybrid organization
UAT_GROUP is the name of the environment group
The above error indicates that
secretfetcher
could not find the K8S secret for the above-mentioned environment group of the Apigee hybrid organization. - Verify that the K8S secrets are indeed missing by using the following command:
kubectl -n istio-system get secrets <no output>
The above example output shows that there are no K8S secrets. They must have been deleted due to one of the situations mentioned in the K8S secrets section above.
Resolution
You can recreate the missing K8S secrets using the TLS Certificates and key files information
from the overrides.yaml
file using the following steps:
- Run the following command to recreate the missing secrets:
apigeectl apply -f overrides/overrides.yaml --settings virtualhosts
- The output of the above command should show that the secret is created:
my-hybrid-project-prod-group created
- Verify that the K8S secrets are successfully created using the following command:
kubectl -n istio-system get secrets
NAME TYPE DATA AGE my-hybrid-project-prod-group Opaque 2 15h my-hybrid-project-uat-group Opaque 2 38m
Must gather diagnostic information
If the problem persists even after following the above instructions, gather the following diagnostic information and then contact Apigee Support:
- The Google Cloud Project ID
- The name of the Apigee hybrid organization
- Kubernetes Cluster name
- Google Cloud project name if kubernetes cluster resides in different Google Cloud project
- The
overrides.yaml
file - Collect the logs from each
istio-ingressgateway
pod in theistio-system
namespace:kubectl logs NAME_OF_ISTIO_INGRESSGATEWAY_POD -n istio-system > /tmp/NAME_OF_ISTIO_INGRESSGATEWAY_POD.log
- Collect the list of the secrets in the
istio-system
namespace:kubectl -n istio-system get secrets
- Collect the list of services in the
istio-system
namespace:kubectl get svc -n istio-system
- Collect the description of the each pod in the
istio-system
namespace:kubectl describe pod NAME_OF_ISTIO_INGRESSGATEWAY_POD -n istio-system > /tmp/NAME_OF_ISTIO_INGRESSGATEWAY_POD.yaml