You're viewing Apigee and Apigee hybrid documentation.
View
Apigee Edge documentation.
Symptom
Apigee hybrid API requests fail with TLS connection errors. These errors are usually connection resets and alert handshake failures.
Error messages
API calls fail with TLS similar to the error shown below, or other but not identical errors:
* TLSv1.2 (OUT), TLS handshake, Client hello (1): * LibreSSL SSL_connect: SSL_ERROR_SYSCALL in connection to example.apis.com:443 * Closing connection 0 curl: (35) LibreSSL SSL_connect: SSL_ERROR_SYSCALL in connection to example.apis.com:443
* (304) (OUT), TLS handshake, Client hello (1): * Recv failure: Connection reset by peer * LibreSSL/3.3.6: error:02FFF036:system library:func(4095):Connection reset by peer * Closing connection curl: (35) Recv failure: Connection reset by peer
* (304) (OUT), TLS handshake, Client hello (1): * LibreSSL/3.3.6: error:1404B410:SSL routines:ST_CONNECT:sslv3 alert handshake failure * Closing connection curl: (35) LibreSSL/3.3.6: error:1404B410:SSL routines:ST_CONNECT:sslv3 alert handshake failure
Possible causes
Cause | Description | Troubleshooting instructions applicable for |
---|---|---|
Missing Ingress Kubernetes secret | The Kubernetes secret for the org-envgroup is missing from the Apigee
namespace. |
Apigee hybrid |
Incorrectly formatted SSL certificate | The SSL certificate which is pointed in the virtualhosts section of overrides file is incorrectly formatted. | Apigee hybrid |
The SSL keys do not match the SSL certificate | The SSL keys do not match the SSL certificate which is pointed in the virtualhosts section of the overrides file. | Apigee hybrid |
Cause: Missing Ingress Kubernetes secret
Diagnosis
- Analyze the logs of
apigee-watcher
pods in the apigee namespace and see if there are any errors.
You may observe an error such as the following:
NOT_FOUND: failed to get secret "MY_HYBRID_PROJECT-ENV_GROUP" in namespace "apigee": secrets "MY_HYBRID_PROJECT-ENV_GROUP" not found
where:MY_HYBRID_PROJECT
is the name of the Apigee hybrid organizationENV_GROUP
is the name of the environment group
apigee-watcher
could not find the Kubernetes secret for the above-mentioned environment group of the Apigee hybrid organization. - Verify that the Kubernetes secrets are indeed missing by using the following command:
kubectl -n apigee get secrets | grep MY_HYBRID_PROJECT-ENV_GROUP
<no output>
This example shows no Kubernetes secrets forMY_HYBRID_PROJECT-ENV_GROUP
. These could have been deleted erroneously.
Resolution
You can recreate the missing Kubernetes secrets using the TLS certificates and key files information
from the overrides.yaml
file:
- Run the following command to recreate the missing secrets:
apigeectl apply -f overrides/overrides.yaml --settings virtualhosts
The output of the above command should show that the secret is created:
secret/MY_HYBRID_PROJECT-ENV_GROUP created
- Verify that the Kubernetes secrets are successfully created using the following command:
kubectl -n apigee get secrets | grep MY_HYBRID_PROJECT-ENV_GROUP
The output of this command should show:
MY_HYBRID_PROJECT-ENV_GROUP Opaque 2 7s
If the problem persists, go to Must Gather Diagnostic Information.
Cause: Incorrectly formatted SSL certificate
Diagnosis
First, ensure that the certificate file is a .PEM file. To check if the correctly formatted
SSL certificate is loaded into apigee-ingressgateway
, follow these steps.
If you've configured one-way TLS using a cert/key pair as described in
Option 1: key/cert pair,
run:
openssl x509 -in $CERT_FILE -text -noout
An example output (without errors):
Certificate: Data: Version: 1 (0x0) Serial Number: 1 (0x1) Signature Algorithm: sha1WithRSAEncryption Issuer: C = US, O = xyz, OU = abc, CN = INTERIM-CN Validity Not Before: Dec 18 09:40:23 2023 GMT Not After : May 1 09:40:23 2025 GMT Subject: C = US, O = xyz, OU = abc, CN = shrey.example.com Subject Public Key Info: Public Key Algorithm: rsaEncryption RSA Public-Key: (2048 bit) Modulus: Trimmed Exponent: 65537 (0x10001) Signature Algorithm: sha1WithRSAEncryption Trimmed
If you've configured one-way TLS using a Kubernetes secret as described in Option 2: Kubernetes secret, run:
kubectl -n apigee get secret <$SECRET_NAME> -o jsonpath='{.data.cert}'| base64 -d > certfile ; openssl x509 -in certfile -text -noout kubectl -n apigee get secret <$SECRET_NAME> -o jsonpath='{.data.cert}'| base64 -d | openssl x509 -noout -text
An example output (without errors):
Certificate: Data: Version: 1 (0x0) Serial Number: 1 (0x1) Signature Algorithm: sha1WithRSAEncryption Issuer: C = US, O = xyz, OU = abc, CN = INTERIM-CN Validity Not Before: Dec 18 09:40:23 2023 GMT Not After : May 1 09:40:23 2025 GMT Subject: C = US, O = xyz, OU = abc, CN = shrey.example.com Subject Public Key Info: Public Key Algorithm: rsaEncryption RSA Public-Key: (2048 bit) Modulus: Trimmed Exponent: 65537 (0x10001) Signature Algorithm: sha1WithRSAEncryption Trimmed
If the output of the above command looks like
unable to load certificate 136613728412992:error:0D078095:asn1 encoding routines:asn1_item_embed_d2i:sequence not constructed:../crypto/asn1/tasn_dec.c:321:Type=X509 136613728412992:error:0906700D:PEM routines:PEM_ASN1_read_bio:ASN1 lib:../crypto/pem/pem_oth.c:33:
For errors like
error loading certificates 8360934016:error:09FFF066:PEM routines:CRYPTO_internal:bad end line
see the Resolution section.
Resolution
The errors could differ depending on the format error in the certificate file. Fix the certificate error if needed.
If the output shows the certificate instead of an error, consider Cause: The SSL Keys do not match the SSL certificate.
Cause: The SSL keys do not match the SSL certificate
Diagnosis
If you've configured one-way TLS using cert/key pair as described in Option 1: key/cert pair, run:
diff -q <(openssl rsa -noout -modulus -in $KEY_FILE ) <(openssl x509 -noout -modulus -in $CERT_FILE)
An example output (without errors):
diff -q <(openssl rsa -noout -modulus -in my_server.key ) <(openssl x509 -noout -modulus -in my_server.pem) <No output>
If you've configured one-way TLS using Kubernetes secret as described in Option 2: Kubernetes secret, run:
diff -q <(kubectl -n apigee get secrets $SECRET_NAME -o jsonpath='{.data.key}'| base64 -d | openssl rsa -noout -modulus) <(kubectl -n apigee get secrets $SECRET_NAME -o jsonpath='{.data.cert}'| base64 -d | openssl x509 -noout -modulus)
An example output (without errors):
diff -q <(kubectl -n apigee get secrets my-apigee-hybrid-env-grp -o jsonpath='{.data.key}'| base64 -d | openssl rsa -noout -modulus) <(kubectl -n apigee get secrets my-apigee-hybrid-env-grp -o jsonpath='{.data.cert}'| base64 -d | openssl x509 -noout -modulus) <No output>
If the output of the above command shows an error like:
unable to load Private Key 133504499987776:error:09091064:PEM routines:PEM_read_bio_ex:bad base64 decode:../crypto/pem/pem_lib.c:949: Files /dev/fd/63 and /dev/fd/62 differ
then the SSL Keys do not match the SSL certificate and please see the Resolution section.
Resolution
Ensure that you are providing the correct private keys and their corresponding SSL certificates. Fix any private key and SSL certificate problems.
If you do not see any output of the command, the SSL certificate and the SSL key match.
Must Gather Diagnostic Information
If the problem persists after following the instructions on this page, please gather the following diagnostic information and provide it to Apigee Support:
- The entire
curl
verbose output in which you see the error. - Capture a
tcpdump
on the machine on which you are seeing the errors and share it by removing any sensitive information and IP addresses. - Have the entire network topology and network flow ready after consulting with your network team to be shared with Apigee Support.