kubectl -n apigee-system get certificates apigee-serving-cert
原因：為續訂 apigee-serving-cert 而建立重複的憑證要求
診斷
檢查 cert-manager 控制器記錄，看看是否傳回類似以下的錯誤訊息。
列出所有 cert-manager Pod：
kubectl -n cert-manager get pods
輸出內容範例：
NAME READY STATUS RESTARTS AGE
cert-manager-66d9545484-772cr 1/1 Running 0 6d19h
cert-manager-cainjector-7d8b6bd6fb-fpz6r 1/1 Running 0 6d19h
cert-manager-webhook-669b96dcfd-6mnm2 1/1 Running 0 6d19h
檢查 cert-manager 控制器記錄：
kubectl -n cert-manager logs cert-manager-66d9545484-772cr | grep "issuance is skipped until there are no more duplicates"
輸出內容範例：
1controller.go:163]cert-manager/certificates-readiness"msg"="re-queuing item due to error processing""error"="multiple CertificateRequests were found for the 'next' revision 3, issuance is skipped until there are no more duplicates""key"="apigee-system/apigee-serving-cert"
1controller.go:167]cert-manager/certificates-readiness"msg"="re-queuing item due to error processing""error"="multiple CertificateRequests were found for the 'next' revision 683, issuance is skipped until there are no more duplicates""key"="apigee/apigee-istiod"
[[["容易理解","easyToUnderstand","thumb-up"],["確實解決了我的問題","solvedMyProblem","thumb-up"],["其他","otherUp","thumb-up"]],[["難以理解","hardToUnderstand","thumb-down"],["資訊或程式碼範例有誤","incorrectInformationOrSampleCode","thumb-down"],["缺少我需要的資訊/範例","missingTheInformationSamplesINeed","thumb-down"],["翻譯問題","translationIssue","thumb-down"],["其他","otherDown","thumb-down"]],["上次更新時間：2025-09-10 (世界標準時間)。"],[[["\u003cp\u003eAPI proxy deployments in Apigee hybrid may fail due to an expired or invalid TLS certificate for the \u003ccode\u003eapigee-webhook-service.apigee-system.svc\u003c/code\u003e, leading to errors in \u003ccode\u003eapigee-watcher\u003c/code\u003e logs.\u003c/p\u003e\n"],["\u003cp\u003eThe issue can be caused by the \u003ccode\u003eapigee-serving-cert\u003c/code\u003e not being found, duplicate certificate requests, or an unhealthy \u003ccode\u003ecert-manager\u003c/code\u003e.\u003c/p\u003e\n"],["\u003cp\u003eResolutions involve updating the \u003ccode\u003eapigee-serving-cert\u003c/code\u003e using Helm, deleting duplicate certificate requests in the \u003ccode\u003eapigee-system\u003c/code\u003e namespace, and ensuring the \u003ccode\u003ecert-manager\u003c/code\u003e is healthy.\u003c/p\u003e\n"],["\u003cp\u003eIf problems persist, diagnostic information such as Google Cloud Project ID, Apigee hybrid organization details, \u003ccode\u003eoverrides.yaml\u003c/code\u003e file, Kubernetes pod status, and a Kubernetes \u003ccode\u003ecluster-info\u003c/code\u003e dump should be gathered for Google Cloud Customer Care.\u003c/p\u003e\n"]]],[],null,["*You're viewing **Apigee** and **Apigee hybrid** documentation.\nView [Apigee Edge](https://docs.apigee.com/api-platform/troubleshoot/support-d) documentation.*\n| **Note:** This document is applicable for Apigee hybrid users only.\n\nSymptoms\n\n\nAPI proxy deployments fail with the following error messages.\n\nError Messages\n\n\nIf the TLS certificate of the\n`apigee-webhook-service.apigee-system.svc` service has expired\nor is not yet valid, the following error message will be shown on\n`apigee-watcher` logs: \n\n```transact-sql\n{\"level\":\"error\",\"ts\":1687991930.7745812,\"caller\":\"watcher/watcher.go:60\",\n\"msg\":\"error during watch\",\"name\":\"ingress\",\"error\":\"INTERNAL: INTERNAL: failed\nto update ApigeeRoute [org-env]-group-84a6bb5, namespace apigee:\nInternal error occurred: failed calling webhook\n\\\"mapigeeroute.apigee.cloud.google.com\\\": Post\n\\\"https://apigee-webhook-service.apigee-system.svc:443/mutate-apigee-cloud-google-com-v1alpha1-apigeeroute?timeout=30s\\\":\nx509:\ncertificate has expired or is not yet valid: current time\n2023-06-28T22:38:50Z is after 2023-06-17T17:14:13Z, INTERNAL: failed to update\nApigeeRoute [org-env]-group-e7b3ff6, namespace apigee \n```\n\nPossible Causes\n\n| Cause | Description |\n|----------------------------------------------------------------------------------------------|--------------------------------------------------------------------------------------------------------------------------------------------------------------------|\n| [The apigee-serving-cert is not found](#not-found) | If the `apigee-serving-cert` is not found in the `apigee-system` namespace, this issue could occur. |\n| [Duplicate certificate requests were created for renewing `apigee-serving-cert`](#duplicate) | If there are duplicate certificate requests created for renewing the `apigee-serving-cert` certificate, the `apigee-serving-cert` certificate may not get renewed. |\n| [cert-manager is not healthy](#not-healthy) | If `cert-manager` is not healthy, the `apigee-serving-cert` certificate may not get renewed. |\n\nCause: The apigee-serving-cert is not found\n\nDiagnosis\n\n1.\n Check the availability of the `apigee-serving-cert` certificate\n in the `apigee-system` namespace:\n\n ```\n kubectl -n apigee-system get certificates apigee-serving-cert\n ```\n\n\n If this certificate is available, an output similar to following should\n be seen: \n\n ```text\n NAME READY SECRET AGE\n apigee-serving-cert True webhook-server-cert 2d10h\n ```\n2. If the apigee-serving-cert certificate is not found in the `apigee-system` namespace, that could be the reason for this issue.\n\nResolution\n\n1. Update the `apigee-serving-cert` using [Helm](/apigee/docs/hybrid/latest/helm-reference): \n\n ```\n helm upgrade ENV_NAME apigee-env/ \\\n --namespace APIGEE_NAMESPACE \\\n --set env=ENV_NAME \\\n --atomic \\\n -f OVERRIDES_FILE\n ```\n\n Make sure to include all of the settings shown, including `--atomic`\n so that the action rolls back on failure.\n2. Verify that the `apigee-serving-cert` certificate has been created: \n\n ```\n kubectl -n apigee-system get certificates apigee-serving-cert\n ```\n\nCause: Duplicate certificate requests were created for\nrenewing apigee-serving-cert\n\nDiagnosis\n\n1.\n Check `cert-manager` controller logs and see whether an error\n message similar to the following has been returned.\n\n List all `cert-manager` pods: \n\n ```\n kubectl -n cert-manager get pods\n ```\n\n An example output: \n\n ```text\n NAME READY STATUS RESTARTS AGE\n cert-manager-66d9545484-772cr 1/1 Running 0 6d19h\n cert-manager-cainjector-7d8b6bd6fb-fpz6r 1/1 Running 0 6d19h\n cert-manager-webhook-669b96dcfd-6mnm2 1/1 Running 0 6d19h\n ```\n\n Check `cert-manager` controller logs: \n\n ```\n kubectl -n cert-manager logs cert-manager-66d9545484-772cr | grep \"issuance is skipped until there are no more duplicates\"\n ```\n\n Example outputs: \n\n ```cbm-basic-v2\n 1 controller.go:163] cert-manager/certificates-readiness \"msg\"=\"re-queuing item due to error processing\" \"error\"=\"multiple CertificateRequests were found for the 'next' revision 3, issuance is skipped until there are no more duplicates\" \"key\"=\"apigee-system/apigee-serving-cert\"\n ``` \n\n ```cbm-basic-v2\n 1 controller.go:167] cert-manager/certificates-readiness \"msg\"=\"re-queuing item due to error processing\" \"error\"=\"multiple CertificateRequests were found for the 'next' revision 683, issuance is skipped until there are no more duplicates\" \"key\"=\"apigee/apigee-istiod\"\n ```\n\n\n If you see either of the messages shown above, the `apigee-serving-cert` and the `apigee-istiod-cert` certificates will not be renewed.\n2. List all certificate requests in the `apigee-system` namespace or the `apigee` namespace depending on the namespace printed in the log entries above and check to see if there are multiple certificate requests created for renewing the same `apigee-serving-cert` or `apigee-istiod-cert` certificate revisions: \n\n ```\n kubectl -n apigee-system get certificaterequests\n ```\n\n\nSee the `cert-manager` issue relevant to this problem at\n[cert-manager created multiple CertificateRequest objects with the same\ncertificate-revision](https://github.com/cert-manager/cert-manager/issues/4956).\n\nResolution **Note:** Execute the following commands for the `apigee` or `apigee-system` namespace depending on the namespace where the duplicate certificate requests were found.\n\n1. Delete all certificate requests in `apigee-system` namespace: \n\n ```\n kubectl -n apigee-system delete certificaterequests --all\n ```\n2. Verify that duplicated certificate requests have been deleted and only one certificate request is available for the `apigee-serving-cert` certificate in `apigee-system` namespace: \n\n ```\n kubectl -n apigee-system get certificaterequests\n ```\n3. Verify that the `apigee-serving-cert` certificate has been renewed: \n\n ```\n kubectl -n apigee-system get certificates apigee-serving-cert -o yaml\n ```\n\n An example output: \n\n ```actionscript-3\n apiVersion: cert-manager.io/v1\n kind: Certificate\n metadata:\n creationTimestamp: \"2023-06-26T13:25:10Z\"\n generation: 1\n name: apigee-serving-cert\n namespace: apigee-system\n resourceVersion: \"11053\"\n uid: e7718341-b3ca-4c93-a6d4-30cf70a33e2b\n spec:\n dnsNames:\n - apigee-webhook-service.apigee-system.svc\n - apigee-webhook-service.apigee-system.svc.cluster.local\n issuerRef:\n kind: Issuer\n name: apigee-selfsigned-issuer\n secretName: webhook-server-cert\n status:\n conditions:\n - lastTransitionTime: \"2023-06-26T13:25:11Z\"\n message: Certificate is up to date and has not expired\n observedGeneration: 1\n reason: Ready\n status: \"True\"\n type: Ready\n notAfter: \"2023-09-24T13:25:11Z\"\n notBefore: \"2023-06-26T13:25:11Z\"\n renewalTime: \"2023-08-25T13:25:11Z\"\n revision: 1\n ```\n\nCause: cert-manager is not healthy\n\nDiagnosis\n\n1. Check the health of the `cert-manager` pods in the `cert-manager` namespace: \n\n ```\n kubectl -n cert-manager get pods\n ```\n\n\n If `cert-manager` pods are healthy, all\n `cert-manager` pods should be ready `(1/1)` and in\n `Running` state, otherwise, that could be the reason for this\n issue: \n\n ```text\n NAME READY STATUS RESTARTS AGE\n cert-manager-59cf78f685-mlkvx 1/1 Running 0 15d\n cert-manager-cainjector-78cc865768-krjcp 1/1 Running 0 15d\n cert-manager-webhook-77c4fb46b6-7g9g6 1/1 Running 0 15d\n ```\n2.\n The `cert-manager` can fail for many reasons. Check the\n `cert-manager` logs and identify the reason for the failure\n and resolve them accordingly.\n\n\n One known reason is that the `cert-manager` will fail if it\n cannot communicate with the Kubernetes API. In this case, an error\n message similar to following is displayed:: \n\n ```transact-sql\n E0601 00:10:27.841516 1 leaderelection.go:330] error retrieving\n resource lock kube-system/cert-manager-controller: Get\n \"https://192.168.0.1:443/api/v1/namespaces/kube-system/configmaps/cert-manager-controller\":\n dial tcp 192.168.0.1:443: i/o timeout\n ```\n\nResolution\n\n1. Check the health of the Kubernetes cluster and fix any issues found. See [Troubleshooting Clusters](https://kubernetes.io/docs/tasks/debug/debug-cluster/).\n2. Refer to [Troubleshooting](https://cert-manager.io/docs/troubleshooting) for additional `cert-manager` troubleshooting information.\n\nMust gather diagnostic information\n\n\nIf the problem persists even after following the above instructions, gather\nthe following diagnostic information, and then contact\n[Google Cloud Customer Care](https://cloud.google.com/support-hub/).\n\n1. Google Cloud Project ID\n2. Apigee hybrid organization\n3. Apigee hybrid `overrides.yaml` file, masking any sensitive information.\n4. Kubernetes pod status in all namespaces: \n\n ```\n kubectl get pods -A \u003e kubectl-pod-status`date +%Y.%m.%d_%H.%M.%S`.txt\n ```\n5. Kubernetes `cluster-info` dump: \n\n ```\n # generate kubernetes cluster-info dump\n kubectl cluster-info dump -A --output-directory=/tmp/kubectl-cluster-info-dump\n # zip kubernetes cluster-info dump\n zip -r kubectl-cluster-info-dump`date +%Y.%m.%d_%H.%M.%S`.zip /tmp/kubectl-cluster-info-dump/*\n ```"]]
