You're viewing Apigee and Apigee hybrid documentation.
View
Apigee Edge documentation.
Providing detailed and required information in the support case makes it easier for the Google Cloud Support team to respond to you quickly and efficiently. When your support case is missing critical details, we need to ask for more information, which may involve going back and forth multiple times. This takes more time and can lead to delays in resolution of issues. This Best Practices Guide lets you know the information we need to resolve your technical support case faster.
Describing the issue
An issue should contain information explaining the details about what happened versus what was expected to happen, as well as when and how it happened. A good support case should contain the following key information for each of the Apigee products:
| Key information | Description | Apigee on Google Cloud | Apigee hybrid |
|---|---|---|---|
| Product | Specific Apigee product in which the problem is being observed, including version information where applicable. |
|
|
| Problem Details | Clear and detailed problem description which outlines the issue, including the complete error message, if any. |
|
|
| Time | The specific timestamp when the issue started and how long it lasted. |
|
|
| Setup | Detailed information where the problem is being observed. |
|
|
The following sections describe these concepts in greater detail.
Product
There are different Apigee products, Apigee on Google Cloud and Apigee hybrid, so we need specific information about which particular product is having the issue.
The following table provides some examples showing complete information in the DOs column, and incomplete information in the DON'Ts column:
| DOs | DON'Ts |
|---|---|
Deployment of API proxy OAuth2 failed on our
Apigee on Google Cloud org ... |
Deployment of API proxy failed (We need to know Apigee product in which you are seeing the issue.) |
We are getting the following error while accessing Cassandra using
cqlsh on Apigee hybrid version 1.3 ... |
We are unable to access Cassandra using (Hybrid version information is missing) |
Problem Details
Provide the precise information about the issue being observed including the error message (if any) and expected and actual behaviour observed.
The following table provides some examples showing complete information in the DOs column, and incomplete information in the DON'Ts column:
| DOs | DON'Ts |
|---|---|
|
New
|
New (The proxy name is unknown. It is not clear whether the proxy is returning an error or any unexpected response.) |
|
Our clients are getting
|
Our clients are getting
(Just conveying |
Time
Time is a very critical piece of information. It is important for the Support Engineer to know when you first noticed this issue, how long it lasted, and if the issue is still going on.
The Support Engineer resolving the issue may not be in your timezone, so relative statements about time make the problem harder to diagnose. Hence, it is recommended to use the ISO 8601 format for the date and time stamp to provide the exact time information on when the issue was observed.
The following table provides some examples showing accurate time and duration for which the problem occurred in the DOs column, and ambiguous or unclear information on when the problem occurred in the DON'Ts column:
| DOs | DON'Ts |
|---|---|
Huge number of 503s were observed yesterday between
2020-11-06 17:30 PDT and 2020-11-06 17:35 PDT... |
Huge number of (We are forced to use the implied date and it is also unclear in which timezone this issue was observed.) |
| High latencies were observed on the following API Proxies from 2020-11-09 15:30 IST to 2020-11-09 18:10 IST ... |
High latencies were observed on some API Proxies last week. (It is unclear which day and duration this issue was observed in the last week.) |
Setup
We need to know the details about where exactly you are seeing the issue. Depending on the product you are using, we need the following information:
- If you are using Apigee on Google Cloud, you may have more
than one organization, so we need to know the specific org and other
details where you are observing the issue:
- Organization and Environment names
- API proxy name and revision numbers (for API request failures)
- If you are using hybrid, you may be using one of the many supported hybrid platforms and installation topologies. So we need to know what hybrid platform and topology you are using, including the details such as number of data centres and nodes.
The following table provides some examples showing complete information in the DOs column, and incomplete information in the DON'Ts column:
| DOs | DON'Ts |
|---|---|
|
Apigee setup details:
Details of the failing API are as follows:
Error:
|
(It does not give any information on the product being used, since when the issue is being observed or any setup details.) |
Debug is failing with the following error on Apigee hybrid version 1.3 Error:
Apigee hybrid setup details:
| Debug is failing on Apigee hybrid. |
Useful artifacts
Providing us with artifacts related to the issue will speed up the resolution, as it helps us understand the exact behavior you are observing and get more insights into it.
This section describes some useful artifacts that are helpful for all Apigee products:
Common artifacts for all Apigee products
The following artifacts are useful for all Apigee products: Apigee on Google Cloud and Apigee hybrid:
| Artifact | Description |
|---|---|
| Debug tool output | The Debug tool output contains detailed information about the API requests flowing through
Apigee products. This is useful for any runtime errors such as 4XX,
5XX, and latency issues. |
| Screenshots | Screenshots help relay the context of the actual behaviour or error being observed. It is helpful for any errors or issues observed, such as in the UI or Analytics. |
| HAR (Http ARchive) | HAR is a file that is captured by HTTP session tools for debugging any UI related issues. This can be captured using browsers such as Chrome, Firefox, or Internet Explorer. |
tcpdumps |
The tcpdump tool captures TCP/IP packets transferred or received over the
network. This is useful for any network-related issues such as TLS handshake failures,
502 errors, and latency issues, etc. |
Additional artifacts for hybrid
For hybrid, we may need some additional artifacts that will facilitate faster diagnosis of issues.
| Artifact | Description |
|---|---|
| Apigee hybrid platform | Specify any of the following
supported hybrid platforms that are used:
|
| Apigee hybrid and dependent component versions |
|
| Network topology | The Apigee installation topology diagram describing your hybrid setup including all the data centers, Kubernetes clusters, namespaces, and pods. |
| Overrides YAML File | The overrides.yaml file used in each data center for installing Apigee hybrid runtime plane.
|
| Status of Apigee hybrid deployment |
The output of the following commands in each data center/Kubernetes cluster:
|
| Apigee hybrid component logs |
Provide links to StackDriver logs for the hybrid components OR You can fetch the Apigee hybrid component logs using the following commands in each data center/Kubernetes cluster and share them with us:
|
| Describe logs |
Detailed information about the pod. This is useful especially if you are observing issues such as pods getting stuck in the
|
| Cloud Monitoring |
|
Apigee Hybrid Must-Gather
You can also execute the Must-Gather script based on the commands listed below;
###--- "kubectl config" commands to get the config details of the whole Apigee Hybrid cluster ---####
kubectl config get-clusters 2>&1 | tee /tmp/k_config_get_clusters_$(date +%Y.%m.%d_%H.%M.%S).txt
kubectl config get-contexts 2>&1 | tee /tmp/k_config_get_contexts_$(date +%Y.%m.%d_%H.%M.%S).txt
kubectl config get-users 2>&1 | tee /tmp/k_config_get_users_$(date +%Y.%m.%d_%H.%M.%S).txt
kubectl config view 2>&1 | tee /tmp/k_config_view_$(date +%Y.%m.%d_%H.%M.%S).txt
### --- Collect all details of all nodes in the Kubernetes cluster.---###
kubectl describe node 2>&1 | tee /tmp/k_describe_node_$(date +%Y.%m.%d_%H.%M.%S).txt
###--- "kubectl get -A " commands to get CRD details for the whole Apigee Hybrid setup ---####
kubectl get clusterissuers -A -o wide 2>&1 | tee /tmp/k_get_clusterissuers_all$(date +%Y.%m.%d_%H.%M.%S).txt
kubectl get certificate -A -o wide 2>&1 | tee /tmp/k_get_certificate_all_$(date +%Y.%m.%d_%H.%M.%S).txt
kubectl get certificaterequest -A -o wide 2>&1 | tee /tmp/k_get_certificaterequest_all_$(date +%Y.%m.%d_%H.%M.%S).txt
kubectl get crd -A 2>&1 | tee /tmp/k_get_crd_all_$(date +%Y.%m.%d_%H.%M.%S).txt
kubectl get ConfigMap -A 2>&1 | tee /tmp/k_get_ConfigMap_all_$(date +%Y.%m.%d_%H.%M.%S).txt
kubectl get ClusterRole -A -o wide 2>&1 | tee /tmp/k_get_clusterrole_all_$(date +%Y.%m.%d_%H.%M.%S).txt
kubectl get ClusterRoleBinding -A -o wide 2>&1 | tee /tmp/k_get_clusterrole_binding_all_$(date +%Y.%m.%d_%H.%M.%S).txt
kubectl get Deployments -A -o wide >&1 | tee /tmp/k_get_deployments_all_$(date +%Y.%m.%d_%H.%M.%S).txt
kubectl get events -A -o wide 2>&1 | tee /tmp/k_get_events_all_$(date +%Y.%m.%d_%H.%M.%S).txt
kubectl get endpoints -A 2>&1 | tee /tmp/k_get_endpoints_all_$(date +%Y.%m.%d_%H.%M.%S).txt
kubectl get issuers -A -o wide 2>&1 | tee /tmp/k_get_issuers_all_$(date +%Y.%m.%d_%H.%M.%S).txt
kubectl get mutatingwebhookconfigurations 2>&1 | tee /tmp/k_get_mutatingwebhookconfigurations_$(date +%Y.%m.%d_%H.%M.%S).txt
kubectl get nodes -o wide --show-labels 2>&1 | tee /tmp/k_get_nodes_labels_$(date +%Y.%m.%d_%H.%M.%S).txt
kubectl get ns 2>&1 | tee /tmp/k_get_namespace_$(date +%Y.%m.%d_%H.%M.%S).txt
kubectl get PriorityClass -A -o wide 2>&1 | tee /tmp/k_get_PriorityClass_all_$(date +%Y.%m.%d_%H.%M.%S).txt
kubectl get pv -A -o wide 2>&1 | tee /tmp/k_get_pv_all_$(date +%Y.%m.%d_%H.%M.%S).txt
kubectl get pvc -A -o wide 2>&1 | tee /tmp/k_get_pvc_all_$(date +%Y.%m.%d_%H.%M.%S).txt
kubectl get Role -A -o wide 2>&1 | tee /tmp/k_get_role_all_$(date +%Y.%m.%d_%H.%M.%S).txt
kubectl get RoleBinding -A -o wide 2>&1 | tee /tmp/k_get_Role_Binding_all_$(date +%Y.%m.%d_%H.%M.%S).txt
kubectl get replicaset -A -o wide 2>&1 | tee /tmp/k_get_replicaset_all_$(date +%Y.%m.%d_%H.%M.%S).txt
kubectl get sa -A -o wide 2>&1 | tee /tmp/k_get_service_accounts_all_$(date +%Y.%m.%d_%H.%M.%S).txt
kubectl get services -A -o wide 2>&1 | tee /tmp/k_get_services_all_$(date +%Y.%m.%d_%H.%M.%S).txt
kubectl get svc -A 2>&1 | tee /tmp/k_get_svc_all$(date +%Y.%m.%d_%H.%M.%S).txt
kubectl get secrets -A 2>&1 | tee /tmp/k_get_secrets_all_$(date +%Y.%m.%d_%H.%M.%S).txt
kubectl get validatingwebhookconfigurations -A 2>&1 | tee /tmp/k_get_validatingwebhookconfigurations_all$(date +%Y.%m.%d_%H.%M.%S).txt
kubectl get validatingwebhookconfigurations apigee-validating-webhook-configuration 2>&1 | tee /tmp/k_get_apigee-validating-webhook-configuration_$(date +%Y.%m.%d_%H.%M.%S).txt
### --- List top resource consuming nodes and pods ---####
kubectl top nodes 2>&1 | tee /tmp/k_top_nodes_$(date +%Y.%m.%d_%H.%M.%S).txt
kubectl top pod -A --containers 2>&1 | tee /tmp/k_top_pod_all_containers_$(date +%Y.%m.%d_%H.%M.%S).txt
###----- "kubectl get" commands to fetch list of all CRD for "apigee" namespace ----- #####
kubectl get all -n apigee -o wide 2>&1 | tee /tmp/k_get_all_n_apigee_$(date +%Y.%m.%d_%H.%M.%S).txt
kubectl get ad -n apigee 2>&1 | tee /tmp/k_get_ad_n_apigee_$(date +%Y.%m.%d_%H.%M.%S).txt
kubectl get apigeeorganization -n apigee 2>&1 | tee /tmp/k_get_apigeeorganization_n_apigee_$(date +%Y.%m.%d_%H.%M.%S).txt
kubectl get apigeeenv -n apigee 2>&1 | tee /tmp/k_get_apigeeenv_n_apigee_$(date +%Y.%m.%d_%H.%M.%S).txt
kubectl get apigeeds -n apigee 2>&1 | tee /tmp/k_get_apigeeds_n_apigee_$(date +%Y.%m.%d_%H.%M.%S).txt
kubectl get apigeedatastore -n apigee 2>&1 | tee /tmp/k_get_apigeedatastore_n_apigee_$(date +%Y.%m.%d_%H.%M.%S).txt
kubectl get ApigeeDeployment -n apigee 2>&1 | tee /tmp/k_get_apigeedeployment_n_apigee_$(date +%Y.%m.%d_%H.%M.%S).txt
kubectl get ApigeeRedis -n apigee 2>&1 | tee /tmp/k_get_ApigeeRedis_n_apigee_$(date +%Y.%m.%d_%H.%M.%S).txt
kubectl get ApigeeRoute -n apigee 2>&1 | tee /tmp/k_get_ApigeeRoute_n_apigee_$(date +%Y.%m.%d_%H.%M.%S).txt
kubectl get ApigeeRouteConfig -n apigee 2>&1 | tee /tmp/k_get_ApigeeRoutesconfig_n_apigee_$(date +%Y.%m.%d_%H.%M.%S).txt
kubectl get Apigeetelemetry -n apigee 2>&1 | tee /tmp/k_get_Apigeetelemetry_n_apigee_$(date +%Y.%m.%d_%H.%M.%S).txt
kubectl get apigeeissues -n apigee 2>&1 | tee /tmp/k_get_apigeeissues_n_apigee_$(date +%Y.%m.%d_%H.%M.%S).txt
kubectl get ControllerRevision -n apigee -o wide 2>&1 | tee /tmp/k_get_ControllerRevision_n_apigee_$(date +%Y.%m.%d_%H.%M.%S).txt
kubectl get cronjob -n apigee -o wide 2>&1 | tee /tmp/k_get_cronjob_n_apigee_$(date +%Y.%m.%d_%H.%M.%S).txt
kubectl get gateway -n apigee 2>&1 | tee /tmp/k_get_gateway_n_apigee_$(date +%Y.%m.%d_%H.%M.%S).txt
kubectl get PodDisruptionBudget -n apigee -o wide 2>&1 | tee /tmp/k_get_PodDisruptionBudget_n_apigee_$(date +%Y.%m.%d_%H.%M.%S).txt
kubectl get sc -n apigee -o wide 2>&1 | tee /tmp/k_get_storageclass_n_apigee_$(date +%Y.%m.%d_%H.%M.%S).txt
kubectl get sts -n apigee 2>&1 | tee /tmp/k_get_sts_n_apigee_$(date +%Y.%m.%d_%H.%M.%S).txt
kubectl get volumesnapshot -n apigee -o wide 2>&1 | tee /tmp/k_get_volumesnapshot_n_apigee_$(date +%Y.%m.%d_%H.%M.%S).txt
###----- "kubectl describe" commands to fetch details of all CRD for "apigee" namespace ----- #####
for p in $(kubectl -n apigee get apigeeorganization --no-headers -o custom-columns=":metadata.name") ; do kubectl describe apigeeorganization ${p} -n apigee 2>&1 | tee /tmp/k_desc_apigeeorganization_n_apigee_${p}_$(date +%Y.%m.%d_%H.%M.%S).txt ; done
for p in $(kubectl -n apigee get apigeeenv --no-headers -o custom-columns=":metadata.name") ; do kubectl describe apigeeenv ${p} -n apigee 2>&1 | tee /tmp/k_desc_apigeeenv_n_apigee_${p}_$(date +%Y.%m.%d_%H.%M.%S).txt ; done
for p in $(kubectl -n apigee get apigeeds --no-headers -o custom-columns=":metadata.name") ; do kubectl describe apigeeds ${p} -n apigee 2>&1 | tee /tmp/k_desc_apigeeds_n_apigee_${p}_$(date +%Y.%m.%d_%H.%M.%S).txt ; done
for p in $(kubectl -n apigee get apigeedatastore --no-headers -o custom-columns=":metadata.name") ; do kubectl describe apigeedatastore ${p} -n apigee 2>&1 | tee /tmp/k_desc_apigeedatastore_n_apigee_${p}_$(date +%Y.%m.%d_%H.%M.%S).txt ; done
for p in $(kubectl -n apigee get ApigeeDeployment --no-headers -o custom-columns=":metadata.name") ; do kubectl describe ApigeeDeployment ${p} -n apigee 2>&1 | tee /tmp/k_desc_ApigeeDeployment_n_apigee_${p}_$(date +%Y.%m.%d_%H.%M.%S).txt ; done
for p in $(kubectl -n apigee get ApigeeRedis --no-headers -o custom-columns=":metadata.name") ; do kubectl describe ApigeeRedis ${p} -n apigee 2>&1 | tee /tmp/k_desc_ApigeeRedis_n_apigee_${p}_$(date +%Y.%m.%d_%H.%M.%S).txt ; done
for p in $(kubectl -n apigee get ApigeeRoute --no-headers -o custom-columns=":metadata.name") ; do kubectl describe ApigeeRoute ${p} -n apigee 2>&1 | tee /tmp/k_desc_ApigeeRoute_n_apigee_${p}_$(date +%Y.%m.%d_%H.%M.%S).txt ; done
for p in $(kubectl -n apigee get ApigeeRouteConfig --no-headers -o custom-columns=":metadata.name") ; do kubectl describe ApigeeRouteConfig ${p} -n apigee 2>&1 | tee /tmp/k_desc_ApigeeRouteConfig_n_apigee_${p}_$(date +%Y.%m.%d_%H.%M.%S).txt ; done
for p in $(kubectl -n apigee get Apigeetelemetry --no-headers -o custom-columns=":metadata.name") ; do kubectl describe Apigeetelemetry ${p} -n apigee 2>&1 | tee /tmp/k_desc_Apigeetelemetry_n_apigee_${p}_$(date +%Y.%m.%d_%H.%M.%S).txt ; done
for p in $(kubectl -n apigee get apigeeissues --no-headers -o custom-columns=":metadata.name") ; do kubectl describe apigeeissues ${p} -n apigee 2>&1 | tee /tmp/k_desc_apigeeissues_n_apigee_${p}_$(date +%Y.%m.%d_%H.%M.%S).txt ; done
for p in $(kubectl -n apigee get ControllerRevision --no-headers -o custom-columns=":metadata.name") ; do kubectl -n apigee describe ControllerRevision ${p} 2>&1 | tee /tmp/k_desc_ControllerRevision_n_apigee_${p}_$(date +%Y.%m.%d_%H.%M.%S).txt ; done
for p in $(kubectl -n apigee get certificate --no-headers -o custom-columns=":metadata.name") ; do kubectl describe certificate ${p} -n apigee 2>&1 | tee /tmp/k_desc_certificate_n_apigee_${p}_$(date +%Y.%m.%d_%H.%M.%S).txt ; done
for p in $(kubectl -n apigee get cronjob --no-headers -o custom-columns=":metadata.name") ; do kubectl describe cronjob ${p} -n apigee 2>&1 | tee /tmp/k_desc_cronjob_n_apigee_${p}_$(date +%Y.%m.%d_%H.%M.%S).txt ; done
for p in $(kubectl -n apigee get daemonset --no-headers -o custom-columns=":metadata.name") ; do kubectl -n apigee describe daemonset ${p} 2>&1 | tee /tmp/k_desc_daemonset_n_apigee_${p}_$(date +%Y.%m.%d_%H.%M.%S).txt ; done
for p in $(kubectl -n apigee get deployments --no-headers -o custom-columns=":metadata.name") ; do kubectl -n apigee describe deployments ${p} 2>&1 | tee /tmp/k_desc_deployment_n_apigee_${p}_$(date +%Y.%m.%d_%H.%M.%S).txt ; done
for p in $(kubectl -n apigee get hpa --no-headers -o custom-columns=":metadata.name") ; do kubectl describe hpa ${p} -n apigee 2>&1 | tee /tmp/k_desc_hpa_n_apigee_${p}_$(date +%Y.%m.%d_%H.%M.%S).txt ; done
for p in $(kubectl -n apigee get jobs --no-headers -o custom-columns=":metadata.name") ; do kubectl describe jobs ${p} -n apigee 2>&1 | tee /tmp/k_desc_jobs_n_apigee_${p}_$(date +%Y.%m.%d_%H.%M.%S).txt ; done
for p in $(kubectl -n apigee get po --no-headers -o custom-columns=":metadata.name") ; do kubectl -n apigee describe po ${p} 2>&1 | tee /tmp/k_desc_pod_n_apigee_${p}_$(date +%Y.%m.%d_%H.%M.%S).txt ; done
for p in $(kubectl -n apigee get PodDisruptionBudget --no-headers -o custom-columns=":metadata.name") ; do kubectl -n apigee describe PodDisruptionBudget ${p} 2>&1 | tee /tmp/k_desc_PodDisruptionBudget_n_apigee_${p}_$(date +%Y.%m.%d_%H.%M.%S).txt ; done
for p in $(kubectl -n apigee get pv --no-headers -o custom-columns=":metadata.name") ; do kubectl -n apigee describe pv ${p} 2>&1 | tee /tmp/k_desc_pv_n_apigee_${p}_$(date +%Y.%m.%d_%H.%M.%S).txt; done
for p in $(kubectl -n apigee get pvc --no-headers -o custom-columns=":metadata.name") ; do kubectl -n apigee describe pvc ${p} 2>&1 | tee /tmp/k_desc_pvc_n_apigee_${p}_$(date +%Y.%m.%d_%H.%M.%S).txt; done
for p in $(kubectl -n apigee get rs --no-headers -o custom-columns=":metadata.name") ; do kubectl describe rs ${p} -n apigee 2>&1 | tee /tmp/k_desc_replicaset_n_apigee_${p}_$(date +%Y.%m.%d_%H.%M.%S).txt ; done
for p in $(kubectl -n apigee get sc --no-headers -o custom-columns=":metadata.name") ; do kubectl -n apigee describe sc ${p} 2>&1 | tee /tmp/k_desc_storageclass_n_apigee_${p}_$(date +%Y.%m.%d_%H.%M.%S).txt; done
for p in $(kubectl -n apigee get sts --no-headers -o custom-columns=":metadata.name") ; do kubectl describe sts ${p} -n apigee 2>&1 | tee /tmp/k_desc_sts_n_apigee_${p}_$(date +%Y.%m.%d_%H.%M.%S).txt ; done
for p in $(kubectl -n apigee get secrets --no-headers -o custom-columns=":metadata.name") ; do kubectl -n apigee describe secrets ${p} 2>&1 | tee /tmp/k_desc_secrets_n_apigee${p}_$(date +%Y.%m.%d_%H.%M.%S).txt; done
for p in $(kubectl -n apigee get services --no-headers -o custom-columns=":metadata.name") ; do kubectl describe service ${p} -n apigee 2>&1 | tee /tmp/k_desc_services_n_apigee_${p}_$(date +%Y.%m.%d_%H.%M.%S).txt ; done
for p in $(kubectl -n apigee get sa --no-headers -o custom-columns=":metadata.name") ; do kubectl describe sa ${p} -n apigee 2>&1 | tee /tmp/k_desc_service_account_n_apigee_${p}_$(date +%Y.%m.%d_%H.%M.%S).txt ; done
for p in $(kubectl -n apigee get svc --no-headers -o custom-columns=":metadata.name") ; do kubectl describe svc ${p} -n apigee 2>&1 | tee /tmp/k_desc_svc_n_apigee_${p}_$(date +%Y.%m.%d_%H.%M.%S).txt ; done
###----- "kubectl logs" command to fetch logs of all containers in the "apigee" namespace ----- #####
for p in $(kubectl -n apigee get po --no-headers -o custom-columns=":metadata.name") ; do kubectl -n apigee logs ${p} --all-containers 2>&1 | tee /tmp/k_logs_n_apigee_${p}_$(date +%Y.%m.%d_%H.%M.%S).log ; done
###----- "kubectl get" commands for "apigee-system" namespace ----- #####
kubectl get all -n apigee-system -o wide 2>&1 | tee /tmp/k_get_all_n_apigee_system_$(date +%Y.%m.%d_%H.%M.%S).txt
kubectl get jobs -o wide -n apigee-system 2>&1 | tee /tmp/k_get_jobs_n_apigee_system_$(date +%Y.%m.%d_%H.%M.%S).txt
###----- "kubectl describe" commands for "apigee-system" namespace ----- #####
for p in $(kubectl -n apigee-system get certificate --no-headers -o custom-columns=":metadata.name") ; do kubectl describe certificate ${p} -n apigee-system 2>&1 | tee /tmp/k_desc_certificate_n_apigee_system_${p}_$(date +%Y.%m.%d_%H.%M.%S).txt ; done
for p in $(kubectl -n apigee-system get deployment --no-headers -o custom-columns=":metadata.name") ; do kubectl describe deployment ${p} -n apigee-system 2>&1 | tee /tmp/k_desc_deployment_n_apigee_system_${p}_$(date +%Y.%m.%d_%H.%M.%S).txt ; done
for p in $(kubectl -n apigee-system get jobs --no-headers -o custom-columns=":metadata.name") ; do kubectl describe jobs ${p} -n apigee-system 2>&1 | tee /tmp/k_desc_jobs_n_apigee_system_${p}_$(date +%Y.%m.%d_%H.%M.%S).txt ; done
for p in $(kubectl -n apigee-system get po --no-headers -o custom-columns=":metadata.name") ; do kubectl -n apigee-system describe po ${p} 2>&1 | tee /tmp/k_desc_pod_n_apigee_system_${p}_$(date +%Y.%m.%d_%H.%M.%S).txt ; done
for p in $(kubectl -n apigee-system get rs --no-headers -o custom-columns=":metadata.name") ; do kubectl describe rs ${p} -n apigee-system 2>&1 | tee /tmp/k_desc_replicaset_n_apigee_system_${p}_$(date +%Y.%m.%d_%H.%M.%S).txt ; done
for p in $(kubectl -n apigee-system get rolebinding --no-headers -o custom-columns=":metadata.name") ; do kubectl describe rolebinding ${p} -n apigee-system 2>&1 | tee /tmp/k_desc_rolebinding_n_apigee_system_${p}_$(date +%Y.%m.%d_%H.%M.%S).txt ; done
for p in $(kubectl -n apigee-system get services --no-headers -o custom-columns=":metadata.name") ; do kubectl describe service ${p} -n apigee-system 2>&1 | tee /tmp/k_desc_services_n_apigee_system_${p}_$(date +%Y.%m.%d_%H.%M.%S).txt ; done
for p in $(kubectl -n apigee-system get sa --no-headers -o custom-columns=":metadata.name") ; do kubectl describe sa ${p} -n apigee-system 2>&1 | tee /tmp/k_desc_serviceaccount_n_apigee_system_${p}_$(date +%Y.%m.%d_%H.%M.%S).txt ; done
for p in $(kubectl -n apigee-system get secrets --no-headers -o custom-columns=":metadata.name") ; do kubectl describe secrets ${p} -n apigee-system 2>&1 | tee /tmp/k_desc_secrets_n_apigee_system_${p}_$(date +%Y.%m.%d_%H.%M.%S).txt ; done
###----- "kubectl logs" command for "apigee-system" namespace ----- #####
for p in $(kubectl -n apigee-system get po --no-headers -o custom-columns=":metadata.name") ; do kubectl -n apigee-system logs ${p} --all-containers 2>&1 | tee /tmp/k_logs_n_apigee_system_${p}_$(date +%Y.%m.%d_%H.%M.%S).log ; done
###----- "kubectl get" command for "cert-manager" namespace ----- #####
kubectl get all -n cert-manager -o wide 2>&1 | tee /tmp/k_get_all_n_cert_manager_$(date +%Y.%m.%d_%H.%M.%S).txt
kubectl get crd -n cert-manager 2>&1 | tee /tmp/k_get_crd_n_cert_manager_$(date +%Y.%m.%d_%H.%M.%S).txt
###----- "kubectl describe" command for "cert-manager" namespace ----- #####
for p in $(kubectl -n cert-manager get deployment --no-headers -o custom-columns=":metadata.name") ; do kubectl -n cert-manager describe deployment $(p) 2>&1 | tee /tmp/k_desc_deployment_n_cert_manager_${p}_$(date +%Y.%m.%d_%H.%M.%S).txt ; done
for p in $(kubectl -n cert-manager get endpoints --no-headers -o custom-columns=":metadata.name") ; do kubectl describe endpoints ${p} -n cert-manager 2>&1 | tee /tmp/k_desc_endpoints_n_cert_manager_${p}_$(date +%Y.%m.%d_%H.%M.%S).txt ; done
for p in $(kubectl -n cert-manager get po --no-headers -o custom-columns=":metadata.name") ; do kubectl -n cert-manager describe po ${p} 2>&1 | tee /tmp/k_desc_po_n_cert_manager_${p}_$(date +%Y.%m.%d_%H.%M.%S).txt ; done
for p in $(kubectl -n cert-manager get rs --no-headers -o custom-columns=":metadata.name") ; do kubectl describe rs ${p} -n cert-manager 2>&1 | tee /tmp/k_desc_replicaset_n_cert_manager_${p}_$(date +%Y.%m.%d_%H.%M.%S).txt ; done
for p in $(kubectl -n cert-manager get sa --no-headers -o custom-columns=":metadata.name") ; do kubectl describe sa ${p} -n cert-manager 2>&1 | tee /tmp/k_desc_serviceaccount_n_cert_manager_${p}_$(date +%Y.%m.%d_%H.%M.%S).txt ; done
for p in $(kubectl -n cert-manager get secrets --no-headers -o custom-columns=":metadata.name") ; do kubectl describe secrets ${p} -n cert-manager 2>&1 | tee /tmp/k_desc_secrets_n_cert_manager_${p}_$(date +%Y.%m.%d_%H.%M.%S).txt ; done
for p in $(kubectl -n cert-manager get services --no-headers -o custom-columns=":metadata.name") ; do kubectl describe service ${p} -n cert-manager 2>&1 | tee /tmp/k_desc_service_n_cert_manager_${p}_$(date +%Y.%m.%d_%H.%M.%S).txt ; done
for p in $(kubectl -n cert-manager get svc --no-headers -o custom-columns=":metadata.name") ; do kubectl describe svc ${p} -n cert-manager 2>&1 | tee /tmp/k_desc_svc_n_cert_manager_${p}_$(date +%Y.%m.%d_%H.%M.%S).txt ; done
###----- "kubectl logs" command for "cert-manager" namespace ----- #####
for p in $(kubectl -n cert-manager get po --no-headers -o custom-columns=":metadata.name") ; do kubectl -n cert-manager logs ${p} --all-containers 2>&1 | tee /tmp/k_logs_n_cert_manager_${p}_$(date +%Y.%m.%d_%H.%M.%S).log ; done
Once logs are generated compress all the output file in a single tar ball using the below command;
# tar -cvzf /tmp/apigee_hybrid_logs_$(date +%Y.%m.%d_%H.%M).tar.gz /tmp/k_*
In case the size of the tar file is larger than 25MB you can upload it to google drive and share the link with us. Or you can use the split command to break the large files into 25MB chunks that can be uploaded to the Support portal.
# split -b 25M diagnostic.tar.gz "diagnostic.tar.gz.part"
Case templates and sample cases
This section provides case templates and sample cases for different products based on the best practices described in this document:
Apigee Cloud
Template
This section provides a sample template for Apigee on Google Cloud.
Problem:
<Provide detailed description of the problem or the behaviour being observed at your end. Include the product name and version where applicable.>
Error message:
<Include the complete error message observed (if any)>
Problem start time (ISO 8601 format):
Problem end time (ISO 8601 format):
Apigee setup details:
Org names:
Env names:
API proxy names:
Revision numbers:
Steps to reproduce:
<Provide steps to reproduce the issue where possible>
Diagnostic information:
<List of files attached>
Sample case
This section provides a sample case for Apigee on Google Cloud.
Problem:
We are seeing a high number of 503 Service Unavailable errors in our Public Cloud org. Can you please look into the issue and resolve it or advise us how to resolve it?
Error message:
{"fault":{"faultstring":"The Service is temporarily available", "detail":{"errorcode":"messaging.adaptors.http.flow.ServiceUnavailable"}}}
Problem start time (ISO 8601 format): 2020-10-04 06:30 IST
Problem end time (ISO 8601 format): The issue is still happening.
Apigee Cloud setup details:
Org names: myorg
Env names: dev
API proxy names: myproxy
Revision numbers: 3
Steps to reproduce:
Run the following curl command to reproduce the issue:
curl -X GET 'https://myorg-dev.apigee.net/v1/myproxy'
Diagnostic information:
Debug tool output (trace-503.xml)
Hybrid
Template
This section provides a sample template for Apigee hybrid.
Problem:
<Provide detailed description of the problem or the behaviour being observed at your end. Include the product name and version where applicable.>
Error message:
<Include the complete error message observed (if any)>
Problem start time (ISO 8601 format):
Problem end time (ISO 8601 format):
Apigee hybrid setup details:
- Apigee hybrid platform:
<Provide the information about the Platform where you have installed hybrid and its version.>
- Google Cloud project, hybrid Organization and Environment:
Google Cloud project ID:
<If you are using Google Kubernetes Engine (GKE), ensure you provide the project ID where the clusters are located. If you are using GKE on-prem, Azure Kubernetes Service or Amazon EKS, then provide the project ID where you are sending the logs.>
Apigee hybrid org:
Apigee hybrid env: - Apigee hybrid and other CLI versions:
Apigee hybrid CLI (apigeectl) version:
Kubectl version: - Kubernetes cluster name details:
k8sCluster:
name:
region: - Network topology:
<Attach the network topology describing the setup of your Apigee hybrid including data centers, Kubernetes clusters, namespaces, and pods.> - Overrides YAML File:
<Attach the Overrides YAML file.>
Steps to reproduce
<Provide steps to reproduce the issue where possible>
Diagnostic information:
<List of files attached>
Sample case
This section provides a sample case for Apigee hybrid.
Problem:
We are getting errors when executing management APIs on Apigee hybrid version 1.3.
Error message:
[ERROR] 400 Bad Request
{
"error": {
"code": 400,
"message": "Error processing MART request: INTERNAL_ERROR",
"errors": [
{
"message": "Error processing MART request: INTERNAL_ERROR",
"domain": "global",
"reason": "failedPrecondition"
}
],
"status": "FAILED_PRECONDITION"
}
}
Problem start time (ISO 8601 format): Since 2020-10-24 10:30 PDT
Problem end time (ISO 8601 format): Continuing to observe the issue.
Apigee hybrid setup details:
- Apigee hybrid platform
GKE version 1.15.1 - Google Cloud project, hybrid Organization and Environment
Google Cloud project ID:apigee-hybrid-123456
Note: This is the project ID where the clusters are located.
Apigee hybrid org:apigee-hybrid-123456
Apigee hybrid env:dev - Apigee hybrid and other CLI versions:
Apigee hybrid CLI (apigeectl) version:
Version:1.2.0
Commit:ac09109
Build ID:214
Build Time:2020-03-30T20:23:36Z
Go Version:go1.12
Kubectl version:
Client Version:version.Info{Major:"1", Minor:"15", GitVersion:"v1.15.0", GitCommit:"e8462b5b5dc2584fdcd18e6bcfe9f1e4d970a529", GitTreeState:"clean", BuildDate:"2019-06-19T16:40:16Z", GoVersion:"go1.12.5", Compiler:"gc", Platform:"darwin/amd64"}
Server Version:version.Info{Major:"1", Minor:"14+", GitVersion:"v1.14.10-gke.36", GitCommit:"34a615f32e9a0c9e97cdb9f749adb392758349a6", GitTreeState:"clean", - Kubernetes cluster name details:
k8sCluster:
name:user-cluster-1
region:us-east1 - Network topology
Attached the filenetwork-topology.png - Overrides YAML File
Attached the fileoverrides.yamlfile
Steps to reproduce:
Run the following management API to observe the error:
curl -X GET --header "Authorization: Bearer <TOKEN>" "https://apigee.googleapis.com/v1/organizations/apigee-hybrid-123456/environments/dev/keyvaluemaps"
Diagnostic Information:
Attached the following files:
network-topology.pngoverrides.yaml file- MART logs
- Synchronizer logs