apigeectl diagnostic
command.
What System data is captured?
Diagnostics collector captures following types of data:
- Changing Log levels.
- Jstack.
- POD configuration yaml.
- PS -ef output.
- TCP dump.
- TOP output.
What happens to the data?
When the Diagnostic collector captures the data, it is uploaded to a storage bucket in your Google Cloud project. You can view the stored data in the Google Cloud Platform: Cloud Storage browser.
You can optionally choose to share this data with Google Apigee Support when you create a support ticket.
Prerequisites for running Diagnostic collector
Before using the Diagnostic collector, you must complete the following prerequisites:
Google Cloud Storage bucket
Create a Google Cloud Storage bucket with a unique name in your Google Cloud project. You can
create and manage buckets with gcloud storage
commands or in the
Google Cloud
Platform: Cloud Storage browser.
For example:
gcloud storage buckets create gs://apigee_diagnostic_data
Creating gs://apigee_diagnostic_data/...
See Creating storage buckets for instructions.
Service account
Create a service account with the Storage Admin role (roles/storage.admin
)
in your project, and download the service account .json
key file.
The service account can have any unique name. This guide uses "apigee-diagnostic
"
for the service account name.
For example:
gcloud config set project ${PROJECT_ID}
gcloud iam service-accounts create apigee-diagnostic
gcloud projects add-iam-policy-binding ${PROJECT_ID} \ --member="serviceAccount:apigee-diagnostic@${PROJECT_ID}.iam.gserviceaccount.com" \ --role="roles/storage.admin"
gcloud iam service-accounts keys create ${PROJECT_ID}-apigee-diagnostic.json \ --iam-account=apigee-diagnostic@${PROJECT_ID}.iam.gserviceaccount.com
See:
- Creating and managing service accounts.
- Creating and managing service account keys.
- Understanding roles: Cloud Storage roles.
Using Diagnostic collector
The sequence to use the Diagnostic collector is:
- Configure the Diagnostic stanza in your
overrides.yaml
file to select the type of information, the Apigee container, and the individual pods you want diagnostic data from. See Configuringoverrides.yaml
for Diagnostic collector. - Run Diagnostic collector with the following
apigeectl
command.apigeectl diagnostic -f OVERRIDES_FILE
Where OVERRIDES_FILE is the path to your
overrides.yaml
file. - Check the logs:
- Get the pods in the
apigee-diagnostic
namespace.kubectl get pods -n apigee-diagnostic
- Make note of the pod with the name containing
diagnostic-collector
- Check the logs with the following command:
kubectl -n apigee-diagnostic logs -f POD_NAME
Where POD_NAME is the name of the Diagnostic collector pod.
You can also view the collected logs in the Google Cloud Platform: Cloud Storage browser.
- Get the pods in the
- After you have collected the data, delete the Diagnostic collector. You cannot run it again
until you have deleted it.
apigeectl diagnostic delete -f OVERRIDES_FILE
Configuring overrides.yaml
for Diagnostic collector
Before you can run the Diagnostic collector, you need to configure it in your
overrides.yaml
file.
For a complete reference of diagnostic
configuration properties, see the
Configuration property reference:
diagnostic
.
Required properties
The following properties are required for the Diagnostic collector to run.
diagnostic.serviceAccountPath
: The path to a service account key file for the service account with the Storage Admin role in Prerequisites.diagnostic.operation
: Specifies whether to collect all statistics or just logs.Values are:
"ALL"
or"LOGGING"
If you set
diagnostic.operation
to"LOGGING"
, the following properties are required:diagnostic.bucket
: The name of the Google Cloud storage bucket where your diagnostic data will be deposited. This is the bucket you created in Prerequisites.diagnostic.container
: This specifies which type of pod you are capturing data from. The values can be one of:container
valueApigee component Kubernetes namespace Example pod name in this container apigee-cassandra
Cassandra apigee
apigee-cassandra-default-0
istio-proxy
Istio ingress istio-system
istio-ingressgateway-696879cdf8-9zzzf
apigee-mart-server
MART apigee
apigee-mart-hybrid-example-d89fed1-151-jj2ux-l7nlb
apigee-runtime
Message Processor apigee
apigee-runtime-hybrid-example-3b2ebf3-151-s64bh-g9qmv
apigee-synchronizer
Synchronizer apigee
apigee-synchronizer-hybrid-example-3b2ebf3-151-xx4z6cg78
apigee-udca
UDCA apigee
apigee-udca-hybrid-example-3b2ebf3-151-q4g2c-vnzg9
apigee-watcher
Watcher apigee
apigee-watcher-hybrid-example-d89fed1-151-cpu3s-sxxdf
diagnostic.namespace
: The Kubernetes namespace in which the pods you are collecting data on reside. The namespace must be the correct one for the container you specify withdiagnostic.container
.diagnostic.podNames
: The names of the individual pods on which you want to collect diagnostic data. For example:diagnostic: … podNames: - apigee-runtime-hybrid-example-3b2ebf3-150-8vfoj-2wcjn - apigee-runtime-hybrid-example-3b2ebf3-150-8vfoj-6xzn2
Properties required only when the operation is set to LOGGING
The following properties are required only when running Diagnostic collector with
diagnostic.operation
is LOGGING
.
diagnostic.loggerNames
: Specifies by name which loggers to collect data from. For Apigee hybrid version 1.6.0, the only value supported isALL
, meaning all loggers. For example:diagnostic: … loggingDetails: loggerNames: - ALL
diagnostic.logLevel
: Specifies the granularity of the logging data to collect. In Apigee hybrid 1.6, OnlyFINE
is supported.diagnostic.logDuration
: The duration in milliseconds of the log data collected. A typical value is30000
.
Optional properties
The following properties are optional.
diagnostic.tcpDumpDetails.maxMsgs
: Sets the maximum number oftcpDump
messages to collect. Apigee recommends a maximum value no greater than1000
.diagnostic.tcpDumpDetails.timeoutInSeconds
: Sets the amount of time in seconds to wait fortcpDump
to return messages.diagnostic.threadDumpDetails.delayInSeconds
: The delay in seconds between collecting each thread dump. Must be used withdiagnostic.threadDumpDetails.iterations
.diagnostic.threadDumpDetails.iterations
: The number of jstack thread dump iterations to collect. Must be used withdiagnostic.threadDumpDetails.delayInSeconds
.
General example
The following is an example diagnostic
stanza showing all possible entries:
diagnostic: # required properties: serviceAccountPath: "service-accounts/apigee-diagnostics.json" operation: "ALL" bucket: "diagnostics_data" container: "apigee-runtime" namespace: "apigee" podNames: - apigee-runtime-hybrid-example-3b2ebf3-150-8vfoj-2wcjn - apigee-runtime-hybrid-example-3b2ebf3-150-8vfoj-6xzn2 # required if operation is Logging loggingDetails: loggerNames: - ALL logLevel: FINE logDuration: 30000 # optional properties: tcpDumpDetails: maxMsgs: 10 timeoutInSeconds: 100 threadDumpDetails: iterations: 5 delayInSeconds: 2
Common use cases
The following examples show how to configure and use Diagnostic collector in some common situations.
High Proxy Latency
In this case Apigee-runtime is taking a long time to process requests, causing customers to see high proxy latencies. You need to collect Jstack and TOP output.
- Select any 2 runtime pods.
- Create your
diagnostic
stanza with the following structure:diagnostic: serviceAccountPath: "service-accounts/apigee-diagnostics.json" operation: "ALL" bucket: "diagnostics_data" container: "apigee-runtime" namespace: "apigee" podNames: - apigee-runtime-hybrid-example-3b2ebf3-150-8vfoj-2wcjn - apigee-runtime-hybrid-example-3b2ebf3-150-8vfoj-6xzn2 tcpDumpDetails: maxMsgs: 10 threadDumpDetails: iterations: 15 delayInSeconds: 1
- After configuring the
diagnostic
stanza, run the Diagnostic collector.apigeectl diagnostic -f OVERRIDES_FILE
- Collect the logs and delete the Diagnostic collector.
apigeectl diagnostic delete -f OVERRIDES_FILE
Network / connectivity problems
You need to run diagnostics on apigee-runtime as well as ingress gateway pods.
- Select any 2 runtime pods.
- Create your
diagnostic
stanza with the following structure:diagnostic: serviceAccountPath: "service-accounts/apigee-diagnostics.json" operation: "ALL" bucket: "diagnostics_data" container: "apigee-runtime" namespace: "apigee" podNames: - apigee-runtime-hybrid-example-3b2ebf3-150-8vfoj-2wcjn - apigee-runtime-hybrid-example-3b2ebf3-150-8vfoj-6xzn2 tcpDumpDetails: maxMsgs: 1000
- After configuring the
diagnostic
stanza, run the Diagnostic collector.apigeectl diagnostic -f OVERRIDES_FILE
- Collect the logs and delete the Diagnostic collector.
apigeectl diagnostic delete -f OVERRIDES_FILE
- Select two pods from the Istio ingress gateway.
- Reconfigure your
diagnostic
stanza with the Istio ingress pods:diagnostic: serviceAccountPath: "service-accounts/apigee-diagnostics.json" operation: "ALL" bucket: "diagnostics_data" container: "istio-proxy" namespace: "istio-system" podNames: - istio-ingressgateway-696879cdf8-9zzzf - istio-ingressgateway-696879cdf8-6abc7 tcpDumpDetails: maxMsgs: 1000
- After configuring the
diagnostic
stanza, run the Diagnostic collector.apigeectl diagnostic -f OVERRIDES_FILE
- Collect the logs and delete the Diagnostic collector.
apigeectl diagnostic delete -f OVERRIDES_FILE
Proxies are throwing unexpected errors or new contracts are not getting applied
In this case you need to change log levels to debug for at least 5 minutes, or even 10 minutes as in this example. This will increase the amount of logs but helpful information will be logged. You will run Diagnostic collector twice, once on Apigee runtime then on Apigee synchronizer.
- Select any 2 runtime pods.
- Create your
diagnostic
stanza with the following structure:diagnostic: serviceAccountPath: "service-accounts/apigee-diagnostics.json" operation: "LOGGING" bucket: "diagnostics_data" namespace: "apigee" container: "apigee-runtime" podNames: - apigee-runtime-hybrid-example-3b2ebf3-150-8vfoj-2wcjn - apigee-runtime-hybrid-example-3b2ebf3-150-8vfoj-6xzn2 loggingDetails: loggerNames: - ALL logLevel: FINE logDuration: 60000
- After configuring the
diagnostic
stanza, run the Diagnostic collector.apigeectl diagnostic -f OVERRIDES_FILE
- Collect the logs and delete the Diagnostic collector.
apigeectl diagnostic delete -f OVERRIDES_FILE
- Select any 2 synchronizer pods.
- Create your
diagnostic
stanza with the following structure:diagnostic: serviceAccountPath: "service-accounts/apigee-diagnostics.json" operation: "LOGGING" bucket: "diagnostics_data" namespace: "apigee" container: "apigee-synchronizer" podNames: - apigee-synchronizer-hybrid-example-3b2ebf3-150-xx4z-6cg78 - apigee-synchronizer-hybrid-example-3b2ebf3-150-xx4z-1a2b3 loggingDetails: loggerNames: - ALL logLevel: FINE logDuration: 60000
- After configuring the
diagnostic
stanza, run the Diagnostic collector.apigeectl diagnostic -f OVERRIDES_FILE
- Collect the logs and delete the Diagnostic collector.
apigeectl diagnostic delete -f OVERRIDES_FILE