Collecting Anthos Service Mesh logs
The following sections explains how to gather the various Anthos Service Mesh logs for troubleshooting issues or contacting Google Support.
Collect logs using the bug report tool
Anthos Service Mesh provides an automated bug report tool that collects the relevant diagnostic logs and lets you attach the logs to a Google Support ticket.
Before you begin, ensure your kubeconfig context is set to the target cluster.
Verify your context using the following command:
kubectl config current-context
Start the log collection
To start the log collection, run the bug-report tool using the following command:
Upload your debug archive
The tool creates an archive of your mesh's logs and configuration in the working directory. You can unpack the archive and use the troubleshooting guides to attempt to perform troubleshooting yourself. However, if you have a support package, you can contact Google Cloud Support, who will provide you with further steps to securely upload your log archive.
Manually collect Anthos Service Mesh logs
Instead of using the Anthos Service Mesh bug report tool, this section explains how to manually collect all the relevant logs.
Envoy access logs
Envoy proxy access logs contain detailed information that is useful for troubleshooting. However, you must enable them and set the correct detail level.
For details about how to interpret the log contents, see Interpret Envoy logs.
Enable or disable Envoy logs
To enable the Envoy proxy access logs, configure an overlay file for in-cluster Anthos Service Mesh or a ConfigMap for managed Anthos Service Mesh.
Increase logging detail
To temporarily increase the detail level of the logs, use the following command. This setting is undone when the pod is recreated.
kubectl -n NAMESPACE exec POD_NAME -c istio-proxy -- curl -X POST http://localhost:15000/logging?level=debug
To set the detail level of the logs back to default, use the following command:
kubectl -n NAMESPACE exec POD_NAME -c istio-proxy -- curl -X POST http://localhost:15000/logging?level=info
Write Envoy logs to a folder
To collect the Envoy proxy access logs and store them in a folder, use the following command:
kubectl logs -l app=APPLICATION_NAME -c istio-proxy > /FILE_PATH
See Getting Envoy's Access Logs for more information.
Kubernetes generates several logs that contain information about the behavior of
Istio components, such as
istiod, Ingress Gateway, and proxies. You can review
these logs for errors, which might narrow the scope of possible causes of a problem.
istiod logs using the following command:
kubectl -n istio-system logs $(kubectl -n istio-system get pods -lapp=istiod -oname) > ./LOGS_FOLDER/istiod.log
Capture Istio Ingress Gateway logs using the following command:
kubectl -n istio-system logs $(kubectl -n istio-system get pods -lapp=istio-ingressgateway -oname) > /FILE_PATH
Capture Istio Proxy logs using the following command:
kubectl -n WORKLOAD_NAMESPACE logs POD_NAME -c istio-proxy > ./LOGS_FOLDER/proxy.log
Kubernetes configuration dump
This information allows users without direct access to the cluster to view the state of various resources and identify possible configuration problems. The following command writes the Kubernetes configuration to a YAML file:
for namespace in "istio-system" "ns1" "ns2"; do kubectl get -oyaml deploy,statefulset,job,ingress,endpoints,configmap,event,secret,service,istio-io > ./LOGS_FOLDER/kubernetes.log; done
Envoy core dump
Envoy core dumps are not typically useful to end users, however Google Support might request that you collect it as part of the troubleshooting process, using the following steps.
To configure the kernel to write Envoy core dumps to a writable directory:
sidecar.istio.io/enableCoreDump=truelabel to a pod.
Restart the pod to enable Envoy core dumps.
Copy the core dump out of the pod.
Configure Envoy proxy
The detailed Envoy proxy configuration contains additional detail that might be helpful for troubleshooting purposes. You can collect this information using the following command. In this example, ENDPOINT is one of the following (shown in order of importance):
kubectl exec -i POD_NAME -c istio-proxy -n NAMESPACE curl 127.0.0.1:15000/ENDPOINT > out.log