Collecting Anthos Service Mesh logs

The following sections explains how to gather the various Anthos Service Mesh logs for troubleshooting issues or contacting Google Support.

Collect logs using the bug report tool

Anthos Service Mesh provides an automated bug report tool that collects the relevant diagnostic logs and lets you attach the logs to a Google Support ticket.

Before you begin, ensure your kubeconfig context is set to the target cluster.

1. Verify your context using the following command:

   kubectl config current-context

2. The procedure to download and use the bug report tool depends on the version of Anthos Service Mesh you use. See the following table to determine whether your current version of istioctl is sufficient or if you must download a standalone version of the tool.

Anthos Service Mesh/Istio version	Bug report tool
Anthos Service Mesh 1.7.* and higher	istioctl bug-report
Anthos Service Mesh 1.6.*	Standalone
Anthos Service Mesh 1.5.*	Standalone
Istio 1.7.*	Standalone
Istio 1.6.*	Standalone
Istio 1.5.*	Standalone

To download the standalone version of the bug report tool, use the following steps:

1. Choose a distribution from the list that matches your OS environment:

   • https://storage.googleapis.com/gke-release/asm/bug-report_darwin_amd64-v2
   • https://storage.googleapis.com/gke-release/asm/bug-report_linux_386-v2
   • https://storage.googleapis.com/gke-release/asm/bug-report_linux_amd64-v2
   • https://storage.googleapis.com/gke-release/asm/bug-report_linux_arm-v2

2. Use curl to download the chosen distribution, for example:

   curl -LO https://storage.googleapis.com/gke-release/asm/bug-report_darwin_amd64-v1

3. Set permissions on the bug report tool binary to allow it to run, for example:

   chmod +x bug-report_darwin_amd64-v1

Start the log collection

To start the log collection, run the bug-report tool and pass the configuration file as a parameter. Additional runtime options are available to overwrite the configuration if needed, which you can view using the --help option.

If your version of Anthos Service Mesh has the bug-report tool already contained within istioctl, use the following command:

istioctl bug-report

If you require the standalone bug report tool, rename the tool and run it like the following commands that use the darwin distribution as an example:

mv ./bug-report_darwin_amd64-v1 ./bug-report

./bug-report

Upload your debug archive

Place your debug log archive in the bug report tool working directory. You can unpack the archive and use the troubleshooting guides to attempt to perform troubleshooting yourself. However, if you have a support package, you can contact Google Cloud Support, who will provide you with further steps to securely upload your log archive.

Important: Archive logs might contain personally identifiable information (PII) or private information about your security certificates. Check your local laws and your organization's policies and procedures regarding how to handle or send such sensitive information externally.

Manually collect Anthos Service Mesh logs

Instead of using the Anthos Service Mesh bug report tool, this section explains how to manually collect all the relevant logs.

Envoy access logs

Envoy proxy access logs contain detailed information that is useful for troubleshooting. However, you must enable them and set the correct detail level.

For details about how to interpret the log contents, see Interpret Envoy logs.

Enable or disable Envoy logs

To enable the Envoy proxy access logs, use istioctl install like the following example:

istioctl install --set profile=PROFILE_NAME \
    --set meshConfig.accessLogFile="/dev/stdout" \
    --set meshConfig.accessLogEncoding="JSON"

Important: This operation when Istio is already installed will cause an interruption of service, because it restarts Istio pods in the new configuration.

The istioctl install command provides the following options to configure the logs:

Value	Description
accessLogFile	Defines the file address for the proxy access log (for example, /dev/stdout). An empty value disables access logging.
accessLogFormat	Defines the format for the proxy access log. An empty value sets the proxy's default access log format.
accessLogEncoding	The encoding for the proxy access log (`TEXT` or `JSON`). The default value is `TEXT`.

To disable the logs, set accessLogFile to an empty string:

istioctl install --set profile=PROFILE_NAME \
    --set meshConfig.accessLogFile="" \
    --set meshConfig.accessLogEncoding="TEXT"

Increase logging detail

To temporarily increase the detail level of the logs, use the following command. This setting is undone when the pod is recreated.

kubectl -n NAMESPACE exec POD_NAME -c istio-proxy -- curl -X POST http://localhost:15000/logging?level=info

Write Envoy logs to a folder

To collect the Envoy proxy access logs and store them in a folder, use the following command:

kubectl logs -l app=APPLICATION_NAME -c istio-proxy > /FILE_PATH

See Getting Envoy's Access Logs for more information.

Kubernetes logs

Kubernetes generates several logs that contain information about the behavior of Istio components, such as Istiod, Ingress Gateway, and proxies. You can review these logs for errors, which might narrow the scope of possible causes of a problem.

Capture Istiod logs using the following command:

kubectl -n istio-system logs $(kubectl -n istio-system get pods -lapp=istiod -oname) > ./LOGS_FOLDER/istiod.log

Capture Istio Ingress Gateway logs using the following command:

kubectl -n istio-system logs $(kubectl -n istio-system get pods -lapp=istio-ingressgateway -oname) > /FILE_PATH

Capture Istio Proxy logs using the following command:

kubectl -n WORKLOAD_NAMESPACE logs POD_NAME -c istio-proxy > ./LOGS_FOLDER/proxy.log

Kubernetes configuration dump

This information allows users without direct access to the cluster to view the state of various resources and identify possible configuration problems. The following command writes the Kubernetes configuration to a YAML file:

for namespace in "istio-system" "ns1" "ns2"; do kubectl get -oyaml deploy,statefulset,job,ingress,endpoints,configmap,event,secret,service,istio-io > ./LOGS_FOLDER/kubernetes.log; done

Envoy core dump

Envoy core dumps are not typically useful to end users, however Google Support might request that you collect it as part of the troubleshooting process, using the following steps.

Important: Core dumps might contain personally identifiable information (PII) or private information about your security certificates. Check your local laws and your organization's policies and procedures regarding how to handle or send such sensitive information externally.

1. Enable core dumps for all proxies in your mesh by adding the following to your IstioOperator configuration:

   spec:
   values:
   global:
     proxy:
       enableCoreDumps: true

2. Re-install using the following command:

   istioctl install -f myOperatorFile.yaml

3. Delete your target pod so that it is recreated with proxy core dumps enabled.

4. Let the process run and when it experiences the problem, trigger the core dump by running the following command in the istio-proxy container:

   kubectl exec -it POD_NAME -c istio-proxy

5. Find the PID for the envoy container:

   ps aux | grep -i envoy

6. Use the PID to stop the envoy process, which generates a core dump:

   kill -3 PID

7. Wait for the container to restart (or use the kill command).

8. Run the following command to extract the core dump to your current directory:

   kubectl cp PID:/var/lib/istio/data/core.proxy -c istio-proxy ./core.proxy

Envoy proxy configuration

The detailed Envoy proxy configuration contains additional detail that might be helpful for troubleshooting purposes. You can collect this information using the following command. In this example, ENDPOINT is one of the following (shown in order of importance): * /certs * /clusters * /listeners * /config_dump * /memory * /server_info * /stats/prometheus * /runtime

kubectl exec -i POD_NAME -c istio-proxy curl 127.0.0.1:15000/ENDPOINT > out.log