Resolving proxy issues in Anthos Service Mesh

This document explains common Anthos Service Mesh problems and how to resolve them. If you need additional assistance, see Getting support.

Connection Refused when reaching an endpoint with Istio

You might intermittently experience connection refused (ECONNREFUSED) errors with communication from your clusters to your endpoints, for example Memorystore Redis, CloudSQL, or any external service your application workload needs to reach.

This can occur when your application workload initiates faster than the istio-proxy (Envoy) container and tries to reach an external endpoint. Because at this stage istio-init (initContainer) has already executed, there are iptables rules in place redirecting all outgoing traffic to Envoy. Since istio-proxy is not ready yet, the iptables rules will redirect traffic to a sidecar proxy that is not yet started and therefore, the application gets the ECONNREFUSED error.

The following steps detail how to check if this is the error you are experiencing:

Check the stackdriver logs with the following Filter to identify which pods had the problem.

The following example shows a typical error message:

Error: failed to create connection to feature-store redis, err=dial tcp   192.168.9.16:19209: connect: connection refused
[ioredis] Unhandled error event: Error: connect ECONNREFUSED

Search for an occurrence of the problem. If you are using legacy Stackdriver, then use resource.type="container".
```
resource.type="k8s_container"
textPayload:"$ERROR_MESSAGE$"
```
Expand the latest occurrence to obtain the name of the pod and then make note of the pod_name under resource.labels.

Obtain the first occurrence of the issue for that pod:

resource.type="k8s_container"
resource.labels.pod_name="$POD_NAME$"

Example output:

E 2020-03-31T10:41:15.552128897Z
post-feature-service post-feature-service-v1-67d56cdd-g7fvb failed to create
connection to feature-store redis, err=dial tcp 192.168.9.16:19209: connect:
connection refused post-feature-service post-feature-service-v1-67d56cdd-g7fvb

Make note of the timestamp of the first error for this pod.

Use the following filter to see the pod startup events.

resource.type="k8s_container"
resource.labels.pod_name="$POD_NAME$"

Example output:

I 2020-03-31T10:41:15Z spec.containers{istio-proxy} Container image "docker.io/istio/proxyv2:1.3.3" already present on machine  spec.containers{istio-proxy}
I 2020-03-31T10:41:15Z spec.containers{istio-proxy} Created container  spec.containers{istio-proxy}
I 2020-03-31T10:41:15Z spec.containers{istio-proxy} Started container  spec.containers{istio-proxy}
I 2020-03-31T10:41:15Z spec.containers{APP-CONTAINER-NAME} Created container  spec.containers{APP-CONTAINER-NAME}
W 2020-03-31T10:41:17Z spec.containers{istio-proxy} Readiness probe failed: HTTP probe failed with statuscode: 503  spec.containers{istio-proxy}
W 2020-03-31T10:41:26Z spec.containers{istio-proxy} Readiness probe failed: HTTP probe failed with statuscode: 503  spec.containers{istio-proxy}
W 2020-03-31T10:41:28Z spec.containers{istio-proxy} Readiness probe failed: HTTP probe failed with statuscode: 503  spec.containers{istio-proxy}
W 2020-03-31T10:41:31Z spec.containers{istio-proxy} Readiness probe failed: HTTP probe failed with statuscode: 503  spec.containers{istio-proxy}
W 2020-03-31T10:41:58Z spec.containers{istio-proxy} Readiness probe failed: HTTP probe failed with statuscode: 503  spec.containers{istio-proxy}

Use the timestamps of errors and istio-proxy startup events to confirm the errors are happening when Envoy is not ready.

If the errors occur while the istio-proxy container is not ready yet, it is normal to obtain connection refused errors. In the preceding example, the pod was trying to connect to Redis as soon as 2020-03-31T10:41:15.552128897Z but by 2020-03-31T10:41:58Z istio-proxy was still failing readiness probes.

Even though the istio-proxy container started first, it is possible that it did not become ready fast enough before the app was already trying to connect to the external endpoint.

If this is the problem you are experiencing, then continue through the following troubleshooting steps.
Annotate the config at the pod level. This is only available at the pod level and not at a global level.
```
annotations:
proxy.istio.io/config: '{ "holdApplicationUntilProxyStarts": true }'
```
Modify the application code so that it checks if Envoy is ready before it tries to make any other requests to external services. For example, on application start, initiate a loop that makes requests to the istio-proxy health endpoint and only continues once a 200 is obtained. The istio-proxy health endpoint is as follows:
```
http://localhost:15020/healthz/ready
```

Race condition during sidecar injection between vault and istio

When using vault for secrets management, sometimes vault injects sidecar before istio, causing that Pods get stuck in Init status. When this happens, the Pods created get stuck in Init status after restarting any deployment or deploying a new one. For example:

E 2020-03-31T10:41:15.552128897Z
post-feature-service post-feature-service-v1-67d56cdd-g7fvb failed to create
connection to feature-store redis, err=dial tcp 192.168.9.16:19209: connect:
connection refused post-feature-service post-feature-service-v1-67d56cdd-g7fvb

This issue is caused by a race condition, both Istio and vault inject the sidecar and Istio must be the last doing this, the istio proxy is not running during init containers. The istio init container sets up iptables rules to redirect all traffic to the proxy. Since it is not running yet, those rules redirect to nothing, blocking all traffic. This is why the init container must be last, so the proxy is up and running immediately after the iptables rules are set up. Unfortunately, the order is not deterministic, so if Istio is injected first it breaks.

To troubleshoot this condition, allow the IP address of vault so the traffic going to the Vault IP is not redirected to the Envoy Proxy which is not ready yet and therefore blocking the communication. To achieve this, a new annotation named excludeOutboundIPRanges should be added.

For managed Anthos Service Mesh, this is only possible at Deployment or Pod level under spec.template.metadata.annotations, for example:

apiVersion: apps/v1
kind: Deployment
...
...
...
spec:
  template:
    metadata:
      annotations:
        traffic.sidecar.istio.io/excludeOutboundIPRanges:

For in-cluster Anthos Service Mesh, there is an option to set it as a global one with an IstioOperator under spec.values.global.proxy.excludeIPRanges, for example:

apiVersion: install.istio.io/v1alpha1
kind: IstioOperator
spec:
  values:
    global:
      proxy:
        excludeIPRanges: ""

After adding the annotation, restart your workloads.