Header rewritten from https to http on Istio

Problem

You are seeing requests with the X-Forwarded-Proto header logged as HTTP at your backend service.

Even when manually adding the custom request header, X-Forwarded-Proto: HTTPS, you still see this header logged as HTTP at the backend. It appears as if this header is being rewritten.

Environment

  • Google Kubernetes Engine Cluster with Istio 1.4+
  • Reverse Proxy (Akamai/Cloudflare) > HTTP(S) Load Balancer > Istio Ingress Gateway > Backend

Solution

Option 1 Deploy an Envoy filter.

apiVersion: networking.istio.io/v1alpha3

kind: EnvoyFilter

metadata:

  name: xff-trust-hops

  namespace: istio-system

spec:

  workloadSelector:

    labels:

      istio: ingressgateway

  configPatches:

  - applyTo: NETWORK_FILTER

    match:

      context: ANY

      listener:

        filterChain:

          filter:

            name: "envoy.http_connection_manager"

    patch:

      operation: MERGE

      value:

        typed_config:

          "@type": "type.googleapis.com/envoy.config.filter.network.http_connection_manager.v2.HttpConnectionManager"

          use_remote_address: true

          xff_num_trusted_hops: 2 # Change as needed 


Option 2 Make use of the Alpha API to deploy an IstioOperator.

apiVersion: install.istio.io/v1alpha1

kind: IstioOperator

spec:

  meshConfig:

    defaultConfig:

      gatewayTopology:

        numTrustedProxies: 2 # Change as needed.

 

Cause

The header X-Forwarded-Proto is being automatically appended/rewritten by the Envoy proxy because it is unaware of the number of proxies in front of it. Different network topologies will require different Istio Ingress Gateway configurations to transport the correct client attributes. 

In this specific scenario, we have an HTTP(S) Load Balancer and a reverse proxy in front of the Istio Gateway, thus requiring us to set the numTrustedProxies to 2.