Edit on GitHub
Report issue
Page history

Author(s): @{[ username ]}  Published: {[ TutorialCtrl.tutorial.date_published | date:'mediumDate' ]}

title: Transparent Proxy and Filtering on Kubernetes description: Learn how to run a transparent proxy on Kubernetes to filter and intercept traffic out of your deployments. author: danisla tags: Kubernetes, mitmproxy, proxy date_published: 2017-09-23


Dan Isla | Google Cloud Solution Architect | Google

There are many application environments and scenarios where you may want to filter and intercept all HTTP/S traffic out of a pod. Example use cases include isolating external access to specific HTTP or HTTPS paths, methods and in-flight altering of requests.

An example of HTTPS interception would be filtering access to Google Cloud Storage buckets. Most proxy solutions do not support HTTPS inspection so the filtering that can be done is limited at the IP/DNS level. For the case of a Cloud Storage bucket, the bucket name is embedded under the https://storage.googleapis.com/BUCKET_NAME/object URL so filtering is limited to all buckets under the storage.googleapis.com domain, not any specific bucket.

If you wanted to transparently add request/response headers to requests between services for tracing purposes, you can do that without the services having to explicitly set them.

Mitmproxy is an open source tool that you can use to intercept and modify HTTP and HTTPS requests transparently using the Python scripting language.

This tutorial uses the tproxy-sidecar init container to create firewall rules in the pod network to block egress traffic out of selected pods. The tproxy-podwatch controller watches for pod changes containing the "initializer.kubernetes.io/tproxy": "true" annotation and automatically add/removes the local firewall REDIRECT rules to apply the transparent proxy to the pod.

architecture diagram

Figure 1. transparent proxy architecture diagram

Objectives

  • Create a Kubernetes cluster with Google Kubernetes Engine.
  • Deploy the tproxy and the tproxy-podwatch pods using Helm.
  • Deploy example apps to test external access to a Cloud Storage bucket.

Before you begin

This tutorial assumes you already have a Google Cloud Platform (GCP) account and are familiar with the high level concepts of Kubernetes Pods and Deployments.

Costs

This tutorial uses billable components of GCP, including:

Use the Pricing Calculator to estimate the costs for your environment.

Checkout the source repository

  1. Open Cloud Shell

  2. Clone the repository containing the code for this tutorial:

    git clone https://github.com/danisla/kubernetes-tproxy
    cd kubernetes-tproxy
    

    The remainder of this tutorial will be run from the root of the cloned repository directory.

Create Kubernetes Engine cluster and install Helm

  1. Create the Kubernetes Engine cluster:

    gcloud container clusters create tproxy-example --zone us-central1-f
    

    This command also automatically configures the kubectl command to use the cluster.

  2. Install Helm locally in your Cloud Shell instance:

    curl -sL https://storage.googleapis.com/kubernetes-helm/helm-v2.5.1-linux-amd64.tar.gz | tar -xzvf - && sudo mv linux-amd64/helm /usr/local/bin/ && rm -Rf linux-amd64
    
  3. Initialize Helm:

    helm init
    

    This installs the server side component of Helm, Tiller, in the Kubernetes cluster. The Tiller pod may take a minute to start, run the command below to verify it has been deployed:

    helm version
    

    You should see the Client and Server versions in the output:

    Client: {SemVer:"v2.5.1", GitCommit:"7cf31e8d9a026287041bae077b09165be247ae66", GitTreeState:"clean"}
    Server: {SemVer:"v2.5.1", GitCommit:"7cf31e8d9a026287041bae077b09165be247ae66", GitTreeState:"clean"}
    

Install the Helm chart

Before installing the chart, you must first extract the certificates generated by mitmproxy. The generated CA cert is used in the example pods to trust the proxy when making HTTPS requests.

  1. Extract the generated certs using Docker:

    cd charts/tproxy
    docker run --rm -v ${PWD}/certs/:/home/mitmproxy/.mitmproxy mitmproxy/mitmproxy >/dev/null 2>&1
    
  2. Install the chart:

    helm install -n tproxy .
    

    The output of this command shows you how to augment your deployments to use the init container and trusted certificate configmap volume. Example output below:

    Add the init container spec below to your deployments:
    
    initContainers:
    - name: tproxy
      image: docker.io/danisla/tproxy-sidecar:0.1.0
      imagePullPolicy: IfNotPresent
      securityContext:
        privileged: true
      env:
      resources:
        limits:
          cpu: 500m
          memory: 128Mi
        requests:
          cpu: 100m
          memory: 64Mi
    
    Add the volumes below to your deployments to use the trusted https tproxy:
    
    volumes:
    - name: ca-certs-debian
      configMap:
        name: tproxy-tproxy-root-certs
        items:
        - key: root-certs.crt
          path: ca-certificates.crt
    
  3. Get the status of the DaemonSet pods:

    kubectl get pods -o wide
    

    Notice in the example output below that there is a tproxy pod for each node:

    NAME                     READY     STATUS    RESTARTS   AGE       IP           NODE
    tproxy-tproxy-2h7lk   2/2       Running   0          21s       10.128.0.8   gke-tproxy-example-default-pool-1e70b38d-xchn
    tproxy-tproxy-4mvtf   2/2       Running   0          21s       10.128.0.7   gke-tproxy-example-default-pool-1e70b38d-hk89
    tproxy-tproxy-ljfq9   2/2       Running   0          21s       10.128.0.6   gke-tproxy-example-default-pool-1e70b38d-jsqd
    

The tproxy chart is now installed and ready to be used by pods with the init container and pod annotation.

Deploy example apps

Deploy the sample apps to demonstrate using and not using the init container to lock down external access from the pod.

  1. Change directories back to the repository root and deploy the example apps:

    cd ../../
    kubectl create -f examples/debian-app.yaml
    kubectl create -f examples/debian-app-locked-manual.yaml
    

    Note that the second deployment is the one that contains the init container and trusted certificate volume mount described in the chart post-install notes.

  2. Get the logs for the pod without the tproxy annotation:

    kubectl logs --selector=app=debian-app,variant=unlocked --tail=10
    

    Example output:

    https://www.google.com: 200
    https://storage.googleapis.com/solutions-public-assets/: 200
    PING www.google.com (209.85.200.105): 56 data bytes
    64 bytes from 209.85.200.105: icmp_seq=0 ttl=52 time=0.758 ms
    

    The output from the example app shows the status codes for the requests and the output of a ping command.

    Notice the following: - The request to https://www.google.com succeeds with status code 200. - The request to the Cloud Storage bucket succeeds with status code 200. - The the ping to www.google.com succeeds.

  3. Get the logs for the pod with the tproxy annotation:

    kubectl logs --selector=app=debian-app,variant=locked --tail=4
    

    Example output:

    https://www.google.com: 418
    https://storage.googleapis.com/solutions-public-assets/: 200
    PING www.google.com (209.85.200.147): 56 data bytes
    ping: sending packet: Operation not permitted
    

    Notice the following: - The proxy blocks the request to https://www.google.com with status code 418. - The proxy allows the request to the Cloud Storage bucket with status code 200. - The the ping to www.google.com is rejected.

  4. Inspect the logs from the mitmproxy DaemonSet pod to show the intercepted requests and responses. Note that the logs have to be retrieved from the tproxy pod that is running on the same node as the example app.

    kubectl logs $(kubectl get pods -o wide | awk '/tproxy.*'$(kubectl get pods --selector=app=debian-app,variant=locked -o=jsonpath={.items..spec.nodeName})'/ {print $1}') -c tproxy-tproxy-mode --tail=10
    

    Example output:

    10.12.1.41:37380: clientconnect
    10.12.1.41:37380: GET https://www.google.com/ HTTP/2.0
                << 418 I'm a teapot 30b
    10.12.1.41:37380: clientdisconnect
    10.12.1.41:36496: clientconnect
    Streaming response from 64.233.191.128
    10.12.1.41:36496: GET https://storage.googleapis.com/solutions-public-assets/adtech/dfp_networkimpressions.py HTTP/2.0
                << 200  (content missing)
    10.12.1.41:36496: clientdisconnect
    

    Notice that the proxy blocks the request to https://www.google.com with status code 418.

Customizing the mitmproxy Python script

This tutorial uses a Python script to filter traffic to a specific Cloud Storage bucket. The Python script is installed as a ConfigMap resource, and mitmproxy can live-reload the script when it changes without restarting.

  1. Modify the script to change the "access denied" status code:

    cd charts/tproxy/
    sed -e 's/418/500/g' config/mitm-script.py
    
  2. Upgrade the Helm chart with the change:

    helm upgrade tproxy .
    

    After about 30 seconds, the new script will be live-updated and in use by mitmproxy.

  3. Verify the output of the locked pod:

    kubectl logs --selector=app=debian-app,variant=locked --tail=4
    

    Example output:

    https://www.google.com: 500
    https://storage.googleapis.com/solutions-public-assets/: 200
    PING www.google.com (209.85.200.147): 56 data bytes
    ping: sending packet: Operation not permitted
    

    Notice that the proxy now blocks the request to https://www.google.com with status code 500.

Cleanup

  1. Delete the sample apps:

    cd ../../
    kubectl delete -f examples/debian-app.yaml
    kubectl delete -f examples/debian-app-locked-manual.yaml
    
  2. Delete the tproxy helm release:

    helm delete --purge tproxy
    
  3. Delete the Kubernetes Engine cluster:

    gcloud container clusters delete tproxy-example --zone=us-central1-f
    

What's next?

See more by @{[ username ]} and more tagged {[ tag ]}{[ $last ? '' : ', ' ]}

Submit a Tutorial

Share step-by-step guides

SUBMIT A TUTORIAL

Request a Tutorial

Ask for community help

SUBMIT A REQUEST

GCP Tutorials

Tutorials published by GCP

VIEW TUTORIALS

Except as otherwise noted, the content of this page is licensed under the Creative Commons Attribution 4.0 License, and code samples are licensed under the Apache 2.0 License. For details, see our Site Policies. Java is a registered trademark of Oracle and/or its affiliates.