Transparent proxy and filtering on Kubernetes
Contributed by Google employees.
There are many application environments and scenarios where you may want to filter and intercept all HTTP and HTTPS traffic out of a pod. Example use cases include isolating external access to specific HTTP or HTTPS paths and methods and in-flight altering of requests.
An example of HTTPS interception is filtering access to Cloud Storage buckets. Most proxy solutions do not support HTTPS inspection, so the filtering that can be
done is limited at the IP/DNS level. For the case of a Cloud Storage bucket, the bucket name is embedded under the
https://storage.googleapis.com/BUCKET_NAME/object
URL, so filtering is limited to all buckets under the storage.googleapis.com
domain, not any specific
bucket.
You can transparently add request and response headers to requests between services for tracing purposes without the services having to explicitly set them.
Mitmproxy is an open-source tool that you can use to intercept and modify HTTP and HTTPS requests transparently using the Python scripting language.
This tutorial uses the tproxy-sidecar container to create firewall rules in the pod network
to block egress traffic out of selected pods. The tproxy-podwatch controller
watches for pod changes containing the "initializer.kubernetes.io/tproxy": "true"
annotation and automatically add and removes the local firewall redirect
rules to apply the transparent proxy to the pod.
Figure 1. transparent proxy architecture diagram
Objectives
- Create a Kubernetes cluster with Google Kubernetes Engine.
- Deploy the tproxy and the tproxy-podwatch pods using Helm.
- Deploy example apps to test external access to a Cloud Storage bucket.
Before you begin
This tutorial assumes you already have a Google Cloud account and are familiar with the high-level concepts of Kubernetes Pods and Deployments.
Costs
This tutorial uses billable components of Google Cloud, including Google Kubernetes Engine.
Use the pricing calculator to estimate the costs for your environment.
Clone the source repository
Open Cloud Shell.
Clone the repository containing the code for this tutorial:
git clone https://github.com/danisla/kubernetes-tproxy cd kubernetes-tproxy
The remainder of this tutorial is run from the root of the cloned repository directory.
Create Kubernetes Engine cluster and install Helm
Create the Kubernetes Engine cluster:
gcloud container clusters create tproxy-example --zone us-central1-f
This command also automatically configures the
kubectl
command to use the cluster.Install Helm in your Cloud Shell instance:
curl -sL https://get.helm.sh/helm-v2.17.0-linux-amd64.tar.gz | tar -xvf - && sudo mv linux-amd64/helm /usr/local/bin/ && rm -Rf linux-amd64
Initialize Helm:
helm init
This installs Tiller—which is the server-side component of Helm—in the Kubernetes cluster. The Tiller pod may take a minute to start.
Verify that the client and server components have been deployed:
helm version
You should see the Client and Server versions in the output.
Install the Helm chart
Before installing the chart, you must first extract the certificates generated by mitmproxy. The generated CA certificate is used in the example pods to trust the proxy when making HTTPS requests.
Extract the generated certificates using Docker:
cd charts/tproxy docker run --rm -v ${PWD}/certs/:/home/mitmproxy/.mitmproxy mitmproxy/mitmproxy >/dev/null 2>&1
Install the chart:
helm install -n tproxy .
The output of this command shows you how to augment your deployments to use the init container and trusted certificate configmap volume. Example output below:
Add the init container spec below to your deployments: initContainers: - name: tproxy image: docker.io/danisla/tproxy-sidecar:0.1.0 imagePullPolicy: IfNotPresent securityContext: privileged: true env: resources: limits: cpu: 500m memory: 128Mi requests: cpu: 100m memory: 64Mi Add the volumes below to your deployments to use the trusted https tproxy: volumes: - name: ca-certs-debian configMap: name: tproxy-tproxy-root-certs items: - key: root-certs.crt path: ca-certificates.crt
Get the status of the DaemonSet pods:
kubectl get pods -o wide
Notice in the example output below that there is a tproxy pod for each node:
NAME READY STATUS RESTARTS AGE IP NODE tproxy-tproxy-2h7lk 2/2 Running 0 21s 10.128.0.8 gke-tproxy-example-default-pool-1e70b38d-xchn tproxy-tproxy-4mvtf 2/2 Running 0 21s 10.128.0.7 gke-tproxy-example-default-pool-1e70b38d-hk89 tproxy-tproxy-ljfq9 2/2 Running 0 21s 10.128.0.6 gke-tproxy-example-default-pool-1e70b38d-jsqd
The tproxy chart is now installed and ready to be used by pods with the init container and pod annotation.
Deploy example apps
Deploy the sample apps to demonstrate using and not using the init container to lock down external access from the pod.
Change directories back to the repository root and deploy the example apps:
cd ../../ kubectl create -f examples/debian-app.yaml kubectl create -f examples/debian-app-locked-manual.yaml
Note that the second deployment is the one that contains the init container and trusted certificate volume mount described in the chart post-installation notes.
Get the logs for the pod without the tproxy annotation:
kubectl logs --selector=app=debian-app,variant=unlocked --tail=10
Example output:
https://www.google.com: 200 https://storage.googleapis.com/solutions-public-assets/: 200 PING www.google.com (209.85.200.105): 56 data bytes 64 bytes from 209.85.200.105: icmp_seq=0 ttl=52 time=0.758 ms
The output from the example app shows the status codes for the requests and the output of a ping command.
Notice the following:
- The request to
https://www.google.com
succeeds with status code 200. - The request to the Cloud Storage bucket succeeds with status code 200.
- The ping to
www.google.com
succeeds.
- The request to
Get the logs for the pod with the tproxy annotation:
kubectl logs --selector=app=debian-app,variant=locked --tail=4
Example output:
https://www.google.com: 418 https://storage.googleapis.com/solutions-public-assets/: 200 PING www.google.com (209.85.200.147): 56 data bytes ping: sending packet: Operation not permitted
Notice the following:
- The proxy blocks the request to
https://www.google.com
with status code 418. - The proxy allows the request to the Cloud Storage bucket with status code 200.
- The ping to
www.google.com
is rejected.
- The proxy blocks the request to
Inspect the logs from the mitmproxy DaemonSet pod to show the intercepted requests and responses:
kubectl logs $(kubectl get pods -o wide | awk '/tproxy.*'$(kubectl get pods --selector=app=debian-app,variant=locked -o=jsonpath={.items..spec.nodeName})'/ {print $1}') -c tproxy-tproxy-mode --tail=10
Note that the logs have to be retrieved from the tproxy pod that is running on the same node as the example app.
Example output:
10.12.1.41:37380: clientconnect 10.12.1.41:37380: GET https://www.google.com/ HTTP/2.0 << 418 I'm a teapot 30b 10.12.1.41:37380: clientdisconnect 10.12.1.41:36496: clientconnect Streaming response from 64.233.191.128 10.12.1.41:36496: GET https://storage.googleapis.com/solutions-public-assets/adtech/dfp_networkimpressions.py HTTP/2.0 << 200 (content missing) 10.12.1.41:36496: clientdisconnect
Notice that the proxy blocks the request to
https://www.google.com
with status code 418.
Customizing the mitmproxy Python script
This tutorial uses a Python script to filter traffic to a specific Cloud Storage bucket. The Python script is installed as a ConfigMap resource, and mitmproxy can live-reload the script when it changes without restarting.
Modify the script to change the "access denied" status code:
cd charts/tproxy/ sed -e 's/418/500/g' config/mitm-script.py
Upgrade the Helm chart with the change:
helm upgrade tproxy .
After about 30 seconds, the new script will be updated and in use by mitmproxy.
Verify the output of the locked pod:
kubectl logs --selector=app=debian-app,variant=locked --tail=4
Example output:
https://www.google.com: 500 https://storage.googleapis.com/solutions-public-assets/: 200 PING www.google.com (209.85.200.147): 56 data bytes ping: sending packet: Operation not permitted
Notice that the proxy now blocks the request to
https://www.google.com
with status code 500.
Cleanup
Delete the sample apps:
cd ../../ kubectl delete -f examples/debian-app.yaml kubectl delete -f examples/debian-app-locked-manual.yaml
Delete the tproxy helm release:
helm delete --purge tproxy
Delete the Kubernetes Engine cluster:
gcloud container clusters delete tproxy-example --zone=us-central1-f
What's next?
- Transparent Proxy and Filtering on Kubernetes with Initializers tutorial: Same approach but simplified using a deployment initializer to inject the InitContainer and ConfigMap.
- tproxy helm chart: See all configuration options and deployment methods.
- Istio: A more broad approach to traffic filtering and network policy.
- Calico Egress NetworkPolicy: Another way to filter egress traffic at the pod level.
Except as otherwise noted, the content of this page is licensed under the Creative Commons Attribution 4.0 License, and code samples are licensed under the Apache 2.0 License. For details, see our Site Policies. Java is a registered trademark of Oracle and/or its affiliates.