Enable kubelet readonly port

Overview

This document shows how to deploy a privileged DaemonSet in each node of GKE on VMware to modify kubelet parameters to enable read-only ports. In version 1.16 and later, the kubelet read-only port is disabled by default.

Prerequisite

Make sure your GKE on VMware is healthy before running the following patch script. You can use this solution to patch 1.16 and later admin clusters and user clusters.

Create a DaemonSet file

Create and save a DaemonSet file named patch.yaml in your current directory, with the following content:

apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: onprem-node-patcher
  namespace: kube-system
spec:
  selector:
    matchLabels:
      name: onprem-node-patcher
  updateStrategy:
    type: RollingUpdate
  template:
    metadata:
      labels:
        name: onprem-node-patcher
    spec:
      tolerations:
      - operator: Exists
      volumes:
      - name: host
        hostPath:
          path: /
      hostPID: true
      initContainers:
      - name: read-only-patcher
        image: "ubuntu"
        env:
        - name: KUBELET_READONLY_PORT
          value: "10255"
        # Number of 1G hugepages. Update the value as desired.
        command:
        - /bin/bash
        - -c
        - |
          set -xeuo pipefail
          configfile="/host/var/lib/kubelet/config.yaml"
          kubeletservice="/host/etc/systemd/system/kubelet.service"
          # $1: The read-only port for the kubelet to serve on with no
          #     authentication/authorization (set to 0 to disable)
          function set-readonly-port-in-config() {
            [[ "$#" -eq 1 ]] || return
            local readonlyport; readonlyport="$1"
            local actual; actual="$(grep readOnlyPort "${configfile}")"
            if [[ "${actual}" == "" ]]; then
              echo "readOnlyPort: ${readonlyport}" >> "${configfile}"
            else
              sed -E -i 's/readOnlyPort: [0-9]+/readOnlyPort: '"${readonlyport}"'/g' ${configfile}
            fi
            echo "Successfully append readOnlyPort: ${readonlyport} to ${configfile}"
          }
          sed -E -i 's/--read-only-port=[0-9]+/--read-only-port='"${KUBELET_READONLY_PORT}"'/g' ${kubeletservice}
          [[ -f ${configfile} ]] && set-readonly-port-in-config "${KUBELET_READONLY_PORT}"
          echo "Restarting kubelet..."
          chroot /host nsenter -a -t1 -- systemctl daemon-reload
          chroot /host nsenter -a -t1 -- systemctl restart kubelet.service
          echo "Success!"
        volumeMounts:
        - name: host
          mountPath: /host
        resources:
          requests:
            memory: 5Mi
            cpu: 5m
        securityContext:
          privileged: true
      containers:
      - image: gcr.io/google-containers/pause:3.2
        name: pause
      # Ensures that the pods will only run on the nodes having the correct
      # label.
      nodeSelector:
        "kubernetes.io/os": "linux"

Update the read-only port number

To change the port number, manually edit the environment variable KUBELET_READONLY_PORT in the DaemonSet YAML.
The default read-only port is 10255, you should not pick 10250 as it will conflict with the predefined secure port.

Patch the admin cluster

   kubectl apply -f patch.yaml \
    --kubeconfig ADMIN_CLUSTER_KUBECONFIG

Patch the user cluster

   kubectl apply -f patch.yaml \
    --kubeconfig USER_CLUSTER_KUBECONFIG

Restore

To disable the read-only port, manually edit the environment variable KUBELET_READONLY_PORT in the DaemonSet YAML.
After you save the changes, the DaemonSet will be re-run to modify the kubelet accordingly.

Caveats

This patch has the same lifecycle as your installed 3P apps. You can run it anytime as a day 2 operation. But it might not persist after you re-create the cluster. To make this change persistent, deploy this DaemonSet as a step in the GKE on VMware post-initialization action.
After running once, the kubelet configuration file should be modified and reloaded. You can safely run kubectl delete -f patch.yaml to clean up DaemonSet resources.
GKE on VMware running on Windows does not support this patch.
Kubernetes does not perform any authentication or authorization checks on this insecure port 10255. Enabling it will leave kubelet data unprotected and exposed to unauthorized users. The kubelet serves the same endpoint on the more secure, authenticated port 10250, consider migrating to that secure port.