Disable Kubelet readonly port

Overview

This document shows how to deploy a privileged daemonset in each node of Google Distributed Cloud to modify kubelet parameters to disable read-only ports.

Prerequisite

Make sure your Google Distributed Cloud is healthy before running the following patch script. This solution can be used to patch both admin clusters and user clusters. And it should work for all Google Distributed Cloud versions.

Save the following Daemonset YAML to your local file (e.g. patch.yaml)

apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: onprem-node-patcher
  namespace: kube-system
spec:
  selector:
    matchLabels:
      name: onprem-node-patcher
  updateStrategy:
    type: RollingUpdate
  template:
    metadata:
      labels:
        name: onprem-node-patcher
    spec:
      tolerations:
      - operator: Exists
      volumes:
      - name: host
        hostPath:
          path: /
      hostPID: true
      initContainers:
      - name: read-only-patcher
        image: "ubuntu"
        env:
        - name: KUBELET_READONLY_PORT
          value: "0"
        # Number of 1G hugepages. Update the value as desired.
        command:
        - /bin/bash
        - -c
        - |
          set -xeuo pipefail
          configfile="/host/var/lib/kubelet/config.yaml"
          kubeletservice="/host/etc/systemd/system/kubelet.service"
          # $1: The read-only port for the kubelet to serve on with no
          #     authentication/authorization (set to 0 to disable)
          function set-readonly-port-in-config() {
            [[ "$#" -eq 1 ]] || return
            local readonlyport; readonlyport="$1"
            local actual; actual="$(grep readOnlyPort "${configfile}")"
            if [[ "${actual}" == "" ]]; then
              echo "readOnlyPort: ${readonlyport}" >> "${configfile}"
            else
              sed -E -i 's/readOnlyPort: [0-9]+/readOnlyPort: 0/g' ${configfile}
            fi
            echo "Successfully append readOnlyPort: ${readonlyport} to ${configfile}"
          }
          sed -E -i 's/--read-only-port=[0-9]+/--read-only-port='"${KUBELET_READONLY_PORT}"'/g' ${kubeletservice}
          [[ -f ${configfile} ]] && set-readonly-port-in-config "${KUBELET_READONLY_PORT}"
          echo "Restarting kubelet..."
          chroot /host nsenter -a -t1 -- systemctl daemon-reload
          chroot /host nsenter -a -t1 -- systemctl restart kubelet.service
          echo "Success!"
        volumeMounts:
        - name: host
          mountPath: /host
        resources:
          requests:
            memory: 5Mi
            cpu: 5m
        securityContext:
          privileged: true
      containers:
      - image: gcr.io/google-containers/pause:3.2
        name: pause
      # Ensures that the pods will only run on the nodes having the correct
      # label.
      nodeSelector:
        "kubernetes.io/os": "linux"

Patch on the admin cluster

   kubectl apply -f patch.yaml \
    --kubeconfig ADMIN_CLUSTER_KUBECONFIG

Patch on the user cluster

   kubectl apply -f patch.yaml \
    --kubeconfig USER_CLUSTER_KUBECONFIG

Restore

  • To re-enable the read-only port, manually edit the environment variable KUBELET_READONLY_PORT in the Daemonset YAML.

  • Once you save the changes, the daemonset will be re-run to modify the kubelet accordingly.

Caveats

  • This patch has the same lifecycle as your installed 3P apps. You can run it anytime as a day 2 operation. But it might not persist after you re-create the cluster. To make this change persistent, deploy this daemonset as a step in the Google Distributed Cloud post-initialization action.

  • After running once, the Kubelet config file should be modified and reloaded. You can safely run kubectl delete -f patch.yaml to clean up daemonset resources.

  • Anthos running on Windows does not currently support this patch.

  • The following metrics are lost for 1.13 and older cluster versions: