CIS Benchmarks

This document introduces the CIS Kubernetes Benchmark, explains how to audit your compliance with the benchmark, and explains what GKE on-prem configures where you cannot implement a recommendation yourself.

Using the CIS Benchmarks

The Center for Internet Security (CIS) releases benchmarks for best practice security recommendations. The CIS Kubernetes Benchmark provides a set of recommendations for configuring Kubernetes to support a strong security posture. The Benchmark is tied to a specific Kubernetes release. The CIS Kubernetes Benchmark is written for the open source Kubernetes distribution and intended to be as universally applicable across distributions as possible.

Versions

Note that the version numbers for different Benchmarks may not be the same.

This document refers to these versions:

Anthos version Kubernetes version CIS Kubernetes Benchmark version
1.4.0 1.16 1.5.1

CIS Kubernetes Benchmark

Accessing the Benchmark

The CIS Kubernetes Benchmark is available on the CIS website.

Recommendation Levels

In the CIS Kubernetes Benchmark,

Level Description
Level 1

Recommendations intend to:

  • be practical and prudent;
  • provide a clear security benefit; and
  • not inhibit the utility of the technology beyond acceptable means.

    		•
    Level 2

    Extends the Level 1 profile.

    Recommendations exhibit one or more of the following characteristics:

  • are intended for environments or use cases where security is paramount;
  • acts as defense in depth measure; or
  • may negatively inhibit the utility or performance of the technology.

    		•

    Recommendation Scoring

    In the CIS Kubernetes Benchmark,

    Scoring Description
    Scored Failure to comply with these recommendations will decrease the final benchmark score.
    Not Scored Failure to comply with these recommendations will not decrease the final benchmark score.

    Evaluation on GKE on-prem

    We use the following values to specify the status of Kubernetes Recommendations in GKE on-prem:

    Status Description
    Pass Complies with a Benchmark Recommendation.
    Fail Does not comply with a Benchmark Recommendation.
    Equivalent Control Does not comply with the exact terms in the Benchmark Recommendation, but other mechanisms in GKE on-prem exist to provide equivalent security controls.
    Depends on Environment GKE on-prem does not configure items related to this Recommendation. The user's configuration determines whether their environment complies with a Benchmark Recommendation.

    GKE on-prem Architecture

    GKE on-prem uses an "admin cluster" to manage one or more "user clusters", which run actual Kubernetes workloads. More information on this architecture can be found in the GKE on-prem overview. The configuration of both admin and user clusters is evaluated against the benchmark below.

    Status on GKE on-prem

    When creating a new GKE on-prem cluster with the specified version, here's how it will perform against the CIS Kubernetes Benchmark. CIS benchmark reports are automatically generated by scripts. The scripts are run on new clusters that aren't running any user workloads. If you install your own workloads and run the scripts, the report may be different.

    Status of GKE on-prem admin cluster:

    # Recommendation Scored/Not Scored Level Status
    1 Master Node Security Configuration
    1.1 Master Node Configuration Files
    1.1.1 Ensure that the API server pod specification file permissions are set to 644 or more restrictive Scored L1 Equivalent Control
    1.1.2 Ensure that the API server pod specification file ownership is set to root:root Scored L1 Equivalent Control
    1.1.3 Ensure that the controller manager pod specification file permissions are set to 644 or more restrictive Scored L1 Equivalent Control
    1.1.4 Ensure that the controller manager pod specification file ownership is set to root:root Scored L1 Equivalent Control
    1.1.5 Ensure that the scheduler pod specification file permissions are set to 644 or more restrictive Scored L1 Equivalent Control
    1.1.6 Ensure that the scheduler pod specification file ownership is set to root:root Scored L1 Equivalent Control
    1.1.7 Ensure that the etcd pod specification file permissions are set to 644 or more restrictive Scored L1 Equivalent Control
    1.1.8 Ensure that the etcd pod specification file ownership is set to root:root Scored L1 Equivalent Control
    1.1.9 Ensure that the Container Network Interface file permissions are set to 644 or more restrictive Not Scored L1 Pass
    1.1.10 Ensure that the Container Network Interface file ownership is set to root:root Not Scored L1 Pass
    1.1.11 Ensure that the etcd data directory permissions are set to 700 or more restrictive Scored L1 Equivalent Control
    1.1.12 Ensure that the etcd data directory ownership is set to etcd:etcd Scored L1 Equivalent Control
    1.1.13 Ensure that the admin.conf file permissions are set to 644 or more restrictive Scored L1 Equivalent Control
    1.1.14 Ensure that the admin.conf file ownership is set to root:root Scored L1 Equivalent Control
    1.1.15 Ensure that the scheduler.conf file permissions are set to 644 or more restrictive Scored L1 Equivalent Control
    1.1.16 Ensure that the scheduler.conf file ownership is set to root:root Scored L1 Equivalent Control
    1.1.17 Ensure that the controller-manager.conf file permissions are set to 644 or more restrictive Scored L1 Equivalent Control
    1.1.18 Ensure that the controller-manager.conf file ownership is set to root:root Scored L1 Equivalent Control
    1.1.19 Ensure that the Kubernetes PKI directory and file ownership is set to root:root Scored L1 Pass
    1.1.20 Ensure that the Kubernetes PKI certificate file permissions are set to 644 or more restrictive Not Scored L1 Pass
    1.1.21 Ensure that the Kubernetes PKI key file permissions are set to 600 Not Scored L1 Pass
    1.2 API Server
    1.2.1 Ensure that the --anonymous-auth argument is set to false Not Scored L1 Warn
    1.2.2 Ensure that the --basic-auth-file argument is not set Scored L1 Pass
    1.2.3 Ensure that the --token-auth-file parameter is not set Scored L1 Pass
    1.2.4 Ensure that the --kubelet-https argument is set to true Scored L1 Pass
    1.2.5 Ensure that the --kubelet-client-certificate and --kubelet-client-key arguments are set as appropriate Scored L1 Pass
    1.2.6 Ensure that the --kubelet-certificate-authority argument is set as appropriate Scored L1 Pass
    1.2.7 Ensure that the --authorization-mode argument is not set to AlwaysAllow Scored L1 Pass
    1.2.8 Ensure that the --authorization-mode argument includes Node Scored L1 Pass
    1.2.9 Ensure that the --authorization-mode argument includes RBAC Scored L1 Pass
    1.2.10 Ensure that the admission control plugin EventRateLimit is set Not Scored L1 Warn
    1.2.11 Ensure that the admission control plugin AlwaysAdmit is not set Scored L1 Pass
    1.2.12 Ensure that the admission control plugin AlwaysPullImages is set Not Scored L1 Warn
    1.2.13 Ensure that the admission control plugin SecurityContextDeny is set if PodSecurityPolicy is not used Not Scored L1 Equivalent Control
    1.2.14 Ensure that the admission control plugin ServiceAccount is set Scored L1 Pass
    1.2.15 Ensure that the admission control plugin NamespaceLifecycle is set Scored L1 Pass
    1.2.16 Ensure that the admission control plugin PodSecurityPolicy is set Scored L1 Equivalent Control
    1.2.17 Ensure that the admission control plugin NodeRestriction is set Scored L1 Pass
    1.2.18 Ensure that the --insecure-bind-address argument is not set Scored L1 Pass
    1.2.19 Ensure that the --insecure-port argument is set to 0 Scored L1 Pass
    1.2.20 Ensure that the --secure-port argument is not set to 0 Scored L1 Pass
    1.2.21 Ensure that the --profiling argument is set to false Scored L1 Pass
    1.2.22 Ensure that the --audit-log-path argument is set Scored L1 Equivalent Control
    1.2.23 Ensure that the --audit-log-maxage argument is set to 30 or as appropriate Scored L1 Equivalent Control
    1.2.24 Ensure that the --audit-log-maxbackup argument is set to 10 or as appropriate Scored L1 Equivalent Control
    1.2.25 Ensure that the --audit-log-maxsize argument is set to 100 or as appropriate Scored L1 Equivalent Control
    1.2.26 Ensure that the --request-timeout argument is set as appropriate Scored L1 Pass
    1.2.27 Ensure that the --service-account-lookup argument is set to true Scored L1 Pass
    1.2.28 Ensure that the --service-account-key-file argument is set as appropriate Scored L1 Pass
    1.2.29 Ensure that the --etcd-certfile and --etcd-keyfile arguments are set as appropriate Scored L1 Pass
    1.2.30 Ensure that the --tls-cert-file and --tls-private-key-file arguments are set as appropriate Scored L1 Equivalent Control
    1.2.31 Ensure that the --client-ca-file argument is set as appropriate Scored L1 Pass
    1.2.32 Ensure that the --etcd-cafile argument is set as appropriate Scored L1 Pass
    1.2.33 Ensure that the --encryption-provider-config argument is set as appropriate Not Scored L1 Warn
    1.2.34 Ensure that encryption providers are appropriately configured Not Scored L1 Warn
    1.2.35 Ensure that the API Server only makes use of Strong Cryptographic Ciphers Not Scored L1 Pass
    1.3 Controller Manager
    1.3.1 Ensure that the --terminated-pod-gc-threshold argument is set as appropriate Not Scored L1 Pass
    1.3.2 Ensure that the --profiling argument is set to false Scored L1 Pass
    1.3.3 Ensure that the --use-service-account-credentials argument is set to true Scored L1 Pass
    1.3.4 Ensure that the --service-account-private-key-file argument is set as appropriate Scored L1 Pass
    1.3.5 Ensure that the --root-ca-file argument is set as appropriate Scored L1 Pass
    1.3.6 Ensure that the RotateKubeletServerCertificate argument is set to true Scored L2 Equivalent Control
    1.3.7 Ensure that the --bind-address argument is set to 127.0.0.1 Scored L1 Pass
    1.4 Scheduler
    1.4.1 Ensure that the --profiling argument is set to false Scored L1 Pass
    1.4.2 Ensure that the --bind-address argument is set to 127.0.0.1 Scored L1 Pass
    2 Etcd Node Configuration
    2 Etcd Node Configuration Files
    2.1 Ensure that the --cert-file and --key-file arguments are set as appropriate Scored L1 Pass
    2.2 Ensure that the --client-cert-auth argument is set to true Scored L1 Pass
    2.3 Ensure that the --auto-tls argument is not set to true Scored L1 Pass
    2.4 Ensure that the --peer-cert-file and --peer-key-file arguments are set as appropriate Scored L1 Pass
    2.5 Ensure that the --peer-client-cert-auth argument is set to true Scored L1 Pass
    2.6 Ensure that the --peer-auto-tls argument is not set to true Scored L1 Pass
    2.7 Ensure that a unique Certificate Authority is used for etcd Not Scored L2 Pass
    3 Control Plane Configuration
    3.1 Authentication and Authorization
    3.1.1 Client certificate authentication should not be used for users Not Scored L2 Warn
    3.2 Logging
    3.2.1 Ensure that a minimal audit policy is created Scored L1 Equivalent Control
    3.2.2 Ensure that the audit policy covers key security concerns Not Scored L2 Equivalent Control
    4 Worker Node Security Configuration
    4.1 Worker Node Configuration Files
    4.1.1 Ensure that the kubelet service file permissions are set to 644 or more restrictive Scored L1 Pass
    4.1.2 Ensure that the kubelet service file ownership is set to root:root Scored L1 Pass
    4.1.3 Ensure that the proxy kubeconfig file permissions are set to 644 or more restrictive Scored L1 Equivalent Control
    4.1.4 Ensure that the proxy kubeconfig file ownership is set to root:root Scored L1 Equivalent Control
    4.1.5 Ensure that the kubelet.conf file permissions are set to 644 or more restrictive Scored L1 Equivalent Control
    4.1.6 Ensure that the kubelet.conf file ownership is set to root:root Scored L1 Equivalent Control
    4.1.7 Ensure that the certificate authorities file permissions are set to 644 or more restrictive Scored L1 Pass
    4.1.8 Ensure that the client certificate authorities file ownership is set to root:root Scored L1 Pass
    4.1.9 Ensure that the kubelet configuration file has permissions set to 644 or more restrictive Scored L1 Pass
    4.1.10 Ensure that the kubelet configuration file ownership is set to root:root Scored L1 Pass
    4.2 Kubelet
    4.2.1 Ensure that the anonymous-auth argument is set to false Scored L1 Pass
    4.2.2 Ensure that the --authorization-mode argument is not set to AlwaysAllow Scored L1 Pass
    4.2.3 Ensure that the --client-ca-file argument is set as appropriate Scored L1 Pass
    4.2.4 Ensure that the --read-only-port argument is set to 0 Scored L1 Fail
    4.2.5 Ensure that the --streaming-connection-idle-timeout argument is not set to 0 Scored L1 Pass
    4.2.6 Ensure that the --protect-kernel-defaults argument is set to true Scored L1 Fail
    4.2.7 Ensure that the --make-iptables-util-chains argument is set to true Scored L1 Pass
    4.2.8 Ensure that the --hostname-override argument is not set Not Scored L1 Pass
    4.2.9 Ensure that the --event-qps argument is set to 0 or a level which ensures appropriate event capture Not Scored L2 Warn
    4.2.10 Ensure that the --tls-cert-file and --tls-private-key-file arguments are set as appropriate Scored L1 Equivalent Control
    4.2.11 Ensure that the --rotate-certificates argument is not set to false Scored L1 Pass
    4.2.12 Ensure that the RotateKubeletServerCertificate argument is set to true Scored L1 Equivalent Control
    4.2.13 Ensure that the Kubelet only makes use of Strong Cryptographic Ciphers Not Scored L1 Warn
    Descriptions of Failures and Equivalent Controls for GKE on-prem admin cluster:
    # Recommendation Scored/Not Scored Level Status Value Justification
    1.1.1 Ensure that the API server pod specification file permissions are set to 644 or more restrictive Scored L1 Equivalent Control Control Plane pod specifications are protected by a similarly restrictive RBAC authorization policy in the Admin Cluster and aren't present as files on disk.
    1.1.2 Ensure that the API server pod specification file ownership is set to root:root Scored L1 Equivalent Control Control Plane pod specifications are protected by a similarly restrictive RBAC authorization policy in the Admin Cluster and aren't present as files on disk.
    1.1.3 Ensure that the controller manager pod specification file permissions are set to 644 or more restrictive Scored L1 Equivalent Control Control Plane pod specifications are protected by a similarly restrictive RBAC authorization policy in the Admin Cluster and aren't present as files on disk.
    1.1.4 Ensure that the controller manager pod specification file ownership is set to root:root Scored L1 Equivalent Control Control Plane pod specifications are protected by a similarly restrictive RBAC authorization policy in the Admin Cluster and aren't present as files on disk.
    1.1.5 Ensure that the scheduler pod specification file permissions are set to 644 or more restrictive Scored L1 Equivalent Control Control Plane pod specifications are protected by a similarly restrictive RBAC authorization policy in the Admin Cluster and aren't present as files on disk.
    1.1.6 Ensure that the scheduler pod specification file ownership is set to root:root Scored L1 Equivalent Control Control Plane pod specifications are protected by a similarly restrictive RBAC authorization policy in the Admin Cluster and aren't present as files on disk.
    1.1.7 Ensure that the etcd pod specification file permissions are set to 644 or more restrictive Scored L1 Equivalent Control Control Plane pod specifications are protected by a similarly restrictive RBAC authorization policy in the Admin Cluster and aren't present as files on disk.
    1.1.8 Ensure that the etcd pod specification file ownership is set to root:root Scored L1 Equivalent Control Control Plane pod specifications are protected by a similarly restrictive RBAC authorization policy in the Admin Cluster and aren't present as files on disk.
    1.1.11 Ensure that the etcd data directory permissions are set to 700 or more restrictive Scored L1 Equivalent Control 755 The etcd data directory has the default 755 permissions but its subdirectories are 700.
    1.1.12 Ensure that the etcd data directory ownership is set to etcd:etcd Scored L1 Equivalent Control root:root The etcd container runs as root and the etcd data directory is owned by root:root.
    1.1.13 Ensure that the admin.conf file permissions are set to 644 or more restrictive Scored L1 Equivalent Control Control Plane pod specifications are protected by a similarly restrictive RBAC authorization policy in the Admin Cluster and aren't present as files on disk.
    1.1.14 Ensure that the admin.conf file ownership is set to root:root Scored L1 Equivalent Control Control Plane pod specifications are protected by a similarly restrictive RBAC authorization policy in the Admin Cluster and aren't present as files on disk.
    1.1.15 Ensure that the scheduler.conf file permissions are set to 644 or more restrictive Scored L1 Equivalent Control Control Plane pod specifications are protected by a similarly restrictive RBAC authorization policy in the Admin Cluster and aren't present as files on disk.
    1.1.16 Ensure that the scheduler.conf file ownership is set to root:root Scored L1 Equivalent Control Control Plane pod specifications are protected by a similarly restrictive RBAC authorization policy in the Admin Cluster and aren't present as files on disk.
    1.1.17 Ensure that the controller-manager.conf file permissions are set to 644 or more restrictive Scored L1 Equivalent Control Control Plane pod specifications are protected by a similarly restrictive RBAC authorization policy in the Admin Cluster and aren't present as files on disk.
    1.1.18 Ensure that the controller-manager.conf file ownership is set to root:root Scored L1 Equivalent Control Control Plane pod specifications are protected by a similarly restrictive RBAC authorization policy in the Admin Cluster and aren't present as files on disk.
    1.2.1 Ensure that the --anonymous-auth argument is set to false Not Scored L1 Warn not set Some GKE on-prem monitoring components use anonymous authentication to perform health-checks.
    1.2.10 Ensure that the admission control plugin EventRateLimit is set Not Scored L1 Warn not set GKE on-prem does not support the Event Rate Limit admission controller as it is a Kubernetes Alpha feature.
    1.2.12 Ensure that the admission control plugin AlwaysPullImages is set Not Scored L1 Warn not set The AlwaysPullImages admission controller provides some protection for private registry images in noncooperative multitenant clusters, at the cost of making container registries a single-point-of-failure for creating new Pods across the entire cluster. GKE on-prem does not enable the AlwaysPullImages admission controller, which leaves it up to cluster admins to implement admission policy to make this tradeoff for themselves.
    1.2.13 Ensure that the admission control plugin SecurityContextDeny is set if PodSecurityPolicy is not used Not Scored L1 Equivalent Control not set Policies about security settings in Pods are managed using Policy Controller and Anthos Config Management.
    1.2.16 Ensure that the admission control plugin PodSecurityPolicy is set Scored L1 Equivalent Control not set Policies about security settings in Pods are managed using Policy Controller and Anthos Config Management.
    1.2.22 Ensure that the --audit-log-path argument is set Scored L1 Equivalent Control not set GKE on-prem captures audit logs but does not use these flags for auditing. See GKE on-prem Audit policy for more details.
    1.2.23 Ensure that the --audit-log-maxage argument is set to 30 or as appropriate Scored L1 Equivalent Control not set GKE on-prem captures audit logs but does not use these flags for auditing. See GKE on-prem Audit policy for more details.
    1.2.24 Ensure that the --audit-log-maxbackup argument is set to 10 or as appropriate Scored L1 Equivalent Control not set GKE on-prem captures audit logs but does not use these flags for auditing. See GKE on-prem Audit policy for more details.
    1.2.25 Ensure that the --audit-log-maxsize argument is set to 100 or as appropriate Scored L1 Equivalent Control not set GKE on-prem captures audit logs but does not use these flags for auditing. See GKE on-prem Audit policy for more details.
    1.2.30 Ensure that the --tls-cert-file and --tls-private-key-file arguments are set as appropriate Scored L1 Equivalent Control not set GKE on-prem manages kubelet server TLS using the --rotate-server-certificates flag.
    1.2.33 Ensure that the --encryption-provider-config argument is set as appropriate Not Scored L1 Warn not set GKE on-prem does not yet support Application Layer Encryption of Secrets.
    1.2.34 Ensure that encryption providers are appropriately configured Not Scored L1 Warn not set GKE on-prem does not yet support Application Layer Encryption of Secrets.
    1.3.6 Ensure that the RotateKubeletServerCertificate argument is set to true Scored L2 Equivalent Control not set GKE on-prem enables Kubelet certificate rotation by setting --rotate-server-certificates=true on kubelets.
    3.1.1 Client certificate authentication should not be used for users Not Scored L2 Warn GKE on-prem generates admin client certificate credentials for bootstrapping permissions for users.
    3.2.1 Ensure that a minimal audit policy is created Scored L1 Equivalent Control not set GKE on-prem captures audit logs but does not use these flags for auditing. See GKE on-prem Audit policy for more details.
    3.2.2 Ensure that the audit policy covers key security concerns Not Scored L2 Equivalent Control not set GKE on-prem captures audit logs but does not use these flags for auditing. See GKE on-prem Audit policy for more details.
    4.1.3 Ensure that the proxy kubeconfig file permissions are set to 644 or more restrictive Scored L1 Equivalent Control not set kube-proxy runs as a daemonset in GKE on-prem and permissions are managed through RBAC.
    4.1.4 Ensure that the proxy kubeconfig file ownership is set to root:root Scored L1 Equivalent Control not set kube-proxy runs as a daemonset in GKE on-prem and permissions are managed through RBAC.
    4.1.5 Ensure that the kubelet.conf file permissions are set to 644 or more restrictive Scored L1 Equivalent Control Control Plane pod specifications are protected by a similarly restrictive RBAC authorization policy in the Admin Cluster and aren't present as files on disk.
    4.1.6 Ensure that the kubelet.conf file ownership is set to root:root Scored L1 Equivalent Control Control Plane pod specifications are protected by a similarly restrictive RBAC authorization policy in the Admin Cluster and aren't present as files on disk.
    4.2.4 Ensure that the --read-only-port argument is set to 0 Scored L1 Fail 10255 Some GKE on-prem monitoring components use the kubelet read-only port to obtain metrics.
    4.2.6 Ensure that the --protect-kernel-defaults argument is set to true Scored L1 Fail false GKE on-prem allows kubelet to set its necessary kernel settings.
    4.2.9 Ensure that the --event-qps argument is set to 0 or a level which ensures appropriate event capture Not Scored L2 Warn not set Events are Kubernetes objects stored in etcd. To avoid overwhelming etcd they are only kept for one hour, and are not an appropriate security auditing mechanism. Allowing unlimited events as suggested in this control exposes the cluster to unnecessary DoS risk and contradicts the recommendation to use admission EventRateLimits. Security relevant events that need permanent storage should be sent to logs.
    4.2.10 Ensure that the --tls-cert-file and --tls-private-key-file arguments are set as appropriate Scored L1 Equivalent Control not set GKE on-prem manages kubelet server TLS using the --rotate-server-certificates flag.
    4.2.12 Ensure that the RotateKubeletServerCertificate argument is set to true Scored L1 Equivalent Control not set GKE on-prem enables Kubelet certificate rotation by setting --rotate-server-certificates=true on kubelets.
    4.2.13 Ensure that the Kubelet only makes use of Strong Cryptographic Ciphers Not Scored L1 Warn Kubelet uses the default set of Cryptographic Ciphers

    Status of GKE on-prem user cluster:

    # Recommendation Scored/Not Scored Level Status
    1 Master Node Security Configuration
    1.1 Master Node Configuration Files
    1.1.1 Ensure that the API server pod specification file permissions are set to 644 or more restrictive Scored L1 Equivalent Control
    1.1.2 Ensure that the API server pod specification file ownership is set to root:root Scored L1 Equivalent Control
    1.1.3 Ensure that the controller manager pod specification file permissions are set to 644 or more restrictive Scored L1 Equivalent Control
    1.1.4 Ensure that the controller manager pod specification file ownership is set to root:root Scored L1 Equivalent Control
    1.1.5 Ensure that the scheduler pod specification file permissions are set to 644 or more restrictive Scored L1 Equivalent Control
    1.1.6 Ensure that the scheduler pod specification file ownership is set to root:root Scored L1 Equivalent Control
    1.1.7 Ensure that the etcd pod specification file permissions are set to 644 or more restrictive Scored L1 Equivalent Control
    1.1.8 Ensure that the etcd pod specification file ownership is set to root:root Scored L1 Equivalent Control
    1.1.9 Ensure that the Container Network Interface file permissions are set to 644 or more restrictive Not Scored L1 Pass
    1.1.10 Ensure that the Container Network Interface file ownership is set to root:root Not Scored L1 Pass
    1.1.11 Ensure that the etcd data directory permissions are set to 700 or more restrictive Scored L1 Equivalent Control
    1.1.12 Ensure that the etcd data directory ownership is set to etcd:etcd Scored L1 Equivalent Control
    1.1.13 Ensure that the admin.conf file permissions are set to 644 or more restrictive Scored L1 Equivalent Control
    1.1.14 Ensure that the admin.conf file ownership is set to root:root Scored L1 Equivalent Control
    1.1.15 Ensure that the scheduler.conf file permissions are set to 644 or more restrictive Scored L1 Equivalent Control
    1.1.16 Ensure that the scheduler.conf file ownership is set to root:root Scored L1 Equivalent Control
    1.1.17 Ensure that the controller-manager.conf file permissions are set to 644 or more restrictive Scored L1 Equivalent Control
    1.1.18 Ensure that the controller-manager.conf file ownership is set to root:root Scored L1 Equivalent Control
    1.1.19 Ensure that the Kubernetes PKI directory and file ownership is set to root:root Scored L1 Pass
    1.1.20 Ensure that the Kubernetes PKI certificate file permissions are set to 644 or more restrictive Not Scored L1 Pass
    1.1.21 Ensure that the Kubernetes PKI key file permissions are set to 600 Not Scored L1 Pass
    1.2 API Server
    1.2.1 Ensure that the --anonymous-auth argument is set to false Not Scored L1 Warn
    1.2.2 Ensure that the --basic-auth-file argument is not set Scored L1 Pass
    1.2.3 Ensure that the --token-auth-file parameter is not set Scored L1 Pass
    1.2.4 Ensure that the --kubelet-https argument is set to true Scored L1 Pass
    1.2.5 Ensure that the --kubelet-client-certificate and --kubelet-client-key arguments are set as appropriate Scored L1 Pass
    1.2.6 Ensure that the --kubelet-certificate-authority argument is set as appropriate Scored L1 Pass
    1.2.7 Ensure that the --authorization-mode argument is not set to AlwaysAllow Scored L1 Pass
    1.2.8 Ensure that the --authorization-mode argument includes Node Scored L1 Pass
    1.2.9 Ensure that the --authorization-mode argument includes RBAC Scored L1 Pass
    1.2.10 Ensure that the admission control plugin EventRateLimit is set Not Scored L1 Warn
    1.2.11 Ensure that the admission control plugin AlwaysAdmit is not set Scored L1 Pass
    1.2.12 Ensure that the admission control plugin AlwaysPullImages is set Not Scored L1 Warn
    1.2.13 Ensure that the admission control plugin SecurityContextDeny is set if PodSecurityPolicy is not used Not Scored L1 Equivalent Control
    1.2.14 Ensure that the admission control plugin ServiceAccount is set Scored L1 Pass
    1.2.15 Ensure that the admission control plugin NamespaceLifecycle is set Scored L1 Pass
    1.2.16 Ensure that the admission control plugin PodSecurityPolicy is set Scored L1 Equivalent Control
    1.2.17 Ensure that the admission control plugin NodeRestriction is set Scored L1 Pass
    1.2.18 Ensure that the --insecure-bind-address argument is not set Scored L1 Fail
    1.2.19 Ensure that the --insecure-port argument is set to 0 Scored L1 Fail
    1.2.20 Ensure that the --secure-port argument is not set to 0 Scored L1 Pass
    1.2.21 Ensure that the --profiling argument is set to false Scored L1 Pass
    1.2.22 Ensure that the --audit-log-path argument is set Scored L1 Equivalent Control
    1.2.23 Ensure that the --audit-log-maxage argument is set to 30 or as appropriate Scored L1 Equivalent Control
    1.2.24 Ensure that the --audit-log-maxbackup argument is set to 10 or as appropriate Scored L1 Equivalent Control
    1.2.25 Ensure that the --audit-log-maxsize argument is set to 100 or as appropriate Scored L1 Equivalent Control
    1.2.26 Ensure that the --request-timeout argument is set as appropriate Scored L1 Pass
    1.2.27 Ensure that the --service-account-lookup argument is set to true Scored L1 Pass
    1.2.28 Ensure that the --service-account-key-file argument is set as appropriate Scored L1 Pass
    1.2.29 Ensure that the --etcd-certfile and --etcd-keyfile arguments are set as appropriate Scored L1 Pass
    1.2.30 Ensure that the --tls-cert-file and --tls-private-key-file arguments are set as appropriate Scored L1 Equivalent Control
    1.2.31 Ensure that the --client-ca-file argument is set as appropriate Scored L1 Pass
    1.2.32 Ensure that the --etcd-cafile argument is set as appropriate Scored L1 Pass
    1.2.33 Ensure that the --encryption-provider-config argument is set as appropriate Not Scored L1 Warn
    1.2.34 Ensure that encryption providers are appropriately configured Not Scored L1 Warn
    1.2.35 Ensure that the API Server only makes use of Strong Cryptographic Ciphers Not Scored L1 Pass
    1.3 Controller Manager
    1.3.1 Ensure that the --terminated-pod-gc-threshold argument is set as appropriate Not Scored L1 Pass
    1.3.2 Ensure that the --profiling argument is set to false Scored L1 Pass
    1.3.3 Ensure that the --use-service-account-credentials argument is set to true Scored L1 Pass
    1.3.4 Ensure that the --service-account-private-key-file argument is set as appropriate Scored L1 Pass
    1.3.5 Ensure that the --root-ca-file argument is set as appropriate Scored L1 Pass
    1.3.6 Ensure that the RotateKubeletServerCertificate argument is set to true Scored L2 Equivalent Control
    1.3.7 Ensure that the --bind-address argument is set to 127.0.0.1 Scored L1 Pass
    1.4 Scheduler
    1.4.1 Ensure that the --profiling argument is set to false Scored L1 Pass
    1.4.2 Ensure that the --bind-address argument is set to 127.0.0.1 Scored L1 Pass
    2 Etcd Node Configuration
    2 Etcd Node Configuration Files
    2.1 Ensure that the --cert-file and --key-file arguments are set as appropriate Scored L1 Pass
    2.2 Ensure that the --client-cert-auth argument is set to true Scored L1 Pass
    2.3 Ensure that the --auto-tls argument is not set to true Scored L1 Pass
    2.4 Ensure that the --peer-cert-file and --peer-key-file arguments are set as appropriate Scored L1 Pass
    2.5 Ensure that the --peer-client-cert-auth argument is set to true Scored L1 Pass
    2.6 Ensure that the --peer-auto-tls argument is not set to true Scored L1 Pass
    2.7 Ensure that a unique Certificate Authority is used for etcd Not Scored L2 Pass
    3 Control Plane Configuration
    3.1 Authentication and Authorization
    3.1.1 Client certificate authentication should not be used for users Not Scored L2 Warn
    3.2 Logging
    3.2.1 Ensure that a minimal audit policy is created Scored L1 Equivalent Control
    3.2.2 Ensure that the audit policy covers key security concerns Not Scored L2 Equivalent Control
    4 Worker Node Security Configuration
    4.1 Worker Node Configuration Files
    4.1.1 Ensure that the kubelet service file permissions are set to 644 or more restrictive Scored L1 Pass
    4.1.2 Ensure that the kubelet service file ownership is set to root:root Scored L1 Pass
    4.1.3 Ensure that the proxy kubeconfig file permissions are set to 644 or more restrictive Scored L1 Equivalent Control
    4.1.4 Ensure that the proxy kubeconfig file ownership is set to root:root Scored L1 Equivalent Control
    4.1.5 Ensure that the kubelet.conf file permissions are set to 644 or more restrictive Scored L1 Equivalent Control
    4.1.6 Ensure that the kubelet.conf file ownership is set to root:root Scored L1 Equivalent Control
    4.1.7 Ensure that the certificate authorities file permissions are set to 644 or more restrictive Scored L1 Pass
    4.1.8 Ensure that the client certificate authorities file ownership is set to root:root Scored L1 Pass
    4.1.9 Ensure that the kubelet configuration file has permissions set to 644 or more restrictive Scored L1 Pass
    4.1.10 Ensure that the kubelet configuration file ownership is set to root:root Scored L1 Pass
    4.2 Kubelet
    4.2.1 Ensure that the anonymous-auth argument is set to false Scored L1 Pass
    4.2.2 Ensure that the --authorization-mode argument is not set to AlwaysAllow Scored L1 Pass
    4.2.3 Ensure that the --client-ca-file argument is set as appropriate Scored L1 Pass
    4.2.4 Ensure that the --read-only-port argument is set to 0 Scored L1 Fail
    4.2.5 Ensure that the --streaming-connection-idle-timeout argument is not set to 0 Scored L1 Pass
    4.2.6 Ensure that the --protect-kernel-defaults argument is set to true Scored L1 Fail
    4.2.7 Ensure that the --make-iptables-util-chains argument is set to true Scored L1 Pass
    4.2.8 Ensure that the --hostname-override argument is not set Not Scored L1 Pass
    4.2.9 Ensure that the --event-qps argument is set to 0 or a level which ensures appropriate event capture Not Scored L2 Warn
    4.2.10 Ensure that the --tls-cert-file and --tls-private-key-file arguments are set as appropriate Scored L1 Equivalent Control
    4.2.11 Ensure that the --rotate-certificates argument is not set to false Scored L1 Pass
    4.2.12 Ensure that the RotateKubeletServerCertificate argument is set to true Scored L1 Equivalent Control
    4.2.13 Ensure that the Kubelet only makes use of Strong Cryptographic Ciphers Not Scored L1 Warn
    Descriptions of Failures and Equivalent Controls for GKE on-prem user cluster:
    # Recommendation Scored/Not Scored Level Status Value Justification
    1.1.1 Ensure that the API server pod specification file permissions are set to 644 or more restrictive Scored L1 Equivalent Control Control Plane pod specifications are protected by a similarly restrictive RBAC authorization policy in the Admin Cluster and aren't present as files on disk.
    1.1.2 Ensure that the API server pod specification file ownership is set to root:root Scored L1 Equivalent Control Control Plane pod specifications are protected by a similarly restrictive RBAC authorization policy in the Admin Cluster and aren't present as files on disk.
    1.1.3 Ensure that the controller manager pod specification file permissions are set to 644 or more restrictive Scored L1 Equivalent Control Control Plane pod specifications are protected by a similarly restrictive RBAC authorization policy in the Admin Cluster and aren't present as files on disk.
    1.1.4 Ensure that the controller manager pod specification file ownership is set to root:root Scored L1 Equivalent Control Control Plane pod specifications are protected by a similarly restrictive RBAC authorization policy in the Admin Cluster and aren't present as files on disk.
    1.1.5 Ensure that the scheduler pod specification file permissions are set to 644 or more restrictive Scored L1 Equivalent Control Control Plane pod specifications are protected by a similarly restrictive RBAC authorization policy in the Admin Cluster and aren't present as files on disk.
    1.1.6 Ensure that the scheduler pod specification file ownership is set to root:root Scored L1 Equivalent Control Control Plane pod specifications are protected by a similarly restrictive RBAC authorization policy in the Admin Cluster and aren't present as files on disk.
    1.1.7 Ensure that the etcd pod specification file permissions are set to 644 or more restrictive Scored L1 Equivalent Control Control Plane pod specifications are protected by a similarly restrictive RBAC authorization policy in the Admin Cluster and aren't present as files on disk.
    1.1.8 Ensure that the etcd pod specification file ownership is set to root:root Scored L1 Equivalent Control Control Plane pod specifications are protected by a similarly restrictive RBAC authorization policy in the Admin Cluster and aren't present as files on disk.
    1.1.11 Ensure that the etcd data directory permissions are set to 700 or more restrictive Scored L1 Equivalent Control 755 The etcd data directory has the default 755 permissions but its subdirectories are 700.
    1.1.12 Ensure that the etcd data directory ownership is set to etcd:etcd Scored L1 Equivalent Control root:root The etcd container runs as root and the etcd data directory is owned by root:root.
    1.1.13 Ensure that the admin.conf file permissions are set to 644 or more restrictive Scored L1 Equivalent Control Control Plane pod specifications are protected by a similarly restrictive RBAC authorization policy in the Admin Cluster and aren't present as files on disk.
    1.1.14 Ensure that the admin.conf file ownership is set to root:root Scored L1 Equivalent Control Control Plane pod specifications are protected by a similarly restrictive RBAC authorization policy in the Admin Cluster and aren't present as files on disk.
    1.1.15 Ensure that the scheduler.conf file permissions are set to 644 or more restrictive Scored L1 Equivalent Control Control Plane pod specifications are protected by a similarly restrictive RBAC authorization policy in the Admin Cluster and aren't present as files on disk.
    1.1.16 Ensure that the scheduler.conf file ownership is set to root:root Scored L1 Equivalent Control Control Plane pod specifications are protected by a similarly restrictive RBAC authorization policy in the Admin Cluster and aren't present as files on disk.
    1.1.17 Ensure that the controller-manager.conf file permissions are set to 644 or more restrictive Scored L1 Equivalent Control Control Plane pod specifications are protected by a similarly restrictive RBAC authorization policy in the Admin Cluster and aren't present as files on disk.
    1.1.18 Ensure that the controller-manager.conf file ownership is set to root:root Scored L1 Equivalent Control Control Plane pod specifications are protected by a similarly restrictive RBAC authorization policy in the Admin Cluster and aren't present as files on disk.
    1.2.1 Ensure that the --anonymous-auth argument is set to false Not Scored L1 Warn not set Some GKE on-prem monitoring components use anonymous authentication to perform health-checks.
    1.2.10 Ensure that the admission control plugin EventRateLimit is set Not Scored L1 Warn not set GKE on-prem does not support the Event Rate Limit admission controller as it is a Kubernetes Alpha feature.
    1.2.12 Ensure that the admission control plugin AlwaysPullImages is set Not Scored L1 Warn not set The AlwaysPullImages admission controller provides some protection for private registry images in noncooperative multitenant clusters, at the cost of making container registries a single-point-of-failure for creating new Pods across the entire cluster. GKE on-prem does not enable the AlwaysPullImages admission controller, which leaves it up to cluster admins to implement admission policy to make this tradeoff for themselves.
    1.2.13 Ensure that the admission control plugin SecurityContextDeny is set if PodSecurityPolicy is not used Not Scored L1 Equivalent Control not set Policies about security settings in Pods are managed using Policy Controller and Anthos Config Management.
    1.2.16 Ensure that the admission control plugin PodSecurityPolicy is set Scored L1 Equivalent Control not set Policies about security settings in Pods are managed using Policy Controller and Anthos Config Management.
    1.2.18 Ensure that the --insecure-bind-address argument is not set Scored L1 Fail 127.0.0.1 The metrics-proxy for user cluster kube-apiserver currently communicates over localhost within the kube-apiserver pod using HTTP.
    1.2.19 Ensure that the --insecure-port argument is set to 0 Scored L1 Fail 80 The metrics-proxy for user cluster kube-apiserver currently communicates over localhost within the kube-apiserver pod using HTTP.
    1.2.22 Ensure that the --audit-log-path argument is set Scored L1 Equivalent Control not set GKE on-prem captures audit logs but does not use these flags for auditing. See GKE on-prem Audit policy for more details.
    1.2.23 Ensure that the --audit-log-maxage argument is set to 30 or as appropriate Scored L1 Equivalent Control not set GKE on-prem captures audit logs but does not use these flags for auditing. See GKE on-prem Audit policy for more details.
    1.2.24 Ensure that the --audit-log-maxbackup argument is set to 10 or as appropriate Scored L1 Equivalent Control not set GKE on-prem captures audit logs but does not use these flags for auditing. See GKE on-prem Audit policy for more details.
    1.2.25 Ensure that the --audit-log-maxsize argument is set to 100 or as appropriate Scored L1 Equivalent Control not set GKE on-prem captures audit logs but does not use these flags for auditing. See GKE on-prem Audit policy for more details.
    1.2.30 Ensure that the --tls-cert-file and --tls-private-key-file arguments are set as appropriate Scored L1 Equivalent Control not set GKE on-prem manages kubelet server TLS using the --rotate-server-certificates flag.
    1.2.33 Ensure that the --encryption-provider-config argument is set as appropriate Not Scored L1 Warn not set GKE on-prem does not yet support Application Layer Encryption of Secrets.
    1.2.34 Ensure that encryption providers are appropriately configured Not Scored L1 Warn not set GKE on-prem does not yet support Application Layer Encryption of Secrets.
    1.3.6 Ensure that the RotateKubeletServerCertificate argument is set to true Scored L2 Equivalent Control not set GKE on-prem enables Kubelet certificate rotation by setting --rotate-server-certificates=true on kubelets.
    3.1.1 Client certificate authentication should not be used for users Not Scored L2 Warn GKE on-prem generates admin client certificate credentials for bootstrapping permissions for users.
    3.2.1 Ensure that a minimal audit policy is created Scored L1 Equivalent Control not set GKE on-prem captures audit logs but does not use these flags for auditing. See GKE on-prem Audit policy for more details.
    3.2.2 Ensure that the audit policy covers key security concerns Not Scored L2 Equivalent Control not set GKE on-prem captures audit logs but does not use these flags for auditing. See GKE on-prem Audit policy for more details.
    4.1.3 Ensure that the proxy kubeconfig file permissions are set to 644 or more restrictive Scored L1 Equivalent Control not set kube-proxy runs as a daemonset in GKE on-prem and permissions are managed through RBAC.
    4.1.4 Ensure that the proxy kubeconfig file ownership is set to root:root Scored L1 Equivalent Control not set kube-proxy runs as a daemonset in GKE on-prem and permissions are managed through RBAC.
    4.1.5 Ensure that the kubelet.conf file permissions are set to 644 or more restrictive Scored L1 Equivalent Control Control Plane pod specifications are protected by a similarly restrictive RBAC authorization policy in the Admin Cluster and aren't present as files on disk.
    4.1.6 Ensure that the kubelet.conf file ownership is set to root:root Scored L1 Equivalent Control Control Plane pod specifications are protected by a similarly restrictive RBAC authorization policy in the Admin Cluster and aren't present as files on disk.
    4.2.4 Ensure that the --read-only-port argument is set to 0 Scored L1 Fail 10255 Some GKE on-prem monitoring components use the kubelet read-only port to obtain metrics.
    4.2.6 Ensure that the --protect-kernel-defaults argument is set to true Scored L1 Fail false GKE on-prem allows kubelet to set its necessary kernel settings.
    4.2.9 Ensure that the --event-qps argument is set to 0 or a level which ensures appropriate event capture Not Scored L2 Warn not set Events are Kubernetes objects stored in etcd. To avoid overwhelming etcd they are only kept for one hour, and are not an appropriate security auditing mechanism. Allowing unlimited events as suggested in this control exposes the cluster to unnecessary DoS risk and contradicts the recommendation to use admission EventRateLimits. Security relevant events that need permanent storage should be sent to logs.
    4.2.10 Ensure that the --tls-cert-file and --tls-private-key-file arguments are set as appropriate Scored L1 Equivalent Control not set GKE on-prem manages kubelet server TLS using the --rotate-server-certificates flag.
    4.2.12 Ensure that the RotateKubeletServerCertificate argument is set to true Scored L1 Equivalent Control not set GKE on-prem enables Kubelet certificate rotation by setting --rotate-server-certificates=true on kubelets.
    4.2.13 Ensure that the Kubelet only makes use of Strong Cryptographic Ciphers Not Scored L1 Warn Kubelet uses the default set of Cryptographic Ciphers

    How to audit Benchmarks

    Specific instructions for auditing each Recommendation is available as part of the relevant CIS Benchmark. However, you may wish to automate some of these checks to simplify the verification of these controls in your environment. The tools listed below can help with this.

    Automated auditing of the CIS Kubernetes Benchmark

    You can use an open-source tool kube-bench to test your cluster configuration against the CIS Kubernetes Benchmark.

    Make sure to specify the appropriate version, for example,

    kube-bench node --benchmark cis-1.5