This page describes the security features, configurations, and settings in Google Kubernetes Engine (GKE) Autopilot, which is the recommended way to run GKE.
Who should use this page?
This page is intended for security administrators who want to understand the security restrictions that Google specifically applies to Autopilot clusters, and the security features that are available for use in Autopilot.
You should also read the GKE security overview, which describes the hardening options, measures, and recommendations that apply to all GKE clusters, network configurations, and workloads.
Security measures in Autopilot
Autopilot clusters enable and apply security best practices and settings by default, including many of the recommendations in the security overview and in Harden your cluster security.
If you want recommended resources based on your use case, skip to Security resources by use case. The following sections describe the security policies that Autopilot applies for you.
Autopilot and the Kubernetes Pod Security Standards
The Kubernetes project has a set of security guidelines named the Pod Security Standards that define the following policies:
- Privileged: No access restrictions. Not used in Autopilot.
- Baseline: Prevents known privilege escalation pathways. Allows most workloads to run without significant changes.
- Restricted: Highest level of security. Requires significant changes to most workloads.
In Autopilot mode, GKE enforces Pod security constraints using a custom admission controller named GKE Warden. We enforce a default security policy that includes all of the recommendations in the Baseline level of the Pod Security Standards, with some modifications for usability. Additionally, the default GKE Warden admission policy for Autopilot enforces many constraints from the Restricted level of the Pod Security Standards, but avoids restrictions that would block a majority of your workloads from running.
If you need to apply additional restrictions to comply with the full Restricted policy, you can optionally use the PodSecurity admission controller in specific namespaces.
The following table describes how the controls in the default Autopilot GKE Warden policy compare to the controls in the Baseline and the Restricted levels of the Pod Security Standards. For descriptions of each control in this table, see the corresponding entry in Pod Security Standards.
When evaluating compliance, we considered how the constraints apply to your own workloads. This excludes verified Google Cloud partner workloads and system workloads that require specific privileges to function.
| Control | Baseline policy compliance | Restricted policy compliance | Additional information |
|---|---|---|---|
| HostProcess | Autopilot blocks HostProcess. | ||
| Host namespaces | Autopilot blocks host namespaces. Some containers from verified partners are allowed to use host namespaces. | ||
| Privileged containers | Autopilot blocks privileged containers by default. Autopilot allows privileged containers from verified partners for purposes such as running security and monitoring tooling. | ||
| Linux capabilities | Autopilot workloads can only access the capabilities specified in the Baseline Pod Security Standard by default. You can manually enable the following capabilities:
Autopilot also allows some verified partner workloads to set dropped capabilities. |
||
| HostPath volumes | Partially complies | Partially complies | Autopilot allows containers to request read-only access to
/var/log for debugging, but denies all other read or write
access. |
| HostPorts | Setting specific host ports is disallowed, which mitigates some scheduling issues, and prevents accidental or deliberate direct network exposure of services. You can manually set up random host port assignment from a known range to support direct-connection networking applications such as game servers. | ||
| AppArmor | The AppArmor docker-default security profile is automatically applied to Container-Optimized OS. | ||
| SELinux | SELinux isn't applied because AppArmor is already applied. SELinux is also not mandatory in the Pod Security Standards. | ||
/proc mount type |
GKE does not set the ProcMountType feature flag.
If the Pod securityContext sets procMount
to "Unmasked", GKE automatically overrides it with
"Default". |
||
| seccomp profile | Autopilot applies the RuntimeDefault seccomp
profile to all workloads. You can manually override this setting for
specific workloads by setting the profile to Unconfined in
the Pod specification. |
||
| sysctls | GKE does not set the --allowed-unsafe-sysctls kubelet flag so pods with unsafe sysctls fail to schedule. For additional protection, as of 11 July 2023, new 1.27+ clusters also have a policy rule to enforce the securityContext settings and reject Pods that use unsafe sysctls. | ||
| Volume types | Autopilot allows only the volume types in the Restricted policy with the following additions: HostPath volumes with read-only access to
/var/log for debugging, gcePersistentDisk for Compute Engine
persistent disks, and nfs for network file system volumes. |
||
| Privilege escalation | This setting only provides protection to containers that aren't running as root. Industry surveys show that 76% of containers run as root, so Autopilot allows running as root to enable most workloads. This setting is also currently useful in de-privileging workloads to non-root by allowing use of file system capabilities to work around deficiencies with Kubernetes root capability handling. | ||
| Run as non-root | Industry surveys show that 76% of containers run as root, so Autopilot allows running as root to enable most workloads. | ||
| Run as non-root user | Containers can set runAsUser to 0.
Industry surveys
show that 76% of containers run as root, so
Autopilot allows running as root to enable most workloads |
Built-in security configurations
Google applies many built-in security settings to Autopilot clusters based on industry best practices and our expertise. The following table describes some of the security configurations that Autopilot applies for you:
| Configuration | Description |
|---|---|
| Host options |
|
| Linux capabilities | You can use the following Linux capabilities: "SETPCAP", "MKNOD", "AUDIT_WRITE", "CHOWN", "DAC_OVERRIDE", "FOWNER", "FSETID", "KILL", "SETGID", "SETUID", "NET_BIND_SERVICE", "SYS_CHROOT", "SETFCAP", "SYS_PTRACE" You can also manually enable the following capabilities:
In GKE versions earlier than 1.21, the |
| Privileged containers | Containers can't run in Privileged mode unless the container is deployed by a Google Cloud partner. Privileged containers can make changes to the underlying node, such as changing the kubelet. This access could increase the impact of a Pod compromise. |
| GKE-managed namespaces | As a security measure, Autopilot doesn't allow deploying
workloads in GKE-managed namespaces, such as
kube-system. |
| Container isolation | Autopilot enforces the following restrictions on containers to limit the impact of container escape vulnerabilities. Linux capabilities and kernel security
|
| Pod-level security policy enforcement | Autopilot supports enforcement mechanisms for Pod-level
security policies such as the
PodSecurity admission controller,
Gatekeeper,
or Policy Controller.
However, you might not need to use any of these if the built-in security
configurations described on this page already meet your requirements. |
| SSH to nodes | Autopilot blocks SSH access to nodes. GKE handles all operational aspects of the nodes, including node health and all Kubernetes components running on the nodes. You can still connect remotely to your running containers using the Kubernetes
|
| User impersonation | GKE version 1.22.4-gke.1501 and later support
user impersonation for all user-defined users and groups. System users and
groups such as the kube-apiserver user and the
system:masters group cannot be impersonated. |
| Mutating dynamic admission webhooks | Autopilot modifies mutating webhooks to exclude resources in
managed namespaces, such as Autopilot also rejects webhooks that specify one or more of the following resources, and any sub-resources of those resources. - group: "" resource: nodes - group: "" resource: persistentVolumes - group: certificates.k8s.io resource: certificatesigningrequests - group: authentication.k8s.io resource: tokenreviews You can't use the |
| ValidatingAdmissionPolicies | Autopilot modifies Autopilot also rejects - group: "" resource: nodes - group: "" resource: persistentVolumes - group: certificates.k8s.io resource: certificatesigningrequests - group: authentication.k8s.io resource: tokenreviews You can't use the |
| Certificate signing requests | You can create CertificateSigningRequests in Autopilot to create certificates that are signed by the cluster certificate authority. To prevent interference with system components, Autopilot rejects CertificateSigningRequests for known privileged identities, such as system groups, system agents, or Google-managed IAM service agents. |
| GKE security features | Autopilot clusters enable recommended GKE security features for you. For a list of enabled and optional security features, refer to security features in Autopilot. |
| Node operating system | Autopilot clusters use Container-Optimized OS with
containerd as the node operating system. Container-Optimized OS
is created and managed by Google. |
| GKE version upgrades | Autopilot clusters are enrolled in a GKE release channel upon creation, and automatic upgrades are always enabled. Google automatically upgrades your control plane and nodes to the latest qualified version in the release channel over time. |
Security boundaries in Autopilot
Autopilot provides access to the Kubernetes API but removes permissions to use some highly privileged Kubernetes features, such as privileged Pods. The goal is to limit the ability to access, modify, or directly control the node virtual machine (VM). Autopilot implements these restrictions to limit workloads from having low-level access to the node VM, so that Google Cloud can offer full management of nodes, and a Pod-level SLA.
Our intent is to prevent unintended access to the node VM. We accept submissions to that effect through the Google Vulnerability Reward Program (VRP) and will reward reports at the discretion of the Google VRP reward panel.
By design, privileged users such as cluster administrators have full control of any GKE cluster. As a security best practice, we recommend that you avoid granting powerful GKE or Kubernetes privileges widely and instead use namespace administrator delegation wherever possible as described in our multi-tenancy guidance.
Autopilot provisions single-tenant VMs in your project for your exclusive use. On each individual VM, your Autopilot workloads might run together, sharing a security-hardened kernel. Since the shared kernel represents a single security boundary, we recommend that if you require strong isolation, such as high-risk or untrusted workloads, run your workloads on GKE Sandbox Pods to provide multi-layer security protection.
Security resources based on use case
The following sections provide you with links and recommendations to plan, implement, and manage the security of your Autopilot clusters depending on your use case.
Plan cluster security
| Use case | Resources |
|---|---|
| Understand how GKE as a platform approaches security |
|
| Understand your role in hardening your environment | Learn about the shared responsibility model. |
| View Google's recommendations for hardening measures and incident response |
|
| Understand how GKE implements audit logging |
|
Authenticate and authorize
After setting up your Autopilot clusters, you might need to authenticate your users and applications to use resources such as the Kubernetes API or Google Cloud APIs.
| Use case | Resources |
|---|---|
| Authenticate users or applications to the cluster API server |
|
| Authenticate applications to Google Cloud APIs and services | Autopilot clusters let you use Workload Identity Federation for GKE to securely authenticate your workloads to Google Cloud APIs by configuring Kubernetes service accounts to act as IAM service accounts. For instructions, refer to Configure applications to use Workload Identity Federation for GKE. |
| Authorize actions at the project level | To authorize actions across clusters at the project level, use IAM. You can grant or deny access to specific GKE and Kubernetes API resources using IAM roles and permissions. For instructions, refer to Create IAM policies. |
| Authorize actions at the cluster level | To authorize actions on Kubernetes API resources in specific clusters, use the built-in Kubernetes role-based access control (RBAC) mechanism. For instructions, refer to Authorize actions in clusters using RBAC. |
| Authorize actions at the organization level | You can use Google Cloud Organization Policy Service to enforce constraints on specific operations on GKE resources across your Google Cloud organization. For instructions, refer to Restrict actions on GKE resources using custom organization policies. |
Harden clusters and workloads
If you have specialized isolation or hardening requirements beyond the pre-configured Autopilot measures, consider the following resources:
| Use case | Resources |
|---|---|
| Restrict public access to your cluster endpoint | Configure the network isolation of your Autopilot clusters and disable the external endpoint of the cluster control plane. For instructions, refer to Configure the control plane access. |
| Restrict cluster access to specific networks | Use control plane authorized networks to specify IP address ranges that can access your cluster. |
| Store sensitive information outside your cluster | Storing sensitive data in an external, encrypted storage provider with versioning enabled is a common compliance requirement and a best practice. Use Secret Manager to store your data and access it from your Autopilot clusters using Workload Identity Federation for GKE. For instructions, refer to Access secrets stored outside GKE clusters using Workload Identity Federation for GKE. |
| Verify container images before deployment to your cluster | Use Binary Authorization to check the integrity of the container images referenced in your Pod manifests at deploy time. For instructions, refer to Verify container images at deploy time using Binary Authorization. |
Monitor your security posture
After setting up your clusters and deploying your workloads, you should set up and configure monitoring and logging so that you have observability over your cluster security posture. We recommend that you do all of the following:
- Enroll your clusters in the GKE security posture dashboard to audit workloads for concerns such as problematic security configurations or vulnerabilities in your container operating system packages and get actionable mitigation information.
- Get notified about new security bulletins and upgrade events using cluster notifications.
- Observe your clusters using the GKE dashboard in Cloud Monitoring or the Observability tab in GKE.
- Learn how to view and manage your GKE audit logs in Cloud Logging.
What's next
- Read the GKE security overview.
- Read the GKE hardening guide.
- Subscribe to security bulletins and release notes.