With the speed of development in Kubernetes, there are often new security features for you to use. This document describes how to harden your GKE on-prem clusters.
This document prioritizes high-value security mitigations that require your action at cluster creation time. Less critical features, secure-by-default settings, and those that can be enabled after cluster creation are mentioned later in the document. For a general overview of security topics, review Security.
Checklist
The following deployment checklist highlights best practices for hardening your GKE clusters platform deployment. For more information about each practice, see the sections in this document.
| Deployment checklist | Description |
|---|---|
| Identity and access control | Use vSphere account privileges: Secure Google Cloud service accounts: Configure OpenID Connect (OIDC): Use Kubernetes namespaces and RBAC to restrict access: |
| Data protection | Encrypt vSphere virtual machines: Manage secrets: |
| Network protection | Restrict network access to the control plane and nodes: Use network policies to restrict traffic: |
| Declarative security | Use Config Management policy controller: |
| Maintenance | Upgrade GKE Enterprise: Monitor security bulletins: |
| Monitoring and logging | Set options for GKE clusters logging: |
Identity and access control
Use vSphere account privileges
The vCenter user account that you use to install GKE on-prem must have sufficient privileges. For example, a user account that is assigned the vCenter's Administrator role has privileges for complete access to all vCenter objects and provides an GKE on-prem cluster administrator with full access.
The principle of least privilege is recommended, granting only the necessary privileges to successfully install GKE Enterprise. We have predefined the minimum set of privileges needed to perform the installation, as well as the commands needed to grant these permissions.
Secure Google Cloud service accounts
GKE on-prem requires four Google Cloud service accounts:
- A predefined service account for accessing the GKE on-prem software. You create this when you purchase GKE Enterprise.
- A register service account to be used by Connect for registering GKE on-prem clusters with Google Cloud.
- A connect service account to be used by Connect for establishing a connection between GKE on-prem clusters and Google Cloud.
- A Cloud Logging service account for collecting cluster logs for use by Cloud Logging.
During installation, you bind Identity and Access Management roles to these service accounts. Those roles grant the service accounts specific privileges within your project and can be generated for you during installation.
Configure OpenID Connect
You can only configure authentication at cluster creation time. To configure user authentication for your clusters, use OpenID Connect (OIDC).
GKE on-prem supports OIDC as one of the authentication mechanisms for interacting with a user cluster's Kubernetes API server. With OIDC, you can manage access to Kubernetes clusters by using the standard procedures in your organization for creating, enabling, and disabling user accounts.
For specific OIDC support, see the following documentation:
Use Kubernetes namespaces and RBAC to restrict access
To give teams least-privilege access to Kubernetes, create Kubernetes Namespaces or environment-specific clusters. Assign cost centers and appropriate labels to each namespace for accountability and chargeback. Only give developers the level of access to their Namespaces that they need to deploy and manage their applications, especially in production.
Map the tasks that users need to complete against the cluster and define the permissions required to complete each task. To grant cluster- and namespace-level permissions, use Kubernetes RBAC.
Beyond the permissions for Google Cloud service accounts used to install GKE on-prem, IAM does not apply to GKE on-prem clusters.
For more information, see the following documentation:
Data protection
Encrypt vSphere virtual machines
GKE on-prem cluster nodes run on virtual machines (VMs) in your vSphere cluster. Google strongly recommends that you encrypt all data at rest. To do so on vSphere, follow the VMware vSphere security PDF and best practice guidance for encrypting VMs.
This must be done prior to the installation of GKE Enterprise.
Manage secrets
To provide an extra layer of protection for sensitive data, such as Kubernetes Secrets stored in etcd, configure a secrets manager that is integrated with GKE on-prem clusters.
If you are running workloads across multiple environments, you might prefer a solution that works for both Google Kubernetes Engine and GKE on-prem. If you choose to use an external secrets manager, such as HashiCorp Vault, set it up prior to integrating your GKE on-prem clusters.
You have several options for secrets management:
- You can use Kubernetes Secrets natively in GKE on-prem. We expect clusters to be using vSphere encryption for VMs as described earlier, which provides basic encryption-at-rest protection for secrets. Secrets are not further encrypted by default.
- You can use an external secrets manager, such as HashiCorp Vault. You can authenticate to HashiCorp using either a Kubernetes service account or a Google Cloud service account.
For more information, see the following documentation:
Network protection
Restrict network access to the control plane and nodes
Limit exposure of your cluster control plane and nodes to the internet. These choices cannot be changed after cluster creation. By default, GKE on-prem cluster nodes are created using RFC 1918 addresses, and it is best practice to not change this. Implement firewall rules in your on-premises network to restrict access to the control plane.
Use network policies to restrict traffic
By default, all Services in a GKE on-prem cluster can communicate with each other. For information about controlling Service-to-Service communication as needed for your workloads, see the following sections.
Restricting network access to services makes it much more difficult for attackers to move laterally within your cluster and offers services some protection against accidental or deliberate denial of service. There are two recommended ways to control traffic:
- To control L7 traffic to your applications' endpoints, use Istio. Choose this if you're interested in load balancing, service authorization, throttling, quota, and metrics.
- To control L4 traffic between Pods, use Kubernetes network policies. Choose this if you're looking for the basic access control functionality managed by Kubernetes.
You can enable both Istio and Kubernetes network policy after you create your GKE on-prem clusters. You can use them together if you need to.
For more information, see the following documentation:
Declarative security
Use Config Management policy controller
Kubernetes admission controllers are plugins that govern and enforce how a Kubernetes cluster is used. Admission controllers are an important part of the defence-in-depth approach to hardening your cluster.
The best practice is to use Config Management's Policy Controller. Policy Controller uses the OPA Constraint Framework to describe and enforce policy as CRDs. The constraints that you apply to your cluster are defined in constraint templates, which are deployed in your clusters.
For information about how to use Policy Controller constraints to achieve many of the same protections as PodSecurityPolicies, with the added ability to test your policies before enforcing them, see Using constraints to enforce Pod security.
For more information, see the following documentation:
Maintenance
Upgrade GKE Enterprise
Kubernetes regularly introduces new security features and provides security patches.
You are responsible for keeping your GKE on-prem clusters up to date. For each release, review the release notes. Additionally, plan to update to new patch releases every month and minor versions every three months. Learn how to upgrade your clusters.
You are also responsible for upgrading and securing the vSphere infrastructure:
- Establish a process to undertake timely patching and upgrades of your VMs
- Keep up to date with the latest VMware security advisories
- Follow the guidance on applying patches to hosts
Monitor security bulletins
The GKE Enterprise security team publishes security bulletins for high and critical severity vulnerabilities.
These bulletins follow a common Google Cloud vulnerability numbering scheme and are linked from the main Google Cloud bulletins page and the GKE on-prem release notes. Each security bulletin page has an RSS feed where users can subscribe to updates.
When customer action is required to address these high and critical vulnerabilities, Google contacts customers by email. In addition, Google might also contact customers with support contracts through support channels.
For more information, see the following documentation:
Monitoring and logging
Set options for GKE clusters logging
GKE on-prem includes multiple options for cluster logging and monitoring, including cloud-based managed services, open source tools, and validated compatibility with third-party commercial solutions:
- Cloud Logging and Cloud Monitoring, enabled by in-cluster agents deployed with GKE on-prem
- Prometheus and Grafana, disabled by default
- Validated configurations with third-party solutions
Whichever logging solution you choose based on business requirements, we strongly advise logging forward-relevant events and alerts to a centralized security information and event management (SIEM) service for management of security incidents.
For more information, see the following documentation:
- Logging and monitoring
- Using logging and monitoring
- Application logging and monitoring
- Monitoring Anthos clusters on VMware with the Elastic Stack
- Collect logs on GKE Enterprise with Splunk Connect