This page provides an overview for the role-based access control (RBAC) system provided by Kubernetes, and how you can use Kubernetes RBAC in Google Kubernetes Engine (GKE).
Overview
Kubernetes includes a built-in role-based access control (RBAC) mechanism that enables you to configure fine-grained and specific sets of permissions that define how a given Google Cloud user, or group of users, can interact with any Kubernetes object in your cluster, or in a specific Namespace of your cluster.
Kubernetes RBAC is enabled by default.
Before you begin
Before you start, make sure you have performed the following tasks:
- Ensure that you have enabled the Google Kubernetes Engine API. Enable Google Kubernetes Engine API
- Ensure that you have installed the Cloud SDK.
Set up default gcloud
settings using one of the following methods:
- Using
gcloud init
, if you want to be walked through setting defaults. - Using
gcloud config
, to individually set your project ID, zone, and region.
Using gcloud init
If you receive the error One of [--zone, --region] must be supplied: Please specify
location
, complete this section.
-
Run
gcloud init
and follow the directions:gcloud init
If you are using SSH on a remote server, use the
--console-only
flag to prevent the command from launching a browser:gcloud init --console-only
-
Follow the instructions to authorize
gcloud
to use your Google Cloud account. - Create a new configuration or select an existing one.
- Choose a Google Cloud project.
- Choose a default Compute Engine zone.
Using gcloud config
- Set your default project ID:
gcloud config set project PROJECT_ID
- If you are working with zonal clusters, set your default compute zone:
gcloud config set compute/zone COMPUTE_ZONE
- If you are working with regional clusters, set your default compute region:
gcloud config set compute/region COMPUTE_REGION
- Update
gcloud
to the latest version:gcloud components update
Interaction with Identity and Access Management
You can use both Identity and Access Management (IAM) and Kubernetes RBAC to control access to your GKE cluster:
IAM is not specific to Kubernetes; it provides identity management for multiple Google Cloud products, and operates primarily at the level of the Google Cloud project.
Kubernetes RBAC is a core component of Kubernetes and lets you create and grant roles (sets of permissions) for any object or type of object within the cluster.
In GKE, IAM and Kubernetes RBAC are integrated to authorize users to perform actions if they have sufficient permissions according to either tool. This is an important part of bootstrapping a GKE cluster, since by default Google Cloud users do not have any Kubernetes RBAC RoleBindings.
To authorize users using Google Cloud accounts, the client must
be correctly configured to authenticate using those accounts first. For example,
if you are using kubectl
, you must
configure the kubectl
command to authenticate to Google Cloud
before running any commands that require authorization.
While Kubernetes RBAC can be used instead of IAM for almost all
cases, GKE users are required at least the
container.clusters.get
IAM permission in the project containing
the cluster. This permission is included by the container.clusterViewer
role,
as well as the other, more highly privileged roles. The container.clusters.get
permission is required for users to authenticate to the clusters in the
project, but does not authorize them to perform any actions inside those
clusters. Authorization may then be provided by either IAM or
Kubernetes RBAC.
Prerequisites for using Kubernetes RBAC on GKE cluster v1.11.x and older
In GKE clusters using GKE v1.11.x and older, there is a limitation that IAM cannot grant the ability to create a Kubernetes RBAC Role or ClusterRole. However, the Kubernetes Engine Admin IAM role does grant users with the ability to create a Kubernetes RBAC RoleBinding or ClusterRoleBinding for any user, including themselves, which can be used to bind Google Cloud users to predefined RBAC Roles.
In particular, the cluster-admin
predefined RBAC role grants users full
permissions in the cluster. Therefore, to bootstrap a user to allow them to
create RBAC Roles and ClusterRoles, issue the following command, replacing
user-account with the target user's Google Cloud login email
address.
kubectl create clusterrolebinding cluster-admin-binding \ --clusterrole cluster-admin \ --user user-account
Google Groups for GKE
Previously, you could only grant roles to Google Cloud user accounts or IAM service accounts. Google Groups for GKE (Beta) enables you to grant roles to the members of a group in Google Groups for Business. With this mechanism, the users and groups themselves are maintained by your Google Workspace administrators, completely outside of Kubernetes or Cloud Console, so your cluster administrators do not need detailed information about your users. Another benefit is integration with your existing user account management practices, such as revoking access when someone leaves your organization.
To use this feature, complete the following tasks:
- Meet the requirements.
- Configure your Google Groups.
- Create a cluster with the feature enabled.
- Associate the Google Groups with sets of cluster permissions.
Requirements
Using Google Groups for GKE has the following requirements:
- You must have a Google Workspace or Cloud Identity subscription.
Configure Google Groups for use with RBAC
Configuring your cluster to use this feature, as well as the syntax for referencing a Google Group in Kubernetes RBAC is discussed later in this topic. First, you need to set up your Google Groups following the steps below:
Create a Google Group in your domain, named
gke-security-groups@yourdomain.com
. The group must be named exactlygke-security-groups
. Make sure thegke-security-groups
group has the "View Members" permission for "Group Members". See this article for an example of how to set this in the Google Workspace Admin Console.You may also refer to the Groups Help Center for more information on managing Groups in Google Workspace.
Create groups, if they do not already exist, that represent groups of users or groups who should have different permissions on your clusters. Each group must have the "View Members" permission for "Group Members".
Add these groups (not users) to the membership of
gke-security-groups@yourdomain.com
.
To check whether a given user has permission to create, modify, or view a
resource in the cluster based on group membership, GKE checks
both whether the user is a member of a group with access, and whether that group
is a direct member of your domain's gke-security-groups
group.
Information about Google Group membership is cached for a short time. It may take a few minutes for changes in group memberships to propagate to all of your clusters. In addition to latency from groups changes, standard caching of user credentials on the cluster is about one hour.
Configuring your cluster to use Google Groups for GKE
After your Google Groups administrator sets up your groups,
create a new cluster using the gcloud
command, and add the
--security-group="gke-security-groups@yourdomain.com"
flag, substituting the value with your own domain name.
The following is an example of the cluster create command:
gcloud beta container clusters create cluster-name \
--security-group="gke-security-groups@yourdomain.com"
Now you are ready to create Roles, ClusterRoles, RoleBindings, and ClusterRoleBindings that reference your Google Groups.
Defining and assigning permissions
You define your RBAC permissions by creating the following kinds of Kubernetes objects:
- ClusterRole or Role: defines a set of resource types and operations that can be assigned to a user or group of users in a cluster (ClusterRole), or a Namespace (Role), but does not specify the user or group of users.
- ClusterRoleBinding or RoleBinding: assigns a ClusterRole or Role to a user or group of users. A ClusterRoleBinding works with a ClusterRole, and a RoleBinding works with either a ClusterRole or a Role.
RBAC roles are purely additive—there are no "deny" rules. When structuring your RBAC roles, you should think in terms of "granting" users access to cluster resources.
Defining Permissions using Roles or ClusterRoles
You define permissions within a Role or ClusterRole object. A Role defines access to resources within a single Namespace, while a ClusterRole defines access to resources in the entire cluster.
Roles and ClusterRoles have the same syntax. Each has a rules
section, where
you define the relevant Namespace, resource type, and allowable operations for
the Role. For example, the following Role grants read access (get
, watch
,
and list
) for all Pods in the accounting
Namespace:
kind: Role
apiVersion: rbac.authorization.k8s.io/v1
metadata:
namespace: accounting
name: pod-reader
rules:
- apiGroups: [""] # "" indicates the core API group
resources: ["pods"]
verbs: ["get", "watch", "list"]
Role
vs. ClusterRole
Because permissions granted by a ClusterRole apply across the entire cluster, you can use ClusterRoles to control access to different kinds of resources than you can with Roles. These include:
- Cluster-scoped resources such as nodes
- Non-resource REST Endpoints such as
/healthz
- Namespaced resources across all Namespaces (for example, all Pods across the entire cluster, regardless of Namespace)
Assigning Roles using RoleBindings or ClusterRoleBindings
After creating a Role or ClusterRole, you assign it to a user or group of users
by creating a RoleBinding or ClusterRoleBinding. Users and groups are called
subjects
, and can be any of the following:
Subject type | Value for kind |
Value for name |
---|---|---|
Google Cloud user account | User |
Google Cloud registered email address |
Kubernetes service account | ServiceAccount |
The name of a Kubernetes ServiceAccount object in the cluster |
IAM service account | User |
Automatically generated IAM service account email address |
Google Group address (Beta) on a verified domain | Group |
Email address of a Google Group that is itself a member of the Google Group gke-security-groups@yourdomain.com |
The following RoleBinding grants the pod-reader
Role to a user, a
Kubernetes service account, an IAM service account, and a Google
Group:
kind: RoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
name: pod-reader-binding
namespace: accounting
subjects:
# Google Cloud user account
- kind: User
name: janedoe@example.com
# Kubernetes service account
- kind: ServiceAccount
name: johndoe
# IAM service account
- kind: User
name: test-account@test-project-123456.google.com.iam.gserviceaccount.com
# Google Group
- kind: Group
name: accounting-group@example.com
roleRef:
kind: Role
name: pod-reader
apiGroup: rbac.authorization.k8s.io
API Usage and Examples
For complete information on using the Kubernetes API to create the necessary
Role
, ClusterRole
, RoleBinding
, and ClusterRoleBinding
objects for
RBAC, see Using Role-Based Access Control Authorization
in the Kubernetes documentation.
Troubleshooting and debugging
To debug issues with RBAC, use the
Admin activity audit log, which
is enabled on all clusters by default. If access to a resource or operation is
denied due to lack of sufficient permissions, the API server logs an RBAC DENY
error, along with additional information such as the user's implicit and
explicit group membership. If you are using Google Groups for
GKE, google groups
appears in the log message.
Debugging issues with Google Groups integration
The following instructions enable you to view logs to validate if your clusters have been successfully configured to use Google Groups in RBAC rolebindings.
Prerequisites
Before you begin examining the logs, make sure:
- You have not interacted with the cluster you want to test (for example, ran any
kubectl
commands) for at least one hour. Authentication is cached for one hour and you need to make sure the request gets logged when it happens. - You are a member of at least one of the groups included in
gke-security-groups
. This ensures some Google Groups information is populated in the logs.
Configuring logs
To use logs for debugging Google Groups with RBAC:
Enable data access logging for your Google Cloud project. To enable the logging:
In the Cloud Console, go to the Audit Logs page in the IAM menu.
In the table, select Kubernetes Engine API.
In the Log Type menu, select:
- Admin Read
- Data Read
- Data Write
Click Save.
For more information about enabling Audit Logging, see Configuring Data Access logs with the Cloud Console in the Cloud management tools documentation.
Run a command using
kubectl
in the cluster. This can be as simple askubectl create ns helloworld
.Enter a custom query in the Logs Viewer page. To run the query:
In the Cloud Console, go to the Logs Viewer page in the Logging menu.
Click the arrow in the Query preview box at the top of the page.
In the dropdown box that appears, copy and paste the following query:
resource.type="k8s_cluster" resource.labels.location="cluster-region" resource.labels.cluster_name="cluster-name" protoPayload.resourceName="authorization.k8s.io/v1beta1/subjectaccessreviews" protoPayload.response.spec.user="email-address"
Where:
- cluster-region is your cluster's region or zone.
- cluster-name is the name of your cluster.
- email-address is the registered email address of your Google account.
Select Run Query. You should see at least one result. If you do not, try increasing the time range.
Select the cluster you want to examine.
Click Expand nested fields.
The field
protoPayload.request.spec.group
contains the groups where:- The groups are members of
gke-security-group
. - You are a member of the group.
This list should match the set of groups you are a member of. If no groups are present, there might be an issue with how the groups are set up.
- The groups are members of
Restore data access logging to previous settings to avoid further charges (if desired).
Limitations
The following sections describe interactions that might not seem obvious when working with Kubernetes RBAC.
Default discovery roles
Clusters are created with a set of
default ClusterRoles and ClusterRoleBindings.
Requests made with valid credentials are placed in the system:authenticated
group, whereas all other requests fall into system:unauthenticated
.
Prior to Kubernetes version 1.14, both system:authenticated
and
system:unauthenticated
grant the system:basic-user
and system:discovery
ClusterRoles by default.
The system:basic-user
ClusterRole lets users to make
SelfSubjectAccessReviews
to test their permissions in the cluster. The
system:discovery
role lets users to read discovery APIs, which can reveal
information about
CustomResourceDefinitions
added to the cluster.
As of Kubernetes 1.14, anonymous users (system:unauthenticated
) will receive
the system:public-info-viewer
ClusterRole instead, which grants read-only
access to /healthz
and /version
APIs.
To see the API endpoints allowed by the system:discovery
ClusterRole, run the
following command:
kubectl get clusterroles system:discovery -o yaml
Forbidden error for service accounts on Google Cloud VM instances
The following error can occur when the VM instance does not have the
userinfo-email
scope:
Error from server (Forbidden): error when creating ... "role-name" is forbidden: attempt to grant extra privileges:...
For example, suppose the VM has cloud-platform
scope but does
not have userinfo-email
scope. When the VM gets an access token, Google Cloud
associates that token with the cloud-platform
scope. When the Kubernetes API
server asks Google Cloud for the identity associated with the access token,
it receives the service account's unique ID, not the service account's email.
To authenticate successfully, either create a new VM with the userinfo-email
scope or create a new role binding that uses the unique ID.
To create a new VM instance with the userinfo-email
scope, run the following command:
gcloud compute instances create instance-name \ --service-account service-account-email \ --scopes userinfo-email
To create a new role binding that uses the service account's unique ID for an existing VM, perform the following steps:
Identify the service account's unique ID:
gcloud iam service-accounts describe service-account-email
For example, the following output displays the
uniqueId
for themy-iam-account@somedomain.com
service account:displayName: Some Domain IAM service account email: my-iam-account@somedomain.com etag: BwWWja0YfJA name: projects/project-name/serviceAccounts/my-iam-account@somedomain.com oauth2ClientId: '123456789012345678901' projectId: project-name uniqueId: '123456789012345678901'
Create a role binding using the
uniqueId
of the service account:kubectl create clusterrolebinding clusterrolebinding-name \ --clusterrole cluster-admin \ --user unique-id
What's next
- Learn how to create IAM policies.