Before you begin
You must have already installed Batch on GKE.
Get the admin tools:
git clone https://github.com/GoogleCloudPlatform/Kbatch.git
Go to the admin tools directory:
To use Batch on GKE, users must be able to authenticate to the Kubernetes
container.clusters.get. This permission allows users to run:
gcloud container clusters get-credentials <var>cluster-name</var>
This permission needs to be granted to every user in each GCP project. Once you grant permissions, it works for every cluster in that project.
There are two options to grant this permission:
Create a custom role and policy binding.
To create a custom role with the
container.clusters.getpermission, run the following command:
gcloud iam roles create BatchUser --project [PROJECT_ID] \ --title GKEClusterReader --permissions container.clusters.get --stage BETA 2>&1
Then grant the role to a user:
gcloud projects add-iam-policy-binding [PROJECT_NAME] \ --member=user:[USER_EMAIL] \ --role=projects/[PROJECT_NAME]/roles/BatchUser
Grant the predefined containers.clusterViewer role to the user.
gcloud projects add-iam-policy-binding [PROJECT_NAME] --member=user:[USER_EMAIL] --role=roles/container.clusterViewer
To allow users to submit batch jobs, you must grant them certain Kubernetes RBAC (role-based access control). To add a new user, go to the users directory:
Adding a user without an auto PVC
Batch provides a script to add all the necessary RBACs in one step:
./add-user.sh -s [SHORT_NAME] -u [USER'S_NAME] \ -n [NAMESPACE] --project [PROJECT_ID] --security-policy [UID],[GROUP_ID]
This command generates the following RBACs for the user:
- A BatchUserContext (named [SHORT_NAME]-[NAMESPACE]) with the
UIDid and group ID configured as
- A k8s ClusterRole and ClusterRoleBinding (named [SHORT_NAME]-cr) to grant the user the read permission of BatchCostModel and BatchPriority.
- A k8s Role and RoleBinding (named [SHORT_NAME]-[NAMESPACE] to grant create and read permission to BatchJobs in the namespace.
With all these permissions, a user can submit BatchJobs in the specified Namespace.
Adding a user with a private PVC
If you set up FileStore and a NFS auto provisioner in your cluster, you can run the following command to add a user and create a private PVC:
./add-user.sh -s [SHORT_NAME] -u [USER_NAME] -n [NAMESPACE] \ --project [PROJECT_ID] --auto-pvc --security-policy "2000,3000"
Where 2000 is the UID and 3000 is the GID.
In addition to the RBACs,
--auto-pvc flag will cause an NFS-based PVC to be
created. The NFS auto provisioner creates a folder under the NFS server.
Additional resources are created inside the namespace for user include:
A PersistentVolumeClaim named [SHORT_NAME]-[NAMESPACE]. A PersistentVolume which is bound to the PVC above. Grant the user the RBACs to use that PVC.
Adding a user with shared PVC
To add a user and grant access to specific PVCs, run the following command:
./add-user.sh -s [SHORT_NAME] -u [USER_NAME] -n [NAMESPACE] \ --project [PROJECT_ID] --pvcs [PVC_NAME],[PVC_NAME] --security-policy "2000,3000"
Deleting a user
To remove a user's ability to submit and view BatchJobs, run the following command:
./delete-user.sh -s [SHORT_NAME] -n [NAMESPACE]
The [SHORT_NAME] is the same as the short name used when the user is added.
Customizing user settings
BatchUserContext (BUC) is a Batch API resource that configures a job's security context. BatchUserContext associates a user with a PodSecurityPolicySpec. It defines the security context for the batch jobs submitted by that user.
Some important things that can be configured by BatchUserContext are:
- Unix Identity (UID) that the job will run as
- Whether the job can run in privilege mode as root.
- The type of volumes that can be used by this user.
When a job is submitted Batch performs the following two operations based on BUC:
Validates that the BatchJob spec does not violate the policy. For example, if the policy prevents a BatchJob from running in privileged mode, but the BatchJob has the "privileged" field set to true, the job submission will fail.
Copies values from the BUC to empty fields in the job spec. For example, if the UID in the BUC is set to
MustRunAs: 3000and the job doesn't have a
RunAsvalue set, Batch will accept the BatchJob and set the
RunAsfield to be 3000.
Refer to PodSecurityPolicySpec for descriptions of all the fields.
For example, the default BatchUserContext is:
apiVersion: kbatch.k8s.io/v1beta1 kind: BatchUserContext metadata: labels: controller-tools.k8s.io: "1.0" name: alice-default namespace: default spec: userName: "*" securityPolicySpec: privileged: false allowPrivilegeEscalation: false requiredDropCapabilities: - ALL volumes: - 'configMap' - 'emptyDir' - 'secret' - 'downwardAPI' - 'persistentVolumeClaim' hostNetwork: false hostIPC: false hostPID: false runAsUser: rule: 'MustRunAs' ranges: - min: 12345 max: 12345 seLinux: rule: 'RunAsAny' runAsGroup: rule: 'MustRunAs' ranges: - min: 12345 max: 12345 fsGroup: rule: 'MustRunAs' ranges: - min: 12345 max: 12345 supplementalGroups: rule: 'MustRunAs' ranges: - min: 12345 max: 12345
Creating a BatchUserContext
To create a BatchUserContext run the following command:
kubectl create -f [NAME]-default-buc.yaml
Updating a BatchUserContext
To change a BatchUserContext run the following command once you have saved your changes to the file:
kubectl apply -f [NAME]-default-buc.yaml
Deleting a BatchUserContext
To delete a BatchUserContext run the following command:
kubectl delete -f [NAME]-default-buc.yaml