Managing Batch on GKE resources

Beta

This product or feature is in a pre-release state and might change or have limited support. For more information, see the product launch stages.

Before you begin

You must have already installed Batch on GKE.

1. Get the admin tools:

   git clone https://github.com/GoogleCloudPlatform/Kbatch.git
   

2. Go to the admin tools directory:

   cd admintools
   

Creating and managing Priorities

Priorities let you prioritize jobs from high priority Queues above jobs from lower priority Queues.

To create a BatchPriority, perform the following steps:

1. Create criticalpriority.yaml from the manifest below:

   apiVersion: kbatch.k8s.io/v1beta1
   kind: BatchPriority
   metadata:
     labels:
       controller-tools.k8s.io: "1.0"
     name: critical
   spec:
     value: 150
     description: Critical Priority
   

2. Run the following command:

   kubectl apply -f criticalpriority.yaml
   

   The output is similar to

   batchpriority.kbatch.k8s.io/critical created
   

3. Verify the BatchPriority:

   kubectl get batchpriorities
   kubectl get batchpriorities --all-namespaces
   

   The output is similar to

   NAME      AGE
   critical  1m
   

   Note: Any previously created priorities will also appear in this output.

4. Describe the BatchPriority:

   kubectl describe batchpriority critical
   

   The output is similar to

   Name:         critical
   Namespace:
   Labels:       controller-tools.k8s.io=1.0
   Annotations:  <none>
   API Version:  kbatch.k8s.io/v1beta1
   
   Metadata:
     Creation Timestamp:  2019-09-11T22:24:15Z
     Finalizers:
       finalizers.kbatch.k8s.io/batchpriority
     Generation:        1
     Resource Version:  28112
     Self Link:         /apis/kbatch.k8s.io/v1beta1/batchpriorities/critical
     UID:               e377117e-d4e2-11e9-a0de-42010a80005f
   Spec:Kind:         BatchPriority
     Description:  Critical Priority
     Value:        150
   Status:
   Events:  <none>
   

Updating BatchPriorities

To update the BatchPriority edit criticalpriority.yaml then run the following command:

kubectl apply -f criticalpriority.yaml

Deleting BatchPriorities

To delete a BatchPriority, run the following command:

kubectl delete batchpriority critical

The output is similar to

batchpriority.kbatch.k8s.io "critical" deleted

You can verify that the BatchBudget is deleted by running:

kubectl get batchpriority

The output is similar to

No resources found.

Note: Any previously created priorities will also appear in this output.

Creating and managing BatchJobConstraints

A constraint specifies the jobs and policies acceptable in a given Queue.

To create a BatchJobConstraints, perform the following steps:

1. Create constraint.yaml from the manifest below:

   apiVersion: kbatch.k8s.io/v1beta1
   kind: BatchJobConstraint
   metadata:
     labels:
       controller-tools.k8s.io: "1.0"
     name: default
   spec:
     # The system supports the following constraints:
     # Cpu, Memory, WallTime, Gpu, GpuModel, RetryLimit
     # Adding a BatchJobConstraint to a BatchQueue means that the BatchQueue
     # will only accept jobs that satisfy the listed constraints.
     constraints:
       - name: WallTime
         operator: LessThan
         values: ["24h"]
   

2. Run the following command:

   kubectl create -f constraint.yaml
   

   The output is similar to

   batchjobconstraint.kbatch.k8s.io/batchjobconstraint-sample created
   

3. Verify the BatchJobConstraints:

   kubectl get batchjobconstraints
   kubectl get batchjobconstraints --all-namespaces
   

   The output is similar to

   NAME                        AGE
   batchjobconstraint-sample   45s
   

4. Describe the BatchJobConstraints:

   kubectl describe batchjobconstraint batchjobconstraint-sample
   

   The output is similar to

   Name:         batchjobconstraint-sample
   Namespace:
   Labels:       <none>
   Annotations:  <none>
   API Version:  kbatch.k8s.io/v1beta1
   Kind:         BatchJobConstraint
   Metadata:
     Creation Timestamp:  2019-09-11T22:39:39Z
     Generation:          1
     Resource Version:    32480
     Self Link:           /apis/kbatch.k8s.io/v1beta1/batchjobconstraints/batchjobconstraint-sample
     UID:                 0a496ff2-d4e5-11e9-a0de-42010a80005f
   Spec:
     Constraints:
       Name:      Memory
       Operator:  LessThan
       Values:
         16G
       Name:      Cpu
       Operator:  In
       Values:
         1
         2
         4
       Name:      Gpu
       Operator:  In
       Values:
         1
         2
         4
       Name:      GpuModel
       Operator:  Equal
       Values:
         nvidia-tesla-v100
       Name:      WallTime
       Operator:  LessThan
       Values:
         24h
       Name:      RetryLimit
       Operator:  LessThan
       Values:
         2
   Events:  <none>
   

Updating BatchJobConstraints

To update BatchJobConstraints, edit constraint.yaml then run the following command:

kubectl apply -f constraint.yaml

Deleting BatchJobConstraints

Note: If the BatchJobConstraint is being used by any BatchQueue, it cannot be deleted.

To delete a BatchJobConstraints, run the following command:

kubectl delete batchjobconstraint batchjobconstraint-sample

The output is similar to

batchjobconstraint.kbatch.k8s.io "batchjobconstraint-sample" deleted

You can verify that the BatchJobConstraints is deleted by running:

kubectl get batchjobconstraints

The output is similar to

No resources found.

Creating and managing CostModels

Batch uses CostModels to deduct resources from a budget.

To create a CostModel, perform the following steps:

1. Create default_costmodel.yaml from the manifest below:

   apiVersion: kbatch.k8s.io/v1beta1
   kind: BatchCostModel
   metadata:
     labels:
       controller-tools.k8s.io: "1.0"
     name: default
   spec:
     resources:
       # These costs are hourly and in USD. For example, based on the numbers below, running using 1 CPU
       # 1 GB of RAM, and 1 Tesla T4 GPU for 2 hours would cost 2 * ($0.031611 + $0.004237 + $0.95)
       # The values are taken from https://cloud.google.com/compute/pricing for
       # region us-west1. The values are accurate as of April 2019. Actual pricing depends on various factors,
       # such as long-term contracts, use of standard machine types, the region, etc.
       cpu: 0.031611         # This is the on-demand price for a single vCPU
       memory: 0.004237      # This is the on-demand price for 1GB of memory
       nvidia.com/gpu: 0.95  # This is the cost for use of the Tesla T4 GPU. Currently
                             # the system does not support breaking out prices for
                             # specific GPU types, so an average price for the intended
                             # GPUs should be used here.
   

2. Run the following command:

   kubectl create -f default_costmodel.yaml
   

   The output is similar to

   batchcostmodel.kbatch.k8s.io/default created
   

3. Verify the CostModel:

   kubectl get batchcostmodels
   

   The output is similar to

   NAME      AGE
   default   2m
   

4. Describe the CostModel:

   kubectl describe batchcostmodel default
   

   The output is similar to

   Name:         default
   Namespace:
   Labels:       controller-tools.k8s.io=1.0
   Annotations:  none
   API Version:  kbatch.k8s.io/v1beta1
   Kind:         BatchCostModel
   Metadata:
     Creation Timestamp:  2019-09-11T21:39:51Z
     Generation:          1
     Resource Version:    15534
     Self Link:           /apis/kbatch.k8s.io/v1beta1/batchcostmodels/default
     UID:                 afacef7a-d4dc-11e9-b6dc-42010a8000a6
   Spec:
     Resources:
       Cpu:                 0.031611
       Memory:              0.004237
       Nvidia . Com / Gpu:  0.95
   Events:                  none
   

Updating CostModels

To update the CostModel editdefault_costmodel.yaml then run the following command:

kubectl apply -f default_costmodel.yaml

Deleting CostModels

To delete a CostModel, run the following command:

kubectl delete batchcostmodel default

The output is similar to

batchcostmodel.kbatch.k8s.io "default" deleted

You can verify that the CostModel is deleted by running:

kubectl get batchcostmodels

The output is similar to

No resources found.

Creating and managing BatchBudgets

A budget defines how many resources a Queue may use or how much it can spend within a time period.

To create a BatchBudget, perform the following steps:

1. Create default_budget.yaml from the manifest below:

   apiVersion: kbatch.k8s.io/v1beta1
   kind: BatchBudget
   metadata:
     labels:
       controller-tools.k8s.io: "1.0"
     name: default
   spec:
     batchCostModelName: "default"
     # Two durations are currently supported: 24h and 720h (30d).
     # These may be combined, by specifying two budget windows, to enforce both a
     # daily and monthly limit.
     budgetWindows:
       - duration: "24h"
         amount: 100
   

2. Run the following command:

   kubectl create -f default_budget.yaml
   

   The output is similar to

   batchbudget.kbatch.k8s.io/default created
   

3. Verify the BatchBudget:

   kubectl get batchbudgets
   kubectl get batchbudgets --all-namespaces
   

   The output is similar to

   NAME      AGE
   default   2m
   

4. Describe the BatchBudget:

   kubectl describe batchbudget default
   

   The output is similar to

   Name:         default
   Namespace:
   Labels:       controller-tools.k8s.io=1.0
   Annotations:  kubectl.kubernetes.io/last-applied-configuration:
                   {"apiVersion":"kbatch.k8s.io/v1beta1","kind":"BatchBudget","metadata":{"annotations":{},"labels":{"controller-tools.k8s.io":"1.0"},"name"...
   API Version:  kbatch.k8s.io/v1beta1
   Kind:         BatchBudget
   Metadata:
     Creation Timestamp:  2019-09-11T23:57:10Z
     Finalizers:
       finalizers.kbatch.k8s.io/batchbudget
     Generation:        1
     Resource Version:  403104
     Self Link:         /apis/kbatch.k8s.io/v1beta1/batchbudgets/default
     UID:               de94325d-d4ef-11e9-a127-42010a8000d1
   Spec:
     Batch Cost Model Name:  default
     Budget Start Time:      2019-09-11T23:57:10Z
     Budget Windows:
       Amount:    100
       Duration:  24h0m0s
   Status:
     Last Updated:  2019-09-12T20:27:40Z
     Window Status List:
       Amount:      0.0003068580833333333
       Duration:    24h0m0s
       Start Time:  2019-09-11T23:57:10Z
   Events:          none
   

Updating BatchBudgets

To update the BatchBudget edit default_batchbudget.yaml then run the following command:

kubectl apply -f default_batchbudget.yaml

Deleting BatchBudgets

To delete a BatchBudget, run the following command:

kubectl delete batchbudget default

The output is similar to

batchbudget.kbatch.k8s.io "default" deleted

You can verify that the BatchBudget is deleted by running:

kubectl get batchbudgets

The output is similar to

No resources found.

Creating and managing Queues

Queues are a way to group, manage and reason closely related Jobs.

To create a BatchQueue, perform the following steps:

1. Create default_queue.yaml from the manifest below:

   apiVersion: kbatch.k8s.io/v1beta1
   kind: BatchQueue
   metadata:
     labels:
       controller-tools.k8s.io: "1.0"
     name: default
     namespace: default
   spec:
     batchPriorityName: high
     batchBudgetName: default
     constraintNames: ["default"]
   

2. Run the following command:

   kubectl apply -f default_queue.yaml
   

   The output is similar to

   batchqueue.kbatch.k8s.io/default created
   

3. Verify the BatchQueue:

   kubectl get batchqueues
   kubectl get batchqueues --all-namespaces
   

   The output is similar to

   NAME      AGE
   default   2m
   

4. Describe the BatchQueue:

   kubectl describe batchqueues
   

   The output is similar to

   Name:         default
   Namespace:    default
   Labels:       constraint_default=default
                 controller-tools.k8s.io=1.0
   Annotations:  kubectl.kubernetes.io/last-applied-configuration:
                   {"apiVersion":"kbatch.k8s.io/v1beta1","kind":"BatchQueue","metadata":{"annotations":{},"labels":{"controller-tools.k8s.io":"1.0"},"name":...
   API Version:  kbatch.k8s.io/v1beta1
   Kind:         BatchQueue
   Metadata:
     Creation Timestamp:  2019-09-11T23:57:12Z
     Finalizers:
       finalizers.kbatch.k8s.io/batchqueue
     Generation:        1
     Resource Version:  403107
     Self Link:         /apis/kbatch.k8s.io/v1beta1/namespaces/default/batchqueues/default
     UID:               dfdde64c-d4ef-11e9-b6dc-42010a8000a6
   Spec:
     Batch Budget Name:  default
     Batch Job Constraint Names:
       default
     Batch Priority Name:  high
   Status:
     Jobs:
       Last Updated:  2019-09-12T20:27:40Z
       Phase Counts:
         Queued:   0
         Ready:    0
         Running:  0
   Events:         none
   

Updating BatchQueues

To update the BatchQueue edit default_queue.yaml then run the following command:

kubectl apply -f default_queue.yaml

Deleting BatchQueues

Note: BatchQueue cannot be deleted if its jobs have not reached a terminal state, such as SUCCESS or FAILED. Once all jobs have reached terminal state, you can delete a BatchQueue.

To delete a BatchQueue, run the following command:

kubectl delete batchqueue default

The output is similar to

batchqueue.kbatch.k8s.io "default" deleted

You can verify that the BatchQueue is deleted by running:

kubectl get batchqueues

The output is similar to

No resources found.

What's next

• Manage Batch on GKE users
• Learn more about Batch on GKE and its resources