Managing Batch on GKE resources

Deprecated: Batch on GKE is now deprecated. For more information on running your HPC applications in Google Cloud, see High performance computing.

BatchQueues

Queues are a way to group and manage closely related Jobs.

Create a BatchQueue

To create a BatchQueue, note the name of an existing BatchPriority or create a new BatchPriority to be used.

The default_queue.yaml file below is a simple manifest for a BatchQueue.

   apiVersion: kbatch.k8s.io/v1beta1
   kind: BatchQueue
   metadata:
     labels:
       controller-tools.k8s.io: "1.0"
     name: default
     namespace: default
   spec:
     batchPriorityName: critical
     # Attaching a budget here is optional.
     batchBudgetName: default
     # Attaching constraints here is optional.
     constraintNames: ["default"]
   

Create the BatchQueue:

   kubectl apply -f default_queue.yaml
   

The output is similar to

   batchqueue.kbatch.k8s.io/default created
   

List BatchQueues:

   kubectl get batchqueues
   kubectl get batchqueues --all-namespaces
   

The output is similar to

   NAME      AGE
   default   2m
   

Describe BatchQueues:

   kubectl describe batchqueue queuename
   

Example:

   kubectl describe batchqueue default
   

The output is similar to

    Name:         default
    Namespace:    default
    Labels:       constraint_default=default
                  controller-tools.k8s.io=1.0
    Annotations:  kubectl.kubernetes.io/last-applied-configuration:
                    {"apiVersion":"kbatch.k8s.io/v1beta1","kind":"BatchQueue","metadata":{"annotations":{},"labels":{"controller-tools.k8s.io":"1.0"},"name":...
    API Version:  kbatch.k8s.io/v1beta1
    Kind:         BatchQueue
    Metadata:
      Creation Timestamp:  2019-09-11T23:57:12Z
      Finalizers:
        finalizers.kbatch.k8s.io/batchqueue
      Generation:        1
      Resource Version:  403107
      Self Link:         /apis/kbatch.k8s.io/v1beta1/namespaces/default/batchqueues/default
      UID:               dfdde64c-d4ef-11e9-b6dc-42010a8000a6
    Spec:
      Batch Budget Name:  default
      Batch Job Constraint Names:
        default
      Batch Priority Name:  critical
    Status:
      Jobs:
        Last Updated:  2019-09-12T20:27:40Z
        Phase Counts:
          Queued:   0
          Ready:    0
          Running:  0
    Events:         none
   

Stop scheduling existing jobs in a BatchQueue

To stop a BatchQueue from scheduling jobs in the queue that are not yet scheduled, you can set PauseScheduling to true.

Example:

Update the default_queue.yaml file as follows -

    apiVersion: kbatch.k8s.io/v1beta1
    kind: BatchQueue
    metadata:
      labels:
        controller-tools.k8s.io: "1.0"
      name: default
      namespace: default
    spec:
      batchPriorityName: critical
      pauseScheduling: true
   

Then update the BatchQueue with the following command -

kubectl apply -f default_queue.yaml

Stop admitting new jobs into a BatchQueue

To stop a BatchQueue from admitting new jobs you can set PauseAdmission to true.

1. Update default_queue.yaml from the following manifest:

   apiVersion: kbatch.k8s.io/v1beta1
   kind: BatchQueue
   metadata:
     labels:
       controller-tools.k8s.io: "1.0"
     name: default
     namespace: default
   spec:
     batchPriorityName: critical
     pauseAdmission: true
   

2. Apply the update to the BatchQueue:

   kubectl apply -f default_queue.yaml
   

Update BatchQueues

To update a BatchQueue, edit the queue's manifest, for example, default_queue.yaml, and then apply the changes:

kubectl apply -f default_queue.yaml

Delete BatchQueues

Note: BatchQueue cannot be deleted if its jobs have not reached a terminal state, such as SUCCESS or FAILED. Once all jobs have reached terminal state, you can delete a BatchQueue.

To delete a BatchQueue, run the following command:

kubectl delete batchqueue default

The output is similar to

batchqueue.kbatch.k8s.io "default" deleted

You can verify that the BatchQueue is deleted by running:

kubectl get batchqueues

The output is similar to

No resources found.

Creating and managing Priorities

BatchPriorities let you prioritize jobs from high priority Queues above jobs from lower priority Queues.

To create a BatchPriority, perform the following steps:

1. Create criticalpriority.yaml from the manifest below:

   apiVersion: kbatch.k8s.io/v1beta1
   kind: BatchPriority
   metadata:
     labels:
       controller-tools.k8s.io: "1.0"
     name: critical
   spec:
     value: 150
     description: Critical Priority
   

2. Run the following command:

   kubectl apply -f criticalpriority.yaml
   

   The output is similar to

   batchpriority.kbatch.k8s.io/critical created
   

3. Verify the BatchPriority:

   kubectl get batchpriorities
   kubectl get batchpriorities --all-namespaces
   

   The output is similar to

   NAME      AGE
   critical  1m
   

   Note: Any previously created priorities will also appear in this output.

4. Describe the BatchPriority:

   kubectl describe batchpriority critical
   

   The output is similar to

   Name:         critical
   Namespace:
   Labels:       controller-tools.k8s.io=1.0
   Annotations:  <none>
   API Version:  kbatch.k8s.io/v1beta1
   
   Metadata:
     Creation Timestamp:  2019-09-11T22:24:15Z
     Finalizers:
       finalizers.kbatch.k8s.io/batchpriority
     Generation:        1
     Resource Version:  28112
     Self Link:         /apis/kbatch.k8s.io/v1beta1/batchpriorities/critical
     UID:               e377117e-d4e2-11e9-a0de-42010a80005f
   Spec:Kind:         BatchPriority
     Description:  Critical Priority
     Value:        150
   Status:
   Events:  <none>
   

Updating BatchPriorities

To update the BatchPriority edit criticalpriority.yaml then run the following command:

kubectl apply -f criticalpriority.yaml

Deleting BatchPriorities

To delete a BatchPriority, run the following command:

kubectl delete batchpriority critical

The output is similar to

batchpriority.kbatch.k8s.io "critical" deleted

You can verify that the BatchBudget is deleted by running:

kubectl get batchpriority

The output is similar to

No resources found.

Note: Any previously created priorities will also appear in this output.

Creating and managing BatchJobConstraints [Optional]

A Queue can optionally have job constraints that specify the policies to accept jobs in a given Queue.

To create a BatchJobConstraints, perform the following steps:

1. Create constraint.yaml from the manifest below:

   apiVersion: kbatch.k8s.io/v1beta1
   kind: BatchJobConstraint
   metadata:
     labels:
       controller-tools.k8s.io: "1.0"
     name: default
   spec:
     # The system supports the following constraints:
     # Cpu, Memory, WallTime, Gpu, GpuModel, RetryLimit
     # Adding a BatchJobConstraint to a BatchQueue means that the BatchQueue
     # will only accept jobs that satisfy the listed constraints.
     constraints:
       - name: WallTime
         operator: LessThan
         values: ["24h"]
   

2. Run the following command:

   kubectl create -f constraint.yaml
   

   The output is similar to

   batchjobconstraint.kbatch.k8s.io/batchjobconstraint-sample created
   

3. Verify the BatchJobConstraints:

   kubectl get batchjobconstraints
   kubectl get batchjobconstraints --all-namespaces
   

   The output is similar to

   NAME                        AGE
   batchjobconstraint-sample   45s
   

4. Describe the BatchJobConstraints:

   kubectl describe batchjobconstraint batchjobconstraint-sample
   

   The output is similar to

   Name:         batchjobconstraint-sample
   Namespace:
   Labels:       <none>
   Annotations:  <none>
   API Version:  kbatch.k8s.io/v1beta1
   Kind:         BatchJobConstraint
   Metadata:
     Creation Timestamp:  2019-09-11T22:39:39Z
     Generation:          1
     Resource Version:    32480
     Self Link:           /apis/kbatch.k8s.io/v1beta1/batchjobconstraints/batchjobconstraint-sample
     UID:                 0a496ff2-d4e5-11e9-a0de-42010a80005f
   Spec:
     Constraints:
       Name:      Memory
       Operator:  LessThan
       Values:
         16G
       Name:      Cpu
       Operator:  In
       Values:
         1
         2
         4
       Name:      Gpu
       Operator:  In
       Values:
         1
         2
         4
       Name:      GpuModel
       Operator:  Equal
       Values:
         nvidia-tesla-v100
       Name:      WallTime
       Operator:  LessThan
       Values:
         24h
       Name:      RetryLimit
       Operator:  LessThan
       Values:
         2
   Events:  <none>
   

Updating BatchJobConstraints

To update BatchJobConstraints, edit constraint.yaml then run the following command:

kubectl apply -f constraint.yaml

Deleting BatchJobConstraints

Note: If the BatchJobConstraint is being used by any BatchQueue, it cannot be deleted.

To delete a BatchJobConstraints, run the following command:

kubectl delete batchjobconstraint batchjobconstraint-sample

The output is similar to

batchjobconstraint.kbatch.k8s.io "batchjobconstraint-sample" deleted

You can verify that the BatchJobConstraints is deleted by running:

kubectl get batchjobconstraints

The output is similar to

No resources found.

Creating and managing CostModels [Optional]

Batch uses CostModels to deduct resources from a budget. A CostModel is required only if you plan to use a BatchBudget for any of your BatchQueues.

To create a CostModel, perform the following steps:

1. Create default_costmodel.yaml from the manifest below:

   apiVersion: kbatch.k8s.io/v1beta1
   kind: BatchCostModel
   metadata:
     labels:
       controller-tools.k8s.io: "1.0"
     name: default
   spec:
     resources:
       # These costs are hourly and in USD. For example, based on the numbers below, running using 1 CPU
       # 1 GB of RAM, and 1 Tesla T4 GPU for 2 hours would cost 2 * ($0.031611 + $0.004237 + $0.95)
       # The values are taken from https://cloud.google.com/compute/pricing for
       # region us-west1. The values are accurate as of April 2019. Actual pricing depends on various factors,
       # such as long-term contracts, use of standard machine types, the region, etc.
       cpu: 0.031611         # This is the on-demand price for a single vCPU
       memory: 0.004237      # This is the on-demand price for 1GB of memory
       nvidia.com/gpu: 0.95  # This is the cost for use of the Tesla T4 GPU. Currently
                             # the system does not support breaking out prices for
                             # specific GPU types, so an average price for the intended
                             # GPUs should be used here.
   

2. Run the following command:

   kubectl create -f default_costmodel.yaml
   

   The output is similar to

   batchcostmodel.kbatch.k8s.io/default created
   

3. Verify the CostModel:

   kubectl get batchcostmodels
   

   The output is similar to

   NAME      AGE
   default   2m
   

4. Describe the CostModel:

   kubectl describe batchcostmodel default
   

   The output is similar to

   Name:         default
   Namespace:
   Labels:       controller-tools.k8s.io=1.0
   Annotations:  none
   API Version:  kbatch.k8s.io/v1beta1
   Kind:         BatchCostModel
   Metadata:
     Creation Timestamp:  2019-09-11T21:39:51Z
     Generation:          1
     Resource Version:    15534
     Self Link:           /apis/kbatch.k8s.io/v1beta1/batchcostmodels/default
     UID:                 afacef7a-d4dc-11e9-b6dc-42010a8000a6
   Spec:
     Resources:
       Cpu:                 0.031611
       Memory:              0.004237
       Nvidia . Com / Gpu:  0.95
   Events:                  none
   

Updating CostModels

To update the CostModel editdefault_costmodel.yaml then run the following command:

kubectl apply -f default_costmodel.yaml

Deleting CostModels

To delete a CostModel, run the following command:

kubectl delete batchcostmodel default

The output is similar to

batchcostmodel.kbatch.k8s.io "default" deleted

You can verify that the CostModel is deleted by running:

kubectl get batchcostmodels

The output is similar to

No resources found.

Creating and managing BatchBudgets [Optional]

A Queue can optionally have a budget. A budget defines how much you spend for resources over a specified time window. A BatchCostModel specifies the cost of resources.

If no budget is attached to a Queue then its jobs are scheduled without using Budget as a critera.

To create a BatchBudget, perform the following steps:

1. Create default_budget.yaml from the manifest below:

   apiVersion: kbatch.k8s.io/v1beta1
   kind: BatchBudget
   metadata:
     labels:
       controller-tools.k8s.io: "1.0"
     name: default
   spec:
     batchCostModelName: "default"
     # Two durations are currently supported: 24h and 720h (30d).
     # These may be combined, by specifying two budget windows, to enforce both a
     # daily and monthly limit.
     budgetWindows:
       - duration: "24h"
         amount: 100
   

2. Run the following command:

   kubectl create -f default_budget.yaml
   

   The output is similar to

   batchbudget.kbatch.k8s.io/default created
   

3. Verify the BatchBudget:

   kubectl get batchbudgets
   kubectl get batchbudgets --all-namespaces
   

   The output is similar to

   NAME      AGE
   default   2m
   

4. Describe the BatchBudget:

   kubectl describe batchbudget default
   

   The output is similar to

   Name:         default
   Namespace:
   Labels:       controller-tools.k8s.io=1.0
   Annotations:  kubectl.kubernetes.io/last-applied-configuration:
                   {"apiVersion":"kbatch.k8s.io/v1beta1","kind":"BatchBudget","metadata":{"annotations":{},"labels":{"controller-tools.k8s.io":"1.0"},"name"...
   API Version:  kbatch.k8s.io/v1beta1
   Kind:         BatchBudget
   Metadata:
     Creation Timestamp:  2019-09-11T23:57:10Z
     Finalizers:
       finalizers.kbatch.k8s.io/batchbudget
     Generation:        1
     Resource Version:  403104
     Self Link:         /apis/kbatch.k8s.io/v1beta1/batchbudgets/default
     UID:               de94325d-d4ef-11e9-a127-42010a8000d1
   Spec:
     Batch Cost Model Name:  default
     Budget Start Time:      2019-09-11T23:57:10Z
     Budget Windows:
       Amount:    100
       Duration:  24h0m0s
   Status:
     Last Updated:  2019-09-12T20:27:40Z
     Window Status List:
       Amount:      0.0003068580833333333
       Duration:    24h0m0s
       Start Time:  2019-09-11T23:57:10Z
   Events:          none
   

Updating BatchBudgets

To update the BatchBudget edit default_batchbudget.yaml then run the following command:

kubectl apply -f default_batchbudget.yaml

Deleting BatchBudgets

To delete a BatchBudget, run the following command:

kubectl delete batchbudget default

The output is similar to

batchbudget.kbatch.k8s.io "default" deleted

You can verify that the BatchBudget is deleted by running:

kubectl get batchbudgets

The output is similar to

No resources found.

What's next

• Manage Batch on GKE users
• Learn more about Batch on GKE and its resources