Managing Batch on GKE resources

BatchQueues

Queues are a way to group and manage closely related Jobs.

Create a BatchQueue

To create a BatchQueue, note the name of an existing BatchPriority or create a new BatchPriority to be used.

The default_queue.yaml file below is a simple manifest for a BatchQueue.

   apiVersion: kbatch.k8s.io/v1beta1
   kind: BatchQueue
   metadata:
     labels:
       controller-tools.k8s.io: "1.0"
     name: default
     namespace: default
   spec:
     batchPriorityName: critical
     # Attaching a budget here is optional.
     batchBudgetName: default
     # Attaching constraints here is optional.
     constraintNames: ["default"]
   

Create the BatchQueue:

   kubectl apply -f default_queue.yaml
   

The output is similar to

   batchqueue.kbatch.k8s.io/default created
   

List BatchQueues:

   kubectl get batchqueues
   kubectl get batchqueues --all-namespaces
   

The output is similar to

   NAME      AGE
   default   2m
   

Describe BatchQueues:

   kubectl describe batchqueue queuename
   

Example:

   kubectl describe batchqueue default
   

The output is similar to

    Name:         default
    Namespace:    default
    Labels:       constraint_default=default
                  controller-tools.k8s.io=1.0
    Annotations:  kubectl.kubernetes.io/last-applied-configuration:
                    {"apiVersion":"kbatch.k8s.io/v1beta1","kind":"BatchQueue","metadata":{"annotations":{},"labels":{"controller-tools.k8s.io":"1.0"},"name":...
    API Version:  kbatch.k8s.io/v1beta1
    Kind:         BatchQueue
    Metadata:
      Creation Timestamp:  2019-09-11T23:57:12Z
      Finalizers:
        finalizers.kbatch.k8s.io/batchqueue
      Generation:        1
      Resource Version:  403107
      Self Link:         /apis/kbatch.k8s.io/v1beta1/namespaces/default/batchqueues/default
      UID:               dfdde64c-d4ef-11e9-b6dc-42010a8000a6
    Spec:
      Batch Budget Name:  default
      Batch Job Constraint Names:
        default
      Batch Priority Name:  critical
    Status:
      Jobs:
        Last Updated:  2019-09-12T20:27:40Z
        Phase Counts:
          Queued:   0
          Ready:    0
          Running:  0
    Events:         none
   

Stop scheduling existing jobs in a BatchQueue

To stop a BatchQueue from scheduling jobs in the queue that are not yet scheduled, you can set PauseScheduling to true.

Example:

Update the default_queue.yaml file as follows -

    apiVersion: kbatch.k8s.io/v1beta1
    kind: BatchQueue
    metadata:
      labels:
        controller-tools.k8s.io: "1.0"
      name: default
      namespace: default
    spec:
      batchPriorityName: critical
      pauseScheduling: true
   

Then update the BatchQueue with the following command -

kubectl apply -f default_queue.yaml

Stop admitting new jobs into a BatchQueue

To stop a BatchQueue from admitting new jobs you can set PauseAdmission to true.

  1. Update default_queue.yaml from the following manifest:

    apiVersion: kbatch.k8s.io/v1beta1
    kind: BatchQueue
    metadata:
      labels:
        controller-tools.k8s.io: "1.0"
      name: default
      namespace: default
    spec:
      batchPriorityName: critical
      pauseAdmission: true
    
  2. Apply the update to the BatchQueue:

    kubectl apply -f default_queue.yaml
    

Update BatchQueues

To update a BatchQueue, edit the queue's manifest, for example, default_queue.yaml, and then apply the changes:

kubectl apply -f default_queue.yaml

Delete BatchQueues

To delete a BatchQueue, run the following command:

kubectl delete batchqueue default

The output is similar to

batchqueue.kbatch.k8s.io "default" deleted

You can verify that the BatchQueue is deleted by running:

kubectl get batchqueues

The output is similar to

No resources found.

Creating and managing Priorities

BatchPriorities let you prioritize jobs from high priority Queues above jobs from lower priority Queues.

To create a BatchPriority, perform the following steps:

  1. Create criticalpriority.yaml from the manifest below:

    apiVersion: kbatch.k8s.io/v1beta1
    kind: BatchPriority
    metadata:
      labels:
        controller-tools.k8s.io: "1.0"
      name: critical
    spec:
      value: 150
      description: Critical Priority
    
  2. Run the following command:

    kubectl apply -f criticalpriority.yaml
    

    The output is similar to

    batchpriority.kbatch.k8s.io/critical created
    
  3. Verify the BatchPriority:

    kubectl get batchpriorities
    kubectl get batchpriorities --all-namespaces
    

    The output is similar to

    NAME      AGE
    critical  1m
    
  4. Describe the BatchPriority:

    kubectl describe batchpriority critical
    

    The output is similar to

    Name:         critical
    Namespace:
    Labels:       controller-tools.k8s.io=1.0
    Annotations:  <none>
    API Version:  kbatch.k8s.io/v1beta1
    
    Metadata:
      Creation Timestamp:  2019-09-11T22:24:15Z
      Finalizers:
        finalizers.kbatch.k8s.io/batchpriority
      Generation:        1
      Resource Version:  28112
      Self Link:         /apis/kbatch.k8s.io/v1beta1/batchpriorities/critical
      UID:               e377117e-d4e2-11e9-a0de-42010a80005f
    Spec:Kind:         BatchPriority
      Description:  Critical Priority
      Value:        150
    Status:
    Events:  <none>
    

Updating BatchPriorities

To update the BatchPriority edit criticalpriority.yaml then run the following command:

kubectl apply -f criticalpriority.yaml

Deleting BatchPriorities

To delete a BatchPriority, run the following command:

kubectl delete batchpriority critical

The output is similar to

batchpriority.kbatch.k8s.io "critical" deleted

You can verify that the BatchBudget is deleted by running:

kubectl get batchpriority

The output is similar to

No resources found.

Creating and managing BatchJobConstraints [Optional]

A Queue can optionally have job constraints that specify the policies to accept jobs in a given Queue.

To create a BatchJobConstraints, perform the following steps:

  1. Create constraint.yaml from the manifest below:

    apiVersion: kbatch.k8s.io/v1beta1
    kind: BatchJobConstraint
    metadata:
      labels:
        controller-tools.k8s.io: "1.0"
      name: default
    spec:
      # The system supports the following constraints:
      # Cpu, Memory, WallTime, Gpu, GpuModel, RetryLimit
      # Adding a BatchJobConstraint to a BatchQueue means that the BatchQueue
      # will only accept jobs that satisfy the listed constraints.
      constraints:
        - name: WallTime
          operator: LessThan
          values: ["24h"]
    
  2. Run the following command:

    kubectl create -f constraint.yaml
    

    The output is similar to

    batchjobconstraint.kbatch.k8s.io/batchjobconstraint-sample created
    
  3. Verify the BatchJobConstraints:

    kubectl get batchjobconstraints
    kubectl get batchjobconstraints --all-namespaces
    

    The output is similar to

    NAME                        AGE
    batchjobconstraint-sample   45s
    
  4. Describe the BatchJobConstraints:

    kubectl describe batchjobconstraint batchjobconstraint-sample
    

    The output is similar to

    Name:         batchjobconstraint-sample
    Namespace:
    Labels:       <none>
    Annotations:  <none>
    API Version:  kbatch.k8s.io/v1beta1
    Kind:         BatchJobConstraint
    Metadata:
      Creation Timestamp:  2019-09-11T22:39:39Z
      Generation:          1
      Resource Version:    32480
      Self Link:           /apis/kbatch.k8s.io/v1beta1/batchjobconstraints/batchjobconstraint-sample
      UID:                 0a496ff2-d4e5-11e9-a0de-42010a80005f
    Spec:
      Constraints:
        Name:      Memory
        Operator:  LessThan
        Values:
          16G
        Name:      Cpu
        Operator:  In
        Values:
          1
          2
          4
        Name:      Gpu
        Operator:  In
        Values:
          1
          2
          4
        Name:      GpuModel
        Operator:  Equal
        Values:
          nvidia-tesla-v100
        Name:      WallTime
        Operator:  LessThan
        Values:
          24h
        Name:      RetryLimit
        Operator:  LessThan
        Values:
          2
    Events:  <none>
    

Updating BatchJobConstraints

To update BatchJobConstraints, edit constraint.yaml then run the following command:

kubectl apply -f constraint.yaml

Deleting BatchJobConstraints

To delete a BatchJobConstraints, run the following command:

kubectl delete batchjobconstraint batchjobconstraint-sample

The output is similar to

batchjobconstraint.kbatch.k8s.io "batchjobconstraint-sample" deleted

You can verify that the BatchJobConstraints is deleted by running:

kubectl get batchjobconstraints

The output is similar to

No resources found.

Creating and managing CostModels [Optional]

Batch uses CostModels to deduct resources from a budget. A CostModel is required only if you plan to use a BatchBudget for any of your BatchQueues.

To create a CostModel, perform the following steps:

  1. Create default_costmodel.yaml from the manifest below:

    apiVersion: kbatch.k8s.io/v1beta1
    kind: BatchCostModel
    metadata:
      labels:
        controller-tools.k8s.io: "1.0"
      name: default
    spec:
      resources:
        # These costs are hourly and in USD. For example, based on the numbers below, running using 1 CPU
        # 1 GB of RAM, and 1 Tesla T4 GPU for 2 hours would cost 2 * ($0.031611 + $0.004237 + $0.95)
        # The values are taken from https://cloud.google.com/compute/pricing for
        # region us-west1. The values are accurate as of April 2019. Actual pricing depends on various factors,
        # such as long-term contracts, use of standard machine types, the region, etc.
        cpu: 0.031611         # This is the on-demand price for a single vCPU
        memory: 0.004237      # This is the on-demand price for 1GB of memory
        nvidia.com/gpu: 0.95  # This is the cost for use of the Tesla T4 GPU. Currently
                              # the system does not support breaking out prices for
                              # specific GPU types, so an average price for the intended
                              # GPUs should be used here.
    
  2. Run the following command:

    kubectl create -f default_costmodel.yaml
    

    The output is similar to

    batchcostmodel.kbatch.k8s.io/default created
    
  3. Verify the CostModel:

    kubectl get batchcostmodels
    

    The output is similar to

    NAME      AGE
    default   2m
    
  4. Describe the CostModel:

    kubectl describe batchcostmodel default
    

    The output is similar to

    Name:         default
    Namespace:
    Labels:       controller-tools.k8s.io=1.0
    Annotations:  none
    API Version:  kbatch.k8s.io/v1beta1
    Kind:         BatchCostModel
    Metadata:
      Creation Timestamp:  2019-09-11T21:39:51Z
      Generation:          1
      Resource Version:    15534
      Self Link:           /apis/kbatch.k8s.io/v1beta1/batchcostmodels/default
      UID:                 afacef7a-d4dc-11e9-b6dc-42010a8000a6
    Spec:
      Resources:
        Cpu:                 0.031611
        Memory:              0.004237
        Nvidia . Com / Gpu:  0.95
    Events:                  none
    

Updating CostModels

To update the CostModel editdefault_costmodel.yaml then run the following command:

kubectl apply -f default_costmodel.yaml

Deleting CostModels

To delete a CostModel, run the following command:

kubectl delete batchcostmodel default

The output is similar to

batchcostmodel.kbatch.k8s.io "default" deleted

You can verify that the CostModel is deleted by running:

kubectl get batchcostmodels

The output is similar to

No resources found.

Creating and managing BatchBudgets [Optional]

A Queue can optionally have a budget. A budget defines how much you spend for resources over a specified time window. A BatchCostModel specifies the cost of resources.

If no budget is attached to a Queue then its jobs are scheduled without using Budget as a critera.

To create a BatchBudget, perform the following steps:

  1. Create default_budget.yaml from the manifest below:

    apiVersion: kbatch.k8s.io/v1beta1
    kind: BatchBudget
    metadata:
      labels:
        controller-tools.k8s.io: "1.0"
      name: default
    spec:
      batchCostModelName: "default"
      # Two durations are currently supported: 24h and 720h (30d).
      # These may be combined, by specifying two budget windows, to enforce both a
      # daily and monthly limit.
      budgetWindows:
        - duration: "24h"
          amount: 100
    
  2. Run the following command:

    kubectl create -f default_budget.yaml
    

    The output is similar to

    batchbudget.kbatch.k8s.io/default created
    
  3. Verify the BatchBudget:

    kubectl get batchbudgets
    kubectl get batchbudgets --all-namespaces
    

    The output is similar to

    NAME      AGE
    default   2m
    
  4. Describe the BatchBudget:

    kubectl describe batchbudget default
    

    The output is similar to

    Name:         default
    Namespace:
    Labels:       controller-tools.k8s.io=1.0
    Annotations:  kubectl.kubernetes.io/last-applied-configuration:
                    {"apiVersion":"kbatch.k8s.io/v1beta1","kind":"BatchBudget","metadata":{"annotations":{},"labels":{"controller-tools.k8s.io":"1.0"},"name"...
    API Version:  kbatch.k8s.io/v1beta1
    Kind:         BatchBudget
    Metadata:
      Creation Timestamp:  2019-09-11T23:57:10Z
      Finalizers:
        finalizers.kbatch.k8s.io/batchbudget
      Generation:        1
      Resource Version:  403104
      Self Link:         /apis/kbatch.k8s.io/v1beta1/batchbudgets/default
      UID:               de94325d-d4ef-11e9-a127-42010a8000d1
    Spec:
      Batch Cost Model Name:  default
      Budget Start Time:      2019-09-11T23:57:10Z
      Budget Windows:
        Amount:    100
        Duration:  24h0m0s
    Status:
      Last Updated:  2019-09-12T20:27:40Z
      Window Status List:
        Amount:      0.0003068580833333333
        Duration:    24h0m0s
        Start Time:  2019-09-11T23:57:10Z
    Events:          none
    

Updating BatchBudgets

To update the BatchBudget edit default_batchbudget.yaml then run the following command:

kubectl apply -f default_batchbudget.yaml

Deleting BatchBudgets

To delete a BatchBudget, run the following command:

kubectl delete batchbudget default

The output is similar to

batchbudget.kbatch.k8s.io "default" deleted

You can verify that the BatchBudget is deleted by running:

kubectl get batchbudgets

The output is similar to

No resources found.

What's next