BatchQueues
Queues are a way to group and manage closely related Jobs.
Create a BatchQueue
To create a BatchQueue, note the name of an existing BatchPriority or create a new BatchPriority to be used.
The default_queue.yaml
file below is a simple manifest for a BatchQueue.
apiVersion: kbatch.k8s.io/v1beta1 kind: BatchQueue metadata: labels: controller-tools.k8s.io: "1.0" name: default namespace: default spec: batchPriorityName: critical # Attaching a budget here is optional. batchBudgetName: default # Attaching constraints here is optional. constraintNames: ["default"]
Create the BatchQueue:
kubectl apply -f default_queue.yaml
The output is similar to
batchqueue.kbatch.k8s.io/default created
List BatchQueues:
kubectl get batchqueues kubectl get batchqueues --all-namespaces
The output is similar to
NAME AGE default 2m
Describe BatchQueues:
kubectl describe batchqueue queuename
Example:
kubectl describe batchqueue default
The output is similar to
Name: default Namespace: default Labels: constraint_default=default controller-tools.k8s.io=1.0 Annotations: kubectl.kubernetes.io/last-applied-configuration: {"apiVersion":"kbatch.k8s.io/v1beta1","kind":"BatchQueue","metadata":{"annotations":{},"labels":{"controller-tools.k8s.io":"1.0"},"name":... API Version: kbatch.k8s.io/v1beta1 Kind: BatchQueue Metadata: Creation Timestamp: 2019-09-11T23:57:12Z Finalizers: finalizers.kbatch.k8s.io/batchqueue Generation: 1 Resource Version: 403107 Self Link: /apis/kbatch.k8s.io/v1beta1/namespaces/default/batchqueues/default UID: dfdde64c-d4ef-11e9-b6dc-42010a8000a6 Spec: Batch Budget Name: default Batch Job Constraint Names: default Batch Priority Name: critical Status: Jobs: Last Updated: 2019-09-12T20:27:40Z Phase Counts: Queued: 0 Ready: 0 Running: 0 Events: none
Stop scheduling existing jobs in a BatchQueue
To stop a BatchQueue from scheduling jobs in the queue that are not yet scheduled,
you can set PauseScheduling
to true
.
Example:
Update the default_queue.yaml
file as follows -
apiVersion: kbatch.k8s.io/v1beta1 kind: BatchQueue metadata: labels: controller-tools.k8s.io: "1.0" name: default namespace: default spec: batchPriorityName: critical pauseScheduling: true
Then update the BatchQueue with the following command -
kubectl apply -f default_queue.yaml
Stop admitting new jobs into a BatchQueue
To stop a BatchQueue from admitting new jobs you can set PauseAdmission
to true.
Update
default_queue.yaml
from the following manifest:apiVersion: kbatch.k8s.io/v1beta1 kind: BatchQueue metadata: labels: controller-tools.k8s.io: "1.0" name: default namespace: default spec: batchPriorityName: critical pauseAdmission: true
Apply the update to the BatchQueue:
kubectl apply -f default_queue.yaml
Update BatchQueues
To update a BatchQueue, edit the queue's manifest, for example,
default_queue.yaml
, and then apply the changes:
kubectl apply -f default_queue.yaml
Delete BatchQueues
To delete a BatchQueue, run the following command:
kubectl delete batchqueue default
The output is similar to
batchqueue.kbatch.k8s.io "default" deleted
You can verify that the BatchQueue is deleted by running:
kubectl get batchqueues
The output is similar to
No resources found.
Creating and managing Priorities
BatchPriorities let you prioritize jobs from high priority Queues above jobs from lower priority Queues.
To create a BatchPriority, perform the following steps:
Create
criticalpriority.yaml
from the manifest below:apiVersion: kbatch.k8s.io/v1beta1 kind: BatchPriority metadata: labels: controller-tools.k8s.io: "1.0" name: critical spec: value: 150 description: Critical Priority
Run the following command:
kubectl apply -f criticalpriority.yaml
The output is similar to
batchpriority.kbatch.k8s.io/critical created
Verify the BatchPriority:
kubectl get batchpriorities kubectl get batchpriorities --all-namespaces
The output is similar to
NAME AGE critical 1m
Describe the BatchPriority:
kubectl describe batchpriority critical
The output is similar to
Name: critical Namespace: Labels: controller-tools.k8s.io=1.0 Annotations: <none> API Version: kbatch.k8s.io/v1beta1 Metadata: Creation Timestamp: 2019-09-11T22:24:15Z Finalizers: finalizers.kbatch.k8s.io/batchpriority Generation: 1 Resource Version: 28112 Self Link: /apis/kbatch.k8s.io/v1beta1/batchpriorities/critical UID: e377117e-d4e2-11e9-a0de-42010a80005f Spec:Kind: BatchPriority Description: Critical Priority Value: 150 Status: Events: <none>
Updating BatchPriorities
To update the BatchPriority edit criticalpriority.yaml
then run the following
command:
kubectl apply -f criticalpriority.yaml
Deleting BatchPriorities
To delete a BatchPriority, run the following command:
kubectl delete batchpriority critical
The output is similar to
batchpriority.kbatch.k8s.io "critical" deleted
You can verify that the BatchBudget is deleted by running:
kubectl get batchpriority
The output is similar to
No resources found.
Creating and managing BatchJobConstraints [Optional]
A Queue can optionally have job constraints that specify the policies to accept jobs in a given Queue.
To create a BatchJobConstraints, perform the following steps:
Create
constraint.yaml
from the manifest below:apiVersion: kbatch.k8s.io/v1beta1 kind: BatchJobConstraint metadata: labels: controller-tools.k8s.io: "1.0" name: default spec: # The system supports the following constraints: # Cpu, Memory, WallTime, Gpu, GpuModel, RetryLimit # Adding a BatchJobConstraint to a BatchQueue means that the BatchQueue # will only accept jobs that satisfy the listed constraints. constraints: - name: WallTime operator: LessThan values: ["24h"]
Run the following command:
kubectl create -f constraint.yaml
The output is similar to
batchjobconstraint.kbatch.k8s.io/batchjobconstraint-sample created
Verify the BatchJobConstraints:
kubectl get batchjobconstraints kubectl get batchjobconstraints --all-namespaces
The output is similar to
NAME AGE batchjobconstraint-sample 45s
Describe the BatchJobConstraints:
kubectl describe batchjobconstraint batchjobconstraint-sample
The output is similar to
Name: batchjobconstraint-sample Namespace: Labels: <none> Annotations: <none> API Version: kbatch.k8s.io/v1beta1 Kind: BatchJobConstraint Metadata: Creation Timestamp: 2019-09-11T22:39:39Z Generation: 1 Resource Version: 32480 Self Link: /apis/kbatch.k8s.io/v1beta1/batchjobconstraints/batchjobconstraint-sample UID: 0a496ff2-d4e5-11e9-a0de-42010a80005f Spec: Constraints: Name: Memory Operator: LessThan Values: 16G Name: Cpu Operator: In Values: 1 2 4 Name: Gpu Operator: In Values: 1 2 4 Name: GpuModel Operator: Equal Values: nvidia-tesla-v100 Name: WallTime Operator: LessThan Values: 24h Name: RetryLimit Operator: LessThan Values: 2 Events: <none>
Updating BatchJobConstraints
To update BatchJobConstraints, edit constraint.yaml
then run the following
command:
kubectl apply -f constraint.yaml
Deleting BatchJobConstraints
To delete a BatchJobConstraints, run the following command:
kubectl delete batchjobconstraint batchjobconstraint-sample
The output is similar to
batchjobconstraint.kbatch.k8s.io "batchjobconstraint-sample" deleted
You can verify that the BatchJobConstraints is deleted by running:
kubectl get batchjobconstraints
The output is similar to
No resources found.
Creating and managing CostModels [Optional]
Batch uses CostModels to deduct resources from a budget. A CostModel is required only if you plan to use a BatchBudget for any of your BatchQueues.
To create a CostModel, perform the following steps:
Create
default_costmodel.yaml
from the manifest below:apiVersion: kbatch.k8s.io/v1beta1 kind: BatchCostModel metadata: labels: controller-tools.k8s.io: "1.0" name: default spec: resources: # These costs are hourly and in USD. For example, based on the numbers below, running using 1 CPU # 1 GB of RAM, and 1 Tesla T4 GPU for 2 hours would cost 2 * ($0.031611 + $0.004237 + $0.95) # The values are taken from https://cloud.google.com/compute/pricing for # region us-west1. The values are accurate as of April 2019. Actual pricing depends on various factors, # such as long-term contracts, use of standard machine types, the region, etc. cpu: 0.031611 # This is the on-demand price for a single vCPU memory: 0.004237 # This is the on-demand price for 1GB of memory nvidia.com/gpu: 0.95 # This is the cost for use of the Tesla T4 GPU. Currently # the system does not support breaking out prices for # specific GPU types, so an average price for the intended # GPUs should be used here.
Run the following command:
kubectl create -f default_costmodel.yaml
The output is similar to
batchcostmodel.kbatch.k8s.io/default created
Verify the CostModel:
kubectl get batchcostmodels
The output is similar to
NAME AGE default 2m
Describe the CostModel:
kubectl describe batchcostmodel default
The output is similar to
Name: default Namespace: Labels: controller-tools.k8s.io=1.0 Annotations: none API Version: kbatch.k8s.io/v1beta1 Kind: BatchCostModel Metadata: Creation Timestamp: 2019-09-11T21:39:51Z Generation: 1 Resource Version: 15534 Self Link: /apis/kbatch.k8s.io/v1beta1/batchcostmodels/default UID: afacef7a-d4dc-11e9-b6dc-42010a8000a6 Spec: Resources: Cpu: 0.031611 Memory: 0.004237 Nvidia . Com / Gpu: 0.95 Events: none
Updating CostModels
To update the CostModel editdefault_costmodel.yaml
then run the following
command:
kubectl apply -f default_costmodel.yaml
Deleting CostModels
To delete a CostModel, run the following command:
kubectl delete batchcostmodel default
The output is similar to
batchcostmodel.kbatch.k8s.io "default" deleted
You can verify that the CostModel is deleted by running:
kubectl get batchcostmodels
The output is similar to
No resources found.
Creating and managing BatchBudgets [Optional]
A Queue can optionally have a budget. A budget defines how much you spend for resources over a specified time window. A BatchCostModel specifies the cost of resources.
If no budget is attached to a Queue then its jobs are scheduled without using Budget as a critera.
To create a BatchBudget, perform the following steps:
Create
default_budget.yaml
from the manifest below:apiVersion: kbatch.k8s.io/v1beta1 kind: BatchBudget metadata: labels: controller-tools.k8s.io: "1.0" name: default spec: batchCostModelName: "default" # Two durations are currently supported: 24h and 720h (30d). # These may be combined, by specifying two budget windows, to enforce both a # daily and monthly limit. budgetWindows: - duration: "24h" amount: 100
Run the following command:
kubectl create -f default_budget.yaml
The output is similar to
batchbudget.kbatch.k8s.io/default created
Verify the BatchBudget:
kubectl get batchbudgets kubectl get batchbudgets --all-namespaces
The output is similar to
NAME AGE default 2m
Describe the BatchBudget:
kubectl describe batchbudget default
The output is similar to
Name: default Namespace: Labels: controller-tools.k8s.io=1.0 Annotations: kubectl.kubernetes.io/last-applied-configuration: {"apiVersion":"kbatch.k8s.io/v1beta1","kind":"BatchBudget","metadata":{"annotations":{},"labels":{"controller-tools.k8s.io":"1.0"},"name"... API Version: kbatch.k8s.io/v1beta1 Kind: BatchBudget Metadata: Creation Timestamp: 2019-09-11T23:57:10Z Finalizers: finalizers.kbatch.k8s.io/batchbudget Generation: 1 Resource Version: 403104 Self Link: /apis/kbatch.k8s.io/v1beta1/batchbudgets/default UID: de94325d-d4ef-11e9-a127-42010a8000d1 Spec: Batch Cost Model Name: default Budget Start Time: 2019-09-11T23:57:10Z Budget Windows: Amount: 100 Duration: 24h0m0s Status: Last Updated: 2019-09-12T20:27:40Z Window Status List: Amount: 0.0003068580833333333 Duration: 24h0m0s Start Time: 2019-09-11T23:57:10Z Events: none
Updating BatchBudgets
To update the BatchBudget edit default_batchbudget.yaml
then run the following
command:
kubectl apply -f default_batchbudget.yaml
Deleting BatchBudgets
To delete a BatchBudget, run the following command:
kubectl delete batchbudget default
The output is similar to
batchbudget.kbatch.k8s.io "default" deleted
You can verify that the BatchBudget is deleted by running:
kubectl get batchbudgets
The output is similar to
No resources found.