Skip to main content

Documentation Index

Fetch the complete documentation index at: https://training-docs.cerebras.ai/llms.txt

Use this file to discover all available pages before exploring further.

Job scheduling is determined by assigned priorities rather than submission time. Jobs are categorized into four priority buckets, ranging from P0 (highest) to P3 (lowest). Within each bucket, jobs are ranked by numerical priority values, where lower numbers indicate higher priority. Scheduling follows these rankings within designated queues.

​
Submit Jobs with Priority

Users can specify a job priority at submission using the --job_priority flag. For example: 
cszoo fit params_model.yaml [additional-args] --job_priority p1
Valid inputs are p1, p2, and p3, corresponding to default values of 199, 299, and 399, respectively. If unspecified, jobs default to P2 (299). Learn more about the additional arguments for launching jobs here.

​
Update Job Priority

Users can view and adjust a job’s priority post-submission using the csctl tool.
  • Only admin users and users with the same GID can modify a job’s priority.
  • Only admin users can access the P0 bucket (0 to 99).
  • Completed jobs cannot have their priorities updated.

​
Monitoring Job Priorities

Use the csctl tool to view job priorities: 
cstcl get jobs
When listing jobs, you will see both priority values and their corresponding buckets. Jobs are first sorted by priority and then by age. Higher-priority jobs appear first, and within the same priority level, newer jobs are listed before older ones.

​
Priority Continuity

Jobs stemming from the same invocation inherit the workflow’s priority. Updating a workflow’s priority affects all associated pending and subsequent jobs, ensuring consistent priority across a workflow.

​
Additional usage examples

Update a job’s priority to a different bucket: 
$ csctl job set-priority wsjob-xxxxxx -n mynamespace p2
This updates the job’s priority to P2. Confirmation is displayed: 
successfully updated wsjob-jv58f5mb95kwpe5hwuujrk job priority to p2
List jobs with their priorities: 
$ csctl job list -a
Displays a list of jobs along with details such as type, priority, age, duration, phase, system, user, labels, and dashboard status. For example, it shows that: 
NAME                                       TYPE     PRIORITY  AGE    DURATION  PHASE      SYSTEMS  USER    LABELS            DASHBOARD
job-operator/wsjob-jv58f5mb95kwpe5hwuujrk  execute  P2 (200)  3h56m  88s       SUCCEEDED  kapi-1   myuser  wsjob-label=test  grafana not deployed
job-operator/wsjob-2ercynhxb44ajhg4qn9bem  compile  P1 (150)  3h58m  2m21s     FAILED              myuser  wsjob-label=test  grafana not deployed