Prior to Release 2.2.0, job scheduling operated on a First In, First Out (FIFO) basis. To prioritize a job, users had to manually coordinate job cancellations and resubmissions among themselves. The job priority feature introduced in Release 2.2.0 enhances user experience by allowing job ordering based on assigned priorities rather than submission times.

​
Job Priority Definition

Jobs are sorted into four priority buckets: P0 (highest priority) to P3 (lowest priority). Within each bucket, individual jobs are ranked based on numerical priority values; lower numbers signify higher priority. Scheduling prioritizes jobs based on these values within designated queues.

​
Detailed Workflow

​
Submitting Jobs with Priority

Users can specify a job priority at submission using --job_priority with run.py. Valid inputs are p1, p2, and p3, corresponding to default values of 199, 299, and 399, respectively. If unspecified, jobs default to P2 (299).

​
Updating Job Priority

Users can adjust a job’s priority post-submission using Command-Line Interface for Job Monitoring(csctl). There are two ways to implement this:
  • By Bucket Name: Update using P0, P1, P2, or P3.
  • By Priority Value: Directly assign a numerical value within the priority bucket ranges.
The mapping between priority values and buckets is crucial for understanding the scheduling precedence.

​
Priority rules

  • Rule #1: Only admin users and users with the same GID can modify a job’s priority.
  • Rule #2: Only admin users can access the P0 bucket (0 to 99).
  • Rule #3: Completed jobs cannot have their priorities updated.

​
Monitoring job priorities

Use csctl to view the priorities of jobs, which aids in understanding the scheduling order and managing workflows effectively. When listing jobs, users will see both priority values and their corresponding buckets, providing clarity on job scheduling order.

​
Priority Continuity

Jobs stemming from the same run.py invocation inherit the workflow’s priority. Updating a workflow’s priority affects all associated pending and subsequent jobs, ensuring consistent priority across a workflow.

​
Usage Enhancements

In addition to allowing users to specify and update job priorities, several other changes enhance the functionality and user experience:

​
Priority column in job listings

The csctl get job command now includes a priority column displaying “bucket-name (priority-value)” for clearer priority visualization.

​
Job sorting

Jobs are now sorted first by priority and then by age within the csctl get job listings. Higher priority jobs appear first, and among jobs with the same priority, newer jobs are listed before older ones.

​
Enhanced csctl commands

New csctl job commands have been introduced to improve job management. The aim is to consolidate all job-related commands under csctl job, enhancing the command structure and help menu.

​
Additional usage examples

1. Submitting a job with specified priority: 
$ python run.py --job_priority p1 \[flags\]
This command submits a job with a priority set to P1. 2. Updating a job’s priority to a different bucket: 
$ csctl job set-priority wsjob-xxxxxx p2
This updates the job’s priority to P2. Confirmation is displayed: 
job-operator/wsjob-jv58f5mb95kwpe5hwuujrk priority successfully updated.
3. Listing jobs with their priorities: 
$ csctl job list -a
Displays a list of jobs along with details such as type, priority, age, duration, phase, system, user, labels, and dashboard status. For example, it shows that: 
NAME                                       TYPE     PRIORITY  AGE    DURATION  PHASE      SYSTEMS  USER    LABELS            DASHBOARD
job-operator/wsjob-jv58f5mb95kwpe5hwuujrk  execute  P2 (200)  3h56m  88s       SUCCEEDED  kapi-1   myuser  wsjob-label=test  grafana not deployed
job-operator/wsjob-2ercynhxb44ajhg4qn9bem  compile  P1 (150)  3h58m  2m21s     FAILED              myuser  wsjob-label=test  grafana not deployed

​
Conclusion

This enhanced user guide offers a comprehensive understanding of the job priority feature, empowering users to optimize job scheduling and execution based on priority settings, thereby improving workflow efficiency and system utilization.