What is Preemption
Our clusters, with Slurm, currently have what is called preemption enabled which is the act of stopping one or more low-priority jobs to let a high-priority job run.
If you have submitted a job that has been allocated to a compute node that is part of another account’s investment then your job could be preempted.
If anyone part of the investment’s account submits a job, then their job will be allocated to run on their investment compute nodes. Any jobs currently running on that list of investment nodes, by users not part of the investment account, will have their jobs stopped and automatically resubmitted to the Slurm queue.
If you do not want your jobs to be preempted, then within your submission script request the non-investor partition which contains nodes not part of any investment.
Further details can be found:
Slurm: Preemption
Job Scheduling Policy: Job Preemption
Beartooth: Condo Model