Overview

Effective January 13, 2025, changes were made to the Medicinebow job scheduler. These change are detailed here. These changes may result in your jobs not running the same way they did previously, or in queue for a longer period of time. Please reference the troubleshooting section below for common problems post maintenance. Below is a table of contents with a list of common issues.

Troubleshooting Error Messages

sbatch/salloc: error: Interactive jobs cannot be longer than 8 hours

Post maintenance, interactive jobs are restricted to an 8 hour walltime. Please submit your salloc command with a walltime 8 hours or less.
Example:

salloc -A projectname -t 8:00:00

I can no longer request an OnDemand job more than 8 hours

To encourage users to use only the time they need, all interactive jobs, including those requested through OnDemand have been limited to 8 hours in length. Please specify a time from the OnDemand webform under 8 hours.

sbatch/salloc: error: Use of --mem=0 is not permitted. Consider using --exclusive instead

Users may no longer request all memory on a node using the --mem=0 flag. If you know you need the use of an entire node, replace your --mem=0 flag specification in your job with --exclusive to get use of an entire node an all it’s resources.

sbatch/salloc: error: QOSMinGRES

Users must specify a GPU device if requesting a GPU partition. Assuming you plan to use a GPU in your computations, please specify a GPU by including either the --gres=gpu:# or --gpus-per-node=# flag in your job submission.

sbatch/salloc: error: Job submit/allocate failed: Job violates accounting/QOS policy (job submit limit, user's size and/or time limits)

This may occur for a number of reasons. Please e-mail arcc-help@uwyo.edu with the location of the batch script you’re attempting to run, or salloc command you’re attempting to run, and the error message you receive.

salloc: error: Interactive jobs must be run under the 'interactive' QOS or 'debug' QOS, not 'fast'

Users must specify the interactive or debug queue, or a time under 8 hrs when requesting an interactive job.

salloc: error: Job submit/allocate failed: Invalid qos specification

Users should specify times that match queues. i.e.,
Debug (<= 1 hr)
Interactive (<= 8 hrs)
Fast (< = 12 hrs)
Normal (<= 3 days)
Long (<= 7 days)

sbatch: error: Batch job submission failed: Requested node configuration is not available

This may occur for a number of reasons, but is likely due to the combination of nodes and hardware you’ve requested, and whether that hardware is available on the node/partition. If you need assistance please e-mail arcc-help@uwyo.edu with the location of the batch script you’re attempting to run, or salloc command you’re attempting to run, and the error message you receive.

My job has been sitting in queue for a very long time without running

This is usually the result of specified walltime. If you have specified a 7 day walltime in your job using --time or -t flag over 3 days, you will be placed in the “long” queue which may result in a longer wait time. If your job doesn’t require 7 days, please try specifying a shorter walltime (ideally under 3 days). This should result in your job being placed in a queue with a shorter wait time.