Unity slurm

Unity Slurm

Use this skill when the user needs Unity-specific scheduler help and generic Slurm knowledge is not enough. Keep explanations short; focus on Unity naming, GPU request patterns, live inspection, and failure triage.

Use This For

• choosing the right Unity partition
• requesting a specific GPU type on Unity
• checking what hardware or partitions are actually available right now
• understanding why a Unity job is pending, rejected, or OOMing
• translating Unity docs into concrete sbatch or srun choices

Do not spend space explaining basic sbatch, squeue, or srun semantics unless Unity differs from standard Slurm expectations.

Partition Model

Unity uses Slurm with a mix of public/shared and group-owned partitions.

• If no partition is specified, jobs default to batch.
• Public GPU work typically goes to batch-gpu or batch-gpu-new.
• Many labs or groups have exclusive partitions; if the user has one, prefer it before overflowing to batch.
• Multiple partitions can be requested as a comma-separated list when that matches local policy.

Useful Unity partition names that came up in the docs and team comms:

• Public/shared: batch, batch-gpu, batch-gpu-new
• Group or exclusive examples: white-1gpu, white-2gpu, moortgat-gpu, moortgat-l40-gpu, moortgat-l40s-gpu, ast, econ, math, stat, trivedi, xiu

Treat partition docs as hints, not truth. Unity hardware changes over time; live sinfo and scontrol beat stale notes.

GPU Requests

Unity supports typed GPU requests. Use them when the exact accelerator matters.

Examples:

sbatch -p batch-gpu --gres=gpu:1 job.sbatch
sbatch -p batch-gpu-new --gres=gpu:h200:1 job.sbatch
sbatch -p moortgat-l40s-gpu --gres=gpu:l40s:1 job.sbatch

Known Unity GPU patterns from the docs:

• batch-gpu includes older public GPUs and may also include newer cards depending on current cluster configuration
• batch-gpu-new is the main public lane to target H200 nodes when available
• Typed requests like gpu:h200:1, gpu:l40s:1, or gpu:v100:1 are appropriate when the partition exposes that hardware

Request one larger GPU when model memory is the problem. More small GPUs do not help if the application loads the model onto a single device.

Live Inspection

Use live inspection before assuming the docs are current.

Primary command from the Unity docs:

sinfo -N -o "%.8N %.5c %.10m %.20R %.20G %.f "

Useful variants:

sinfo -N -o "%.8N %.5c %.10m %.20R %.20G %.f " | grep -i h200
sinfo -N -p batch-gpu,batch-gpu-new -o "%.8N %.20R %.20G %.f "
scontrol show partition
scontrol show node <node>
squeue -u "$USER"
scontrol show job <jobid>
seff <jobid>

Interpretation:

• sinfo tells you what nodes and GRES exist now
• scontrol show partition tells you partition limits and access details
• scontrol show job shows the scheduler's view of a pending or failed job
• seff is useful after completion to see whether the resource request matched reality

Queue And Failure Triage

Separate scheduler failures from runtime failures.

Scheduler-side:

• sbatch returns a job ID: the request syntax was accepted
• job stays pending: this is a queue or placement problem, not a submission syntax problem
• immediate rejection about partition, account, or GRES: fix the request or permissions

Runtime-side:

• CUDA unavailable: wrong node type, wrong environment, or no GPU was actually assigned
• CUDA out of memory: the card is too small for the current workload, or the workload needs quantization or a larger GPU
• import or package failures: environment issue, not a scheduler issue

Unity-specific guidance:

• If a GPU job OOMs on a 12 to 16 GB public card, the first fix is usually a larger GPU request, not more GPUs
• If sbatch accepted --gres=gpu:h200:1, the scheduler syntax is probably fine even if the job waits a long time
• If squeue shows a maintenance-style reason such as reserved or unavailable nodes, the queue is blocked by cluster state, not your script

Scheduler Guidance Specific To Unity

• Do not run installs, indexing, model loads, or heavy preprocessing on the login nodes
• For real compute, use sbatch, sinteractive, or an OnDemand-launched compute session
• On Unity, shared module roots and package caches may be read-only; do not assume named conda envs under the system root are writable
• Prefix envs are often safer than named envs on this cluster
• Do not over-request memory. Asking for nearly all advertised node RAM can make scheduling much harder or force placement onto unintended nodes

Quick Patterns

Public GPU smoke test:

sbatch -p batch-gpu job.sbatch

Explicit H200 request:

sbatch -p batch-gpu-new --gres=gpu:h200:1 job.sbatch

Check whether Unity actually has the GPU you want right now:

sinfo -N -o "%.8N %.5c %.10m %.20R %.20G %.f " | grep -i 'h200\\|l40\\|v100\\|p100'

Debug a pending job:

squeue -j <jobid>
scontrol show job <jobid>