Slurm job debug template

Purpose

Generate a conservative sbatch template, submit it to Slurm, and capture the accounting record needed to verify the cluster path end to end.

When to use

• You need a tiny Slurm smoke job.
• You want a reusable starting point for queue, environment, and log validation.

When not to use

• You need multi-node or GPU tuning guidance beyond a first smoke job.

Inputs

• Command string
• Optional job name, partition, runtime, memory, and output path

Outputs

• Rendered sbatch script
• JSON submission and sacct summary

Requirements

• Python 3.13+
• sbatch, squeue, and sacct
• A real Slurm cluster

Procedure

1. Run python3 skills/hpc/slurm-job-debug-template/scripts/render_sbatch.py --command "echo hello" --job-name smoke.
2. Inspect the generated script or save it with --out.
3. Submit a verified smoke job with python3 skills/hpc/slurm-job-debug-template/scripts/submit_smoke_job.py --partition cpu --job-name slurm-smoke --sleep 2 --out slurm/reports/slurm-smoke.json.
4. Inspect the returned job_id, the log paths in slurm/logs/, and the accounting block from sacct.

Validation

• Renderer exits successfully.
• Output contains #SBATCH directives and the command body.
• Submitted smoke job reaches State=COMPLETED.
• ExitCode is 0:0.

Failure modes and fixes

• Missing partition/account information: add them before submission.
• Output paths wrong: switch to cluster-appropriate scratch or log paths.
• sacct lags briefly after completion: retry once accounting catches up.

Safety and limits

• Keep resource requests small for initial smoke jobs.
• Use a CPU partition and a short walltime for smoke checks.

Examples

• python3 .../render_sbatch.py --command "python analysis.py" --partition short --time 00:10:00 --mem 2G
• python3 .../submit_smoke_job.py --partition cpu --job-name slurm-smoke --sleep 1

Provenance

• Slurm quick start: https://slurm.schedmd.com/quickstart.html
• Slurm sbatch reference: https://slurm.schedmd.com/sbatch.html
• sacct reference: https://slurm.schedmd.com/sacct.html

Related skills

• snakemake-toy-workflow-starter