Nextflow

Nextflow

Workflow language for scalable and reproducible computational pipelines — write once, run anywhere (local, HPC, AWS, GCP, Azure).

When to Use This Skill

• Writing or debugging Nextflow DSL2 pipelines (.nf files)
• Composing processes into workflows with channel dataflow
• Configuring executors (SLURM, LSF, AWS Batch, Google Batch)
• Managing containers (Docker, Singularity/Apptainer, Conda) for reproducibility
• Building chemistry/bioinformatics pipelines (BLAST, aligners, RDKit, ORCA, Gaussian)
• Understanding -resume / cache behavior
• Modularizing pipelines with include / module aliases

Quick Start — Minimal DSL2 Pipeline

// main.nf
params.input  = 'data/*.sdf'
params.outdir = 'results'

process RUN_ORCA {
    publishDir params.outdir, mode: 'copy'
    container  'quay.io/biocontainers/orca:5.0.4--h2f1ea3e_0'

    input:
    path mol

    output:
    path "*.out"

    script:
    """
    orca ${mol}.inp > ${mol}.out
    """
}

workflow {
    mols = channel.fromPath(params.input)
    RUN_ORCA(mols)
}

Run it:

nextflow run main.nf -profile docker -resume

Router — What to Read

Task	Reference
Processes, channels, input/output qualifiers, script types	references/core-concepts.md
Workflows, named workflows, pipe/and operators, modules, composition	references/pipeline-patterns.md
nextflow.config, executors, profiles, HPC/cloud, cache/resume	references/execution-config.md
Docker, Apptainer/Singularity, Conda, Wave, reproducibility	references/containers-envs.md
Channel factories, operators, file handling, remote files	references/files-channels.md
Chemistry/bioinformatics patterns (BLAST, RDKit, ORCA, MD)	references/chem-bioinformatics.md

Key Concepts at a Glance

Concept	What it is
process	Runs a script/command; defines input, output, directives
workflow	Composes processes and operators via dataflow channels
channel	Asynchronous stream of values connecting processes
val / path	Input qualifiers — val for data, path for staged files
publishDir	Copies task output to a user-visible results directory
executor	Where tasks run: local, slurm, awsbatch, google-batch…
-resume	Reuses cached task results; skips unchanged tasks
module	Reusable .nf file included with include { X } from './module'

Installation

# Requires Java 11+
curl -s https://get.nextflow.io | bash
./nextflow self-update          # upgrade to latest
nextflow -version               # verify

# Enable DSL2 strict parser (recommended for new pipelines)
export NXF_SYNTAX_PARSER=v2

Related Skills

• rdkit — Molecular preprocessing before pipeline ingestion
• deepchem — ML models on molecular datasets (can be wrapped in NF processes)
• cheminformatics — SMILES, molecular file formats (SDF, MOL2, XYZ)