Micromamba container

Docker + Micromamba Skill

Package projects into Docker containers using micromamba for fast, reliable dependency management.

⚠️ Critical Rules

These two rules cause the hardest-to-debug failures if ignored. Read them before writing any files.

1. NEVER include channels in environment.yml — mirror URLs are configured via .condarc in the Dockerfile only. Putting channels in environment.yml overrides .condarc and causes micromamba to fall back to conda.anaconda.org, resulting in SSL errors, DNS failures, and timeouts.
2. NEVER install deep learning frameworks (PyTorch, TensorFlow, JAX) via conda — conda pulls the complete CUDA toolkit (4-6GB), duplicating any CUDA in the base image. Always use pip. If unsure whether a package pulls pytorch, use scripts/detect_cuda_pull.sh <pkg1> [pkg2] ... to verify. See deep-learning.md.

Workflow

Step 1: Verify Package Names

Before writing environment.yml, verify every package name and source. Never guess package names — always search conda-forge, bioconda, or PyPI to confirm each package exists and get its correct name. Substituting a similar-sounding package can cause silent runtime failures.

For each dependency the user requests:

1. Search conda-forge first: check https://anaconda.org/conda-forge/<package-name> or use micromamba search <package-name>
2. Search bioconda if needed: check https://anaconda.org/bioconda/<package-name>
3. Search PyPI if it's a Python-only package: check https://pypi.org/project/<package-name>/
4. If not found in any channel, ask the user for direction — do NOT substitute similar-sounding packages

Naming conventions to remember:

• R packages from CRAN on conda-forge use r- prefix: r-seurat, r-qs2, r-data.table
• R packages from Bioconductor on bioconda use bioconductor- prefix: bioconductor-genomeinfodb, bioconductor-DESeq2
• Python packages use their PyPI name: scanpy, anndata, pandas
• Some packages exist on both conda and PyPI — prefer conda for binary packages (numpy, scipy, etc.)

When to STOP and ASK:

• Package not found: ask the user for the correct name or alternative.
• Multiple possibilities: present options and ask for clarification.
• Secondary conda channel required: present options (secondary channel vs pip vs GitHub). When a secondary channel IS used, the environment.yml MUST specify the channel prefix (e.g. genomedk::r-doubletfinder=2.0.4).
• R package not available on conda: see the R Projects section for the source install workflow.

Step 2: Pre-Build Validation

Before writing environment.yml, run these two checks to catch build failures early. Each takes only a few seconds and saves 15-20 minutes of wasted rebuild time.

2a. Python Version Compatibility Check

For every package, verify its Python version constraint is compatible with the selected Python version. A package may exist on conda-forge or PyPI but require a newer Python than what you're using.

# PyPI packages — check requires_python via JSON API
curl -s https://pypi.org/pypi/<package>/json | python -c "import sys,json; d=json.load(sys.stdin); print(d['info']['requires_python'])"

If incompatible: STOP and ask the user which Python version to target, or add a version pin (e.g., scanpy<1.10 for Python 3.8).

2b. Pip Package Import Dependency Scan

For packages installed via pip (not conda), scan their source for top-level import statements to find undeclared dependencies:

bash scripts/scan_pip_imports.sh <package-name>

If missing dependencies found: add them to environment.yml (prefer conda). If system libraries needed: If the build fails with libxxx.so not found, use apt-file search libxxx.so to find the package name, then add apt-get install to the Dockerfile.

Note: The scan is a heuristic — it may produce false positives (imports inside if TYPE_CHECKING: blocks, optional features). Use judgment to determine which missing imports are actual hard dependencies vs. optional extras.

Step 3: Create environment.yml

Every micromamba Docker build starts with an environment.yml. This file declares all conda dependencies.

name: myapp
dependencies:
  - python=3.11
  - pip
  # Add conda packages here
  - numpy
  - pandas
  # pip packages go in a nested pip section
  - pip:
    - some-pypi-package

Key rules:

• Always include python
• Do NOT include channels in environment.yml — see Critical Rule 1 for why
• Pin Python version explicitly (e.g., python=3.11)
• Put pip-only packages under - pip: as a nested list

Step 4: Choose a Dockerfile Pattern

Choose the right template for your project:

Pattern	Use when	Template
Basic	Simple Python/conda project	assets/templates/basic/Dockerfile
Mixed pip	conda + pip packages	assets/templates/mixed-pip/Dockerfile
Mixed npm	conda + Node.js packages	assets/templates/mixed-npm/Dockerfile
Jupyter	Data science notebooks	assets/templates/jupyter/Dockerfile
GPU	PyTorch/TensorFlow with CUDA	assets/templates/gpu/Dockerfile

User rules when adapting templates:

• ALWAYS use USER root for all installation steps — the base image defaults to mambauser which cannot write to /etc/ or /usr/. Keep USER root for every RUN that installs software, switch to USER mambauser only as the final step. Do NOT create a new user with useradd — mambauser already exists.
• ALWAYS order Dockerfile instructions by change frequency — put stable layers first (mirror config, environment.yml), heavy pip packages (PyTorch) in their own RUN step, and application code last. This way, changing application code doesn't invalidate the dependency installation cache.
• NEVER use git clone inside the Dockerfile — base micromamba images do NOT have git installed. Clone on the host first, then COPY the source into the build context.

Auto-detect mirror region: templates use Tsinghua mirrors by default. Run the following to automatically remove mirror configs if you are outside China:

# If outside China, remove mirror config lines to use default repositories
if [ "$(curl -s --connect-timeout 2 https://ipinfo.io/country 2>/dev/null)" != "CN" ]; then
    sed -i "/configs\/tsinghua/d" Dockerfile
fi

Mirror rules:

• Mirror URLs go in .condarc only; environment.yml should NOT have a channels key (it overrides .condarc and causes fallback to conda.anaconda.org)
• Order matters: conda-forge before bioconda (bioconda depends on conda-forge)
• channel_priority: strict is critical — without it, micromamba runs in flexible mode and will still attempt to reach conda.anaconda.org as a fallback, causing SSL errors in restricted networks
• Mirror fallback order: Tsinghua → Aliyun → USTC → default. For resilience, add these env vars to the Dockerfile:

  ENV MAMBA_DOWNLOAD_RETRIES=15
  ENV MAMBA_REMOTE_FETCH_TIMEOUT_SECS=180
  

Before building, automatically create a .dockerignore file. This reduces build context size and speeds up builds:

.git
__pycache__
*.pyc
.env
.venv
node_modules
*.md
tests/
docs/

Add project-specific exclusions based on what you find in the directory — look for large data files (*.h5, *.fastq, *.bam), output directories (data/, results/, output/), and local config files (.env.local, secrets/).

All patterns use one of two base images depending on GPU needs:

• GPU (CUDA 12.4): mambaorg/micromamba:cuda12.4.1-ubuntu22.04
• CPU only: mambaorg/micromamba:debian12-slim — note: icu>=76 requires a newer libstdc++.so.6 than Debian 12 provides. If you see CXXABI_1.3.15 not found, pin icu<76 in environment.yml.

R Projects

If the project uses R, you need to modify the chosen Dockerfile template. Follow the complete R Source Install Checklist, which covers:

• Scanning R package dependencies from DESCRIPTION files (scripts/scan_r_deps.sh)
• Configuring CRAN mirrors dynamically based on location
• Installing R packages from GitHub/CRAN/Bioconductor source with dependencies=FALSE
• The correct Dockerfile step ordering (conda packages → Rprofile.site → source install)

Quick checklist before building:

1. All conda-resolvable R deps are in environment.yml (with r- / bioconductor- prefixes, see Step 1)
2. CRAN mirror URL is set in the Dockerfile (not inherited from host shell)
3. Source installs use dependencies=FALSE — conda already installed the deps

Step 5: Build, Lock, Verify, and Clean

Execute these sub-steps in order. Do NOT skip any.

5.1 Detect Runtime and Determine Tag

source scripts/detect_runtime.sh

IMAGE_NAME=$(basename "$(pwd)")
TAG="$(date +%Y%m%d)-$(git rev-parse --short HEAD 2>/dev/null || echo local)"
IMAGE_TAG="${IMAGE_NAME}:${TAG}"

Tag naming rules:

• Format: <image-name>:YYYYMMDD-<short-hash> (or local if no git repo)
• Do NOT add a latest tag
• If the user provides an image reference, parse it intelligently:
  • myapp → myapp:20260401-local
  • myapp:v1.0 → myapp:v1.0 (user's tag takes precedence)
  • myrepo/myapp → myrepo/myapp:20260401-local
  • myrepo/myapp:v1.0 → myrepo/myapp:v1.0
• If no image name is provided, derive from the project directory name

5.2 Build the Image

timeout 7200 $RUNTIME build $BUILD_FLAGS -t "${IMAGE_TAG}" .

ALWAYS execute the build — do not just show the commands. The timeout 7200 wrapper ensures the build fails after 2 hours instead of hanging indefinitely.

5.3 Lock Resolved Versions (MANDATORY — do NOT skip)

After a successful build, export the resolved package versions and update environment.yml. This is what makes the build reproducible — without it, the next build may pull different package versions and break.

# Extract pinned versions for the packages the user requested
# Example: user asked for samtools, bedtools, numpy, r-ggplot2
bash scripts/lock_versions.sh "$RUNTIME" "${IMAGE_TAG}" samtools bedtools numpy r-ggplot2 > /tmp/locked-deps.txt

Then update environment.yml by replacing the unpinned package names with the pinned versions from the script output. For example, replace - numpy with - numpy=1.26.4. Keep the name, python, and pip: sections intact — only replace the package lines with their pinned equivalents.

5.4 Verify Installation

Run verification commands for each user-requested package inside the container to confirm they import correctly.

ALWAYS prefix commands with micromamba run — the base image defaults to mambauser without an active conda environment, so bare commands like python script.py or R -e "..." will fail.

# For Python packages:
$RUNTIME run --rm "${IMAGE_TAG}" micromamba run python -c "import numpy; import pandas; print('Python packages OK')"

# For R packages:
$RUNTIME run --rm "${IMAGE_TAG}" micromamba run R -e "library(ggplot2); library(ggpubr); cat('R packages OK\n')"

# If the image has multiple package types, run ALL applicable verification commands.

Rules:

• Include all user-requested packages (Python: import, R: library() without r- prefix)
• End with a success message (print() or cat()) so the output is unambiguous
• For R packages that print long banners (e.g. ComplexHeatmap), wrap in suppressPackageStartupMessages()

5.5 Clean Dangling Images (MANDATORY — do NOT skip)

$RUNTIME image prune -f

5.6 Report to User

Report the following to the user:

• Image name and tag
• Key package versions (from the locked environment.yml)
• Verification result (pass/fail)
• Build failures (if multiple attempts were needed, summarize error, root cause, and fix)