Optimization project structure

Optimization Project Structure

You are an expert in structuring research code for combinatorial optimization: package layout, configuration systems, algorithm/problem factories, seeding discipline, atomic result writing, and a light testing tier. This skill is a pattern catalog: each pattern gives a short motivation, a complete implementation, and the pitfall that most often invalidates experiments. Use the framework below to size the structure to the project, then adapt the matching patterns.

Initial Assessment

Establish the following before recommending any structure:

• Project lifetime and stakes. A one-week exploration tolerates a single script. A thesis or paper project will produce thousands of runs over months and must support the question "which code and parameters produced this number in Table 3?"
• Count the axes of variation. How many problems, algorithms, parameter settings, instances, and seeds will be combined? The product of these axes is the number of runs the structure must name, store, and aggregate without collisions.
• Compute environment. Laptop only, or also a cluster/colleague's machine? Anything beyond one machine forbids absolute paths, machine-local state, and manual run bookkeeping.
• Current state of the code. Greenfield, a pile of notebooks, or a working flat script? Migration order matters: extract the library first, configs second, orchestration last.
• Result volume and format. Hundreds of scalar records fit JSON-per-run plus a CSV aggregate. Millions of rows or per-iteration traces need parquet and a separate trace directory from day one.
• Failure tolerance of campaigns. Will runs take seconds or hours? Long campaigns need resumability (skip completed runs) and per-run failure isolation; short ones do not.
• Sources of randomness. List them: metaheuristic moves, instance generation, tie-breaking, solver internals. Every source must trace back to a recorded seed.
• External solvers and licenses. Solver version and parameter files are part of provenance; a Gurobi version bump can change every number in a results table.
• Who else runs this code. Solo use allows conventions in your head; a second user (advisor, reviewer, future you in 18 months) requires the conventions to be in the repository: README, configs, and tests.
• What the paper needs. Identify the final artifacts (tables, figures, per-instance bests) and design the results directory and record schema backward from them.

Project Anatomy and the Run Contract

The reference layout, sized for a typical metaheuristic-vs-baseline paper project:

myproject/
├── pyproject.toml            # package metadata; enables `pip install -e .`
├── README.md                 # how to install, run one experiment, run tests
├── configs/
│   ├── base.yaml             # campaign definitions: data, not code
│   └── sweeps/
│       └── ils_kick.yaml
├── data/
│   └── instances/            # read-only inputs; never written by runs
├── results/                  # write-only outputs; gitignored; regenerable
│   ├── raw/                  # one JSON record per run (immutable)
│   ├── tables/               # aggregated CSV/parquet, regenerated by scripts
│   └── figures/
├── scripts/
│   ├── run_experiment.py     # thin entry point: args -> config -> library
│   ├── aggregate_results.py
│   └── make_figures.py
├── src/
│   └── myopt/
│       ├── __init__.py       # imports problem/algorithm modules -> fills registries
│       ├── algorithms/       # one module per algorithm
│       ├── problems/         # one module per problem class
│       ├── io/               # instance parsing, atomic result writing
│       ├── config.py
│       ├── provenance.py
│       ├── registry.py
│       └── seeding.py
└── tests/
    └── test_core.py          # fast tier; slow tests marked separately

Each layer has one rule that gives the separation its value:

Layer	Lives in	Rule
Library code	src/myopt/	Importable, no import-time work, no hard-coded paths, no I/O surprises
Entry points	scripts/	Thin: parse arguments, load config, call the library, nothing else
Experiment definitions	configs/	Data, not code; committed to git; one file per campaign
Inputs	data/instances/	Read-only; runs never modify instances
Outputs	results/	Write-only by runs; gitignored; raw records immutable
Tests	tests/	Fast tier runs in seconds, always; slow tier opt-in

The run contract. Everything in this skill serves one formal statement. A run is a function

$$ r = \mathrm{run}(I,; A,; \theta,; s) $$

mapping an instance $I$, an algorithm $A$, a parameter vector $\theta$, and a seed $s$ to a result record $r$ (objective, solution, runtime, status, provenance). Reproducibility means: given the same $(I, A, \theta, s)$ and the same code version, $\mathrm{run}$ returns the same record up to wall-clock fields. The project structure exists to make every argument of this function explicit, recorded, and recoverable — instances by name in data/, algorithms by registered name, $\theta$ in a config file, $s$ in the record, and the code version as a git hash inside $r$. This is the operational core of Sandve et al. (2013), "Ten Simple Rules for Reproducible Computational Research," and of the reporting standards argued by Johnson (2002), "A Theoretician's Guide to the Experimental Analysis of Algorithms."

Decision guidance — how much structure:

• Single flat script when: one problem, one algorithm, exploration that will be discarded. Still use default_rng(seed) and print the seed; throwaway code has a habit of surviving.
• Script + config file when: one algorithm but parameter studies begin. The moment you edit a constant in code to launch a second run, move constants to a config.
• Full layout above when: two or more algorithms or problems will be compared, or any result might appear in a paper. Half a day of migration, amortized over every later run.
• Do not add plugin systems, abstract base class towers, or microservice splits. Research code optimizes for change velocity and auditability, not for extensibility by strangers (Wilson et al. 2017, "Good Enough Practices in Scientific Computing").

Complexity note: runtime overhead is negligible — config parsing and JSON writing cost microseconds against seconds-to-hours solves. The cost is cognitive, paid once, and repaid on every run, debug session, and reviewer question.

Configuration Patterns

Pattern: frozen dataclass configs loaded from YAML/JSON

Configs as plain dicts spread cfg["algorithm"]["params"]["kick_strenght"] typos through the code and fail silently. A frozen dataclass gives attribute access, type hints, one schema definition, and immutability — nothing downstream can quietly edit the experiment definition. Unknown keys must be rejected at load time: a misspelled YAML key must crash, not be ignored.

# src/myopt/config.py
from __future__ import annotations

import json
from dataclasses import dataclass, field, fields
from pathlib import Path
from typing import Any

import yaml


@dataclass(frozen=True)
class AlgorithmConfig:
    """Which algorithm to build and with which parameters."""

    name: str
    params: dict[str, Any] = field(default_factory=dict)


@dataclass(frozen=True)
class ExperimentConfig:
    """Full description of one experiment campaign."""

    experiment_name: str
    instances: list[str]
    algorithm: AlgorithmConfig
    seeds: list[int]
    time_limit_s: float
    results_dir: str = "results/raw"


def _build(cls: type, raw: dict[str, Any]) -> Any:
    """Construct a dataclass from a dict, rejecting unknown keys loudly."""
    known = {f.name for f in fields(cls)}
    unknown = set(raw) - known
    if unknown:
        raise ValueError(f"unknown config keys for {cls.__name__}: {sorted(unknown)}")
    kwargs = dict(raw)
    if isinstance(kwargs.get("algorithm"), dict):
        kwargs["algorithm"] = _build(AlgorithmConfig, kwargs["algorithm"])
    return cls(**kwargs)


def load_config(path: str | Path) -> ExperimentConfig:
    """Load an ExperimentConfig from a .yaml/.yml or .json file."""
    path = Path(path)
    text = path.read_text(encoding="utf-8")
    raw = yaml.safe_load(text) if path.suffix in {".yaml", ".yml"} else json.loads(text)
    return _build(ExperimentConfig, raw)


# Tiny synthetic instance: parse a config from text and read it back.
cfg_text = """
experiment_name: ils_baseline
instances: [rand50_a, rand50_b]
algorithm: {name: ils, params: {kick_strength: 4, max_iters: 2000}}
seeds: [0, 1, 2]
time_limit_s: 10.0
"""
cfg = _build(ExperimentConfig, yaml.safe_load(cfg_text))
print(cfg.algorithm.name, cfg.seeds, cfg.results_dir)
# Expected: "ils [0, 1, 2] results/raw"; a misspelled key such as 'sedes' raises ValueError

Pitfall: defaults defined in two places. If kick_strength defaults to 3 in the algorithm's __init__ and is 4 in base.yaml, the recorded config and the executed parameters diverge whenever a config omits the key. Rule: defaults live in exactly one place (usually the constructor), and the result record stores the resolved parameters actually passed to the constructor, never the raw config fragment.

Pattern: run identity from a canonical config hash

Runs need stable, collision-free names for files and skip-completed logic. Deriving the name from a hash of the canonical JSON of (parameters, instance, seed) makes identity follow content: the same experiment always maps to the same filename, any change produces a new one.

# src/myopt/run_id.py
import hashlib
import json
from typing import Any


def canonical_json(obj: Any) -> str:
    """Serialize with sorted keys so semantically equal configs hash equally."""
    return json.dumps(obj, sort_keys=True, separators=(",", ":"), default=str)


def run_id(algo_config: dict[str, Any], instance: str, seed: int) -> str:
    """Stable short identifier for one (algorithm config, instance, seed) run."""
    payload = canonical_json({"algorithm": algo_config, "instance": instance, "seed": seed})
    digest = hashlib.sha256(payload.encode("utf-8")).hexdigest()[:12]
    return f"{instance}_s{seed}_{digest}"


algo = {"name": "ils", "params": {"kick_strength": 4, "max_iters": 2000}}
reordered = {"params": {"max_iters": 2000, "kick_strength": 4}, "name": "ils"}
print(run_id(algo, "rand50_a", 7))
print(run_id(algo, "rand50_a", 7) == run_id(reordered, "rand50_a", 7))
# Expected: an id like "rand50_a_s7_<12 hex chars>", then True (key order is irrelevant)

Pitfall: hashing objects with unstable string forms. default=str makes numpy scalars and Path objects serializable, but a Python set serializes in arbitrary order and a float computed two ways may differ in the last bit. Keep config values JSON-native (str, int, float, bool, list, dict), and hash the inputs that generate derived floats, never the floats.

Factory and Interface Patterns

Pattern: registry with decorator registration

The run script must turn the string "ils" from a config file into an algorithm object. A chain of if name == "ils": ... branches grows stale and hides what is available. A registry maps names to factories, registration happens next to the class definition, and the error message for an unknown name lists every valid option.

# src/myopt/registry.py
from __future__ import annotations

from typing import Callable, TypeVar

T = TypeVar("T")


class Registry:
    """Name -> factory mapping with decorator-based registration."""

    def __init__(self, kind: str) -> None:
        self.kind = kind
        self._factories: dict[str, Callable[..., object]] = {}

    def register(self, name: str) -> Callable[[Callable[..., T]], Callable[..., T]]:
        """Decorator that registers a class or factory function under `name`."""
        def wrap(factory: Callable[..., T]) -> Callable[..., T]:
            if name in self._factories:
                raise ValueError(f"{self.kind} '{name}' registered twice")
            self._factories[name] = factory
            return factory
        return wrap

    def create(self, name: str, **kwargs: object) -> object:
        """Instantiate the registered factory, failing with the list of known names."""
        if name not in self._factories:
            raise KeyError(f"unknown {self.kind} '{name}'; known: {self.names()}")
        return self._factories[name](**kwargs)

    def names(self) -> list[str]:
        """Sorted registered names — useful in tests and error messages."""
        return sorted(self._factories)


ALGORITHMS = Registry("algorithm")
PROBLEMS = Registry("problem")


@ALGORITHMS.register("ils")
class IteratedLocalSearch:
    """Stand-in algorithm class; real moves live in their own module."""

    def __init__(self, kick_strength: int = 3, max_iters: int = 1000) -> None:
        self.kick_strength = kick_strength
        self.max_iters = max_iters


algo = ALGORITHMS.create("ils", kick_strength=5)
print(type(algo).__name__, algo.kick_strength, ALGORITHMS.names())
# Expected: "IteratedLocalSearch 5 ['ils']"; ALGORITHMS.create("tabu") raises KeyError naming 'ils'

Pitfall: the empty-registry trap. Registration runs when the defining module is imported; if nothing imports myopt/algorithms/ils.py, the registry stays empty and create("ils") fails only at run time. Fix structurally: myopt/__init__.py imports every algorithm and problem module explicitly, and a one-line test asserts the expected names exist. Avoid importlib directory scans — they trade an explicit list for import-order surprises.

Pattern: Protocol interfaces between problems and algorithms

Algorithms should depend on a minimal problem interface, not on concrete classes, so a new problem plugs into every existing algorithm. typing.Protocol expresses the contract with static checking and zero inheritance coupling — concrete classes do not subclass anything.

# src/myopt/problems/base.py (Protocol) and a concrete pair to show the fit
from __future__ import annotations

from typing import Protocol

import numpy as np


class Problem(Protocol):
    """Contract every problem satisfies; algorithms depend only on this."""

    name: str

    def evaluate(self, solution: np.ndarray) -> float:
        """Objective value of one solution (minimization)."""
        ...

    def random_solution(self, rng: np.random.Generator) -> np.ndarray:
        """Sample one feasible solution."""
        ...


class TSPProblem:
    """Symmetric TSP on a distance matrix; solutions are permutations of range(n)."""

    def __init__(self, name: str, dist: np.ndarray) -> None:
        self.name = name
        self.dist = dist
        self.n = dist.shape[0]

    def evaluate(self, solution: np.ndarray) -> float:
        """Tour length, closing the cycle, fully vectorized."""
        return float(self.dist[solution, np.roll(solution, -1)].sum())

    def random_solution(self, rng: np.random.Generator) -> np.ndarray:
        """Uniform random permutation."""
        return rng.permutation(self.n)


class RandomRestartSearch:
    """Baseline algorithm: best of n_samples random solutions."""

    def __init__(self, n_samples: int = 1000) -> None:
        self.n_samples = n_samples

    def solve(self, problem: Problem, rng: np.random.Generator) -> tuple[np.ndarray, float]:
        """Return (best_solution, best_objective) using only the Problem protocol."""
        best_sol = problem.random_solution(rng)
        best_obj = problem.evaluate(best_sol)
        for _ in range(self.n_samples - 1):
            sol = problem.random_solution(rng)
            obj = problem.evaluate(sol)
            if obj < best_obj:
                best_sol, best_obj = sol, obj
        return best_sol, best_obj


rng = np.random.default_rng(0)
pts = rng.random((8, 2))
dist = np.linalg.norm(pts[:, None, :] - pts[None, :, :], axis=2)
problem = TSPProblem("rand8", dist)
sol, obj = RandomRestartSearch(n_samples=500).solve(problem, np.random.default_rng(42))
print(sorted(sol.tolist()) == list(range(8)), round(obj, 3))
# Expected: "True <tour length>" — a valid permutation and a length near 2.5-3.0 for 8 unit-square points

Pitfall: fat interfaces. Adding neighbors(), crossover(), plot(), and to_mip() to the Problem protocol forces every problem to implement methods most algorithms never call, and soon a "problem" import drags in matplotlib and gurobipy. Keep the shared protocol to what every algorithm needs; operator-specific capabilities (neighborhood moves, decoders) belong to small extra protocols required only by the algorithms that use them.

Reproducibility Patterns

Pattern: seed discipline with SeedSequence spawning

"Seeds everywhere" fails when implemented as np.random.seed(42) once at program start: any change in execution order — a new log line that draws a number, a reordered loop — shifts the global stream and changes every later result. The discipline: one Generator per run, created from a per-run seed, passed explicitly to everything that draws randomness. Derive per-run seeds from one campaign seed with SeedSequence.spawn, which guarantees independent streams (better than base_seed + i, which correlates generators for some bit patterns).

# src/myopt/seeding.py
import numpy as np


def spawn_run_seeds(base_seed: int, n_runs: int) -> list[int]:
    """Derive independent, reproducible per-run seeds from one campaign seed."""
    seq = np.random.SeedSequence(base_seed)
    return [int(child.generate_state(1)[0]) for child in seq.spawn(n_runs)]


def rng_for(run_seed: int) -> np.random.Generator:
    """The single construction point for all randomness in a run."""
    return np.random.default_rng(run_seed)


seeds = spawn_run_seeds(base_seed=2026, n_runs=4)
print(len(set(seeds)) == 4, spawn_run_seeds(2026, 4) == seeds)
a = rng_for(seeds[0]).integers(0, 100, size=3)
b = rng_for(seeds[0]).integers(0, 100, size=3)
print(np.array_equal(a, b))
# Expected: "True True" then "True" — distinct seeds, stable derivation, reproducible streams

Pitfall: hidden randomness outside the passed generator. random.shuffle, set/dict iteration under PYTHONHASHSEED, legacy np.random.* global state, sklearn random_state defaults, and MIP solvers (Gurobi Seed, Threads) all draw from streams your Generator does not control. Grep for import random and np.random. outside seeding.py; pass rng into every constructor; set and record solver seeds. The determinism test catches regressions.

Pattern: provenance capture in every record

A result record that cannot answer "which code produced you" is unusable for a paper revision eight months later. Capture the git commit, Python and key package versions, platform, and a UTC timestamp at campaign start, and merge that dict into every record. The subprocess and package lookups are real external boundaries, so the fallbacks below are legitimate.

# src/myopt/provenance.py
import platform
import subprocess
import sys
from datetime import datetime, timezone
from importlib import metadata


def git_commit() -> str:
    """Short commit hash of the repository, '<hash>-dirty' if uncommitted changes exist."""
    try:
        head = subprocess.run(["git", "rev-parse", "--short", "HEAD"],
                              capture_output=True, text=True, check=True).stdout.strip()
        status = subprocess.run(["git", "status", "--porcelain"],
                                capture_output=True, text=True, check=True).stdout.strip()
        return f"{head}-dirty" if status else head
    except (OSError, subprocess.CalledProcessError):
        return "nogit"


def provenance(packages: tuple[str, ...] = ("numpy", "pandas", "gurobipy")) -> dict[str, str]:
    """Environment snapshot merged into every result record."""
    versions: dict[str, str] = {}
    for pkg in packages:
        try:
            versions[f"pkg_{pkg}"] = metadata.version(pkg)
        except metadata.PackageNotFoundError:
            versions[f"pkg_{pkg}"] = "absent"
    return {
        "git_commit": git_commit(),
        "python": sys.version.split()[0],
        "platform": platform.platform(),
        "timestamp_utc": datetime.now(timezone.utc).isoformat(timespec="seconds"),
        **versions,
    }


snap = provenance(packages=("numpy",))
print(sorted(snap))
# Expected: ['git_commit', 'pkg_numpy', 'platform', 'python', 'timestamp_utc'] with real values

Pitfall: running paper experiments from a dirty working tree. The commit hash is worthless if uncommitted edits changed the algorithm. The -dirty suffix above makes this visible; stricter projects refuse to start a campaign when the tree is dirty unless the config sets an explicit allow_dirty: true for debugging. Pair this with the tagging workflow in git-for-research-code so each paper table maps to a tag.

Result Writing and Run Orchestration

Pattern: atomic result files

A run killed mid-write (Ctrl-C, cluster preemption, out-of-memory) must not leave a truncated JSON file: half-written files poison aggregation and, worse, make skip-completed logic treat a broken run as done. The POSIX-and-Windows-safe recipe: write to a temporary file in the same directory, flush and fsync, then os.replace onto the final name — the rename is atomic on one filesystem, so readers see either the old state or the complete new file, never a fragment.

# src/myopt/io/results.py
import json
import os
import tempfile
from pathlib import Path
from typing import Any


def write_json_atomic(record: dict[str, Any], path: Path) -> None:
    """Write a JSON record so that readers can never observe a partial file."""
    path.parent.mkdir(parents=True, exist_ok=True)
    fd, tmp_name = tempfile.mkstemp(dir=path.parent, suffix=".tmp")
    try:
        with os.fdopen(fd, "w", encoding="utf-8") as fh:
            json.dump(record, fh, indent=2)
            fh.flush()
            os.fsync(fh.fileno())
        os.replace(tmp_name, path)
    except BaseException:
        os.unlink(tmp_name)
        raise


record = {"run_id": "demo_s0_abc123def456", "objective": 42.0, "status": "ok"}
with tempfile.TemporaryDirectory() as tmp:
    target = Path(tmp) / "raw" / "demo_s0_abc123def456.json"
    write_json_atomic(record, target)
    print(json.loads(target.read_text(encoding="utf-8"))["objective"])
# Expected: 42.0; a crash before os.replace leaves only a .tmp file, never a partial record

Pitfall: appending many runs to one shared CSV/JSONL from parallel workers. Concurrent appends interleave bytes, and a crash mid-append truncates the tail. The robust pattern is one file per run named by run_id (writes never contend), with aggregation into a single table as a separate, rerunnable step (see pandas-experiment-management). If one append-only JSONL is genuinely required, serialize writers with filelock and write each line in a single call.

Pattern: the campaign runner script

The entry point ties the previous patterns together: load config, iterate instances × seeds, skip completed runs, isolate per-run failures as recorded error records, write each record atomically. Note what it does not contain: algorithm logic, hard-coded paths, global state.

# scripts/run_experiment.py
"""Run one experiment campaign. Usage: python scripts/run_experiment.py configs/base.yaml"""
from __future__ import annotations

import argparse
import time
import traceback
from pathlib import Path

import numpy as np

from myopt.config import ExperimentConfig, load_config
from myopt.io.instances import load_instance
from myopt.io.results import write_json_atomic
from myopt.provenance import provenance
from myopt.registry import ALGORITHMS
from myopt.run_id import run_id


def run_campaign(cfg: ExperimentConfig) -> None:
    """Loop over instances x seeds; one atomic JSON record per run; resumable."""
    results_dir = Path(cfg.results_dir) / cfg.experiment_name
    env = provenance()
    algo_dict = {"name": cfg.algorithm.name, "params": dict(cfg.algorithm.params)}
    for instance_name in cfg.instances:
        problem = load_instance(instance_name)
        for seed in cfg.seeds:
            rid = run_id(algo_dict, instance_name, seed)
            out_path = results_dir / f"{rid}.json"
            if out_path.exists():
                continue  # resumable: a completed run is identified by its file
            algorithm = ALGORITHMS.create(cfg.algorithm.name, **cfg.algorithm.params)
            rng = np.random.default_rng(seed)
            base = {"run_id": rid, "instance": instance_name, "seed": seed,
                    "algorithm": algo_dict, **env}
            t0 = time.perf_counter()
            try:
                solution, objective = algorithm.solve(problem, rng, cfg.time_limit_s)
                record = {**base, "status": "ok", "objective": float(objective),
                          "solution": np.asarray(solution).tolist(),
                          "runtime_s": time.perf_counter() - t0}
            except Exception:
                record = {**base, "status": "error",
                          "traceback": traceback.format_exc(),
                          "runtime_s": time.perf_counter() - t0}
            write_json_atomic(record, out_path)


def main() -> None:
    """Thin CLI wrapper around the library."""
    parser = argparse.ArgumentParser(description=__doc__)
    parser.add_argument("config", type=Path, help="path to a campaign YAML/JSON file")
    args = parser.parse_args()
    run_campaign(load_config(args.config))


if __name__ == "__main__":
    main()
# Expected: results/raw/<experiment_name>/<run_id>.json per run; rerunning skips finished runs

Pitfall: the broad except Exception is correct here and only here. At the per-run boundary of an unattended thousand-run campaign, recording the traceback and continuing is right — one bad instance must not kill 900 pending runs, and status == "error" rows surface in aggregation. Inside library code the same construct hides bugs. Keep exactly one such catch, at the run boundary, and make error records loud in the aggregate report.

Pattern: the light test tier

Research code does not need 90% coverage; it needs the handful of tests that catch the bugs that invalidate experiments: a config typo silently ignored, a missing registry import, a nondeterministic run, an objective function that disagrees with brute force on a tiny instance. These five tests run in about a second, so they actually get run before every campaign.

# tests/test_core.py
"""Fast test tier: every test guards a way an experiment can become invalid."""
import itertools

import numpy as np
import pytest

from myopt.config import load_config
from myopt.registry import ALGORITHMS, PROBLEMS
from myopt.seeding import spawn_run_seeds


def test_config_rejects_unknown_keys(tmp_path) -> None:
    """A typo in a YAML key must crash at load time, not be ignored."""
    bad = tmp_path / "bad.yaml"
    bad.write_text("experiment_name: x\ninstances: []\nseeds: [0]\n"
                   "time_limit_s: 1.0\nalgorithm: {name: ils}\nsedes: [1]\n",
                   encoding="utf-8")
    with pytest.raises(ValueError, match="unknown config keys"):
        load_config(bad)


def test_registry_contains_paper_algorithms() -> None:
    """Guards the empty-registry trap: imports in myopt/__init__.py fill these."""
    for name in ("ils", "tabu", "ga"):
        assert name in ALGORITHMS.names()


def test_same_seed_same_result() -> None:
    """Iteration-capped so the time limit never binds; wall clock must not matter."""
    problem = PROBLEMS.create("random_tsp", n=15, seed=3)
    algo = ALGORITHMS.create("ils", max_iters=200)
    obj_a = algo.solve(problem, np.random.default_rng(7), time_limit_s=60.0)[1]
    obj_b = algo.solve(problem, np.random.default_rng(7), time_limit_s=60.0)[1]
    assert obj_a == obj_b


def test_seed_spawning_is_stable() -> None:
    """Per-run seeds derived from the campaign seed must never change."""
    assert spawn_run_seeds(2026, 3) == spawn_run_seeds(2026, 3)


def test_tiny_instance_matches_brute_force() -> None:
    """On n=6 the heuristic must find the enumerated optimum."""
    problem = PROBLEMS.create("random_tsp", n=6, seed=0)
    algo = ALGORITHMS.create("ils", max_iters=500)
    obj = algo.solve(problem, np.random.default_rng(0), time_limit_s=60.0)[1]
    brute = min(problem.evaluate(np.array(p)) for p in itertools.permutations(range(6)))
    assert obj == pytest.approx(brute)
# Expected: `pytest -q tests/` reports 5 passed in about a second

Pitfall: time-based assertions. Any test whose pass/fail depends on wall clock (a real time limit binding, "finishes within 2 s") flakes on loaded machines and CI. Keep the default tier iteration-capped and tiny-instance only; mark anything slower with @pytest.mark.slow and run it deliberately. Deeper validation — independent feasibility checkers, known-optimum regression suites — belongs to solution-validation-testing.

Advanced Techniques

Config sweeps with dotted-path overrides

Parameter studies should be generated, not hand-edited: a base config plus a grid of dotted keys yields one resolved config per grid point, each hashing to its own run_id.

import copy
import itertools
from typing import Any


def expand_grid(base: dict[str, Any], grid: dict[str, list[Any]]) -> list[dict[str, Any]]:
    """One config dict per point of the Cartesian grid of dotted-key overrides."""
    def set_dotted(cfg: dict[str, Any], dotted: str, value: Any) -> None:
        parts = dotted.split(".")
        node = cfg
        for part in parts[:-1]:
            node = node.setdefault(part, {})
        node[parts[-1]] = value

    keys = list(grid)
    configs: list[dict[str, Any]] = []
    for combo in itertools.product(*(grid[k] for k in keys)):
        cfg = copy.deepcopy(base)
        for key, value in zip(keys, combo):
            set_dotted(cfg, key, value)
        configs.append(cfg)
    return configs


base = {"algorithm": {"name": "ils", "params": {"kick_strength": 3}}, "time_limit_s": 10.0}
grid = {"algorithm.params.kick_strength": [2, 4, 8], "time_limit_s": [10.0, 60.0]}
sweep = expand_grid(base, grid)
print(len(sweep), sweep[0]["algorithm"]["params"]["kick_strength"], sweep[-1]["time_limit_s"])
# Expected: "6 2 60.0" — the full 3x2 grid with nested overrides applied

Grids explode combinatorially; for more than ~3 swept parameters, replace grid search with a tuning study (optuna-hyperparameter-tuning) and keep expand_grid for the final 2-3 sensitivity sweeps reported in the paper.

Determinism beyond the seed

A recorded seed is necessary, not sufficient. Remaining nondeterminism sources, by frequency: solver threading (set Gurobi Threads=1 and Seed explicitly for paper runs; record both), BLAS threading in numpy reductions (OMP_NUM_THREADS=1 for bit-exact reruns), PYTHONHASHSEED changing set/dict-of-object iteration order (sort before iterating), and wall-clock termination (iterations reached in 10 s vary across machines — record iterations performed and prefer evaluation budgets when comparing algorithms). State in the README which level the project guarantees: statistical reproducibility (same distribution) is honest and often sufficient; bit-exact reproducibility costs the thread restrictions above.

Packaging with pyproject.toml and editable installs

sys.path.append("..") at the top of scripts breaks the moment a script moves or runs from another directory. A minimal package definition fixes imports everywhere — scripts, tests, notebooks — with one pip install -e .:

[build-system]
requires = ["setuptools>=68"]
build-backend = "setuptools.build_meta"

[project]
name = "myopt"
version = "0.1.0"
requires-python = ">=3.10"
dependencies = ["numpy>=1.26", "pyyaml>=6.0", "pandas>=2.2"]

[project.optional-dependencies]
dev = ["pytest>=8.0"]

[tool.setuptools.packages.find]
where = ["src"]

The src/ layout (importable only after installation) makes tests exercise the installed package, so an import that "worked from the repo root" fails before a cluster run does. Pin exact versions in a lock file at submission time and record it next to the tagged results.

Immutable raw results, regenerable derivatives

Treat results/raw/ as append-only: no script modifies or deletes a raw record; renaming an experiment means a new experiment_name, not editing files. Everything in results/tables/ and results/figures/ must be regenerable by rerunning aggregate_results.py and make_figures.py. A plotting bug then never forces rerunning a week of solves, and a reviewer request ("medians instead of means") is a 30-second aggregation rerun. Archive published raw records outside the working tree, keyed by the paper's git tag.

Run manifests for large campaigns

Beyond roughly 10^4 runs, globbing JSON files to find pending work becomes slow and opaque. Generate the run list up front — scripts/plan_campaign.py expands configs × instances × seeds into a manifest parquet with one row per run_id and a status column — then workers claim rows and the aggregator joins results against the manifest. One pandas query then answers "what is missing, what failed, what is running" instead of directory archaeology.

Practical Challenges

The logic lives in notebooks and every experiment is a manual cell sequence. Migrate in one direction only: move functions (not cells) into src/myopt/ modules, import them back into the notebook, and re-verify outputs after each move. Nothing a result depends on may exist only in a notebook; notebooks stay useful for exploration and figure prototyping, calling the same library and reading the same results/raw/ records as the scripts.

A re-run silently overwrites last month's results. Content-addressed run_ids prevent collisions between different experiments, but rerunning the same config targets the same files. Make raw records write-once: the runner skips existing files, and an explicit --force-rerun flag that first moves old records to results/raw/_superseded/<date>/ is the only path to replacement — never destroy the evidence a submitted table was built on.

Results exist but nobody knows which code produced them. This is fatal at revision time. Provenance in every record (commit hash, dirty flag, package versions) plus the tag-per-table workflow of git-for-research-code solves it forward; for orphaned legacy results, the only honest fix is rerunning them under the new regime — treat unprovenanced numbers as lost.

np.random.seed deep inside a module makes results depend on call order. Symptom: results change when an unrelated feature is added, or when two algorithms run in one process in a different order. Grep for np.random.seed, np.random.<draw>, and bare import random; replace every hit with an explicit Generator parameter threaded from the runner. Add a two-algorithm variant of the determinism test if this bites once.

Defaults in code disagree with the config file. A config omits tenure, a refactor silently changed the constructor default from 8 to 12, and three weeks of "tenure=8" results are actually mixed. Record resolved parameters (read back from the constructed object) in every record, and let aggregation group by recorded parameters, never by what the config claims.

Absolute paths break on the cluster. C:/Users/.../instances/ or /home/alice/... in code or configs guarantees failure on the second machine. Config paths are relative to the repository root; the runner resolves them against Path(__file__).resolve().parents[1] or an explicit --data-root argument, and nothing else is machine-specific.

Run 900 of 1000 crashes and the campaign dies overnight. Two structural fixes, both in the runner pattern: per-run exception capture writes an error record and continues, and skip-completed logic makes a restart cost nothing. For cluster array jobs, the manifest pattern lets each task claim disjoint rows so a single failed task is re-submittable in isolation.

Two runs share mutable state through a cached instance object. An algorithm "repairs" the distance matrix in place or appends to problem.history, and every later run on that instance sees modified data. Rules: problems are immutable after construction (arr.flags.writeable = False makes violations crash loudly), algorithms own their mutable state, and the runner constructs a fresh algorithm object per run — reuse across runs saves nothing.

The registry is empty in the cluster job but fine on the laptop. Locally the notebook imported myopt.algorithms.ils at some point; the cluster entry point did not. The fix is the explicit import block in myopt/__init__.py plus test_registry_contains_paper_algorithms, which turns a KeyError at hour three into a red test before submission.

Tests are slow, so they stop being run. A suite that solves real instances takes minutes and gets skipped exactly when it matters. Enforce the two-tier split: the default tier (config, registry, determinism, tiny brute-force) stays under ~2 s; everything touching real instances or real time limits is @pytest.mark.slow. Run the fast tier before every campaign launch.

Tools & Libraries

Library / tool	When to use	Note
dataclasses (stdlib)	Config schemas, result records	Frozen dataclasses + a strict builder cover most projects; zero dependencies
pydantic	Configs needing coercion/validation	Heavier; worth it when configs come from many hands or web forms
PyYAML (safe_load)	Human-edited campaign configs	Always safe_load; YAML 1.1 quirk: unquoted no/yes parse as booleans
json (stdlib)	Machine-generated configs, result records	Canonical dumps (sorted keys) for hashing; JSONL for traces
argparse (stdlib)	Script CLIs	Sufficient for thin entry points; resist framework creep
hydra / omegaconf	Many-layered config composition	Powerful overrides and sweeps; adds magic — adopt only when plain YAML hurts
pathlib (stdlib)	All filesystem paths	Cross-platform; never concatenate path strings
pytest	The light test tier	tmp_path fixture, pytest.approx, markers for the slow tier
filelock	Shared-file writes from parallel workers	Only needed if the one-file-per-run pattern is abandoned
pandas + pyarrow	Aggregating raw records into tables	Parquet for big traces; see pandas-experiment-management
pip + venv / uv	Environment per project	Export a lock file at submission time; record it with the tag

Output Format

A structured-project deliverable consists of checklists and templates, not prose. Provide:

New-project checklist:

• pyproject.toml with src/ layout; pip install -e .[dev] succeeds in a fresh venv
• Directory skeleton: configs/, data/instances/, results/{raw,tables,figures}/, scripts/, src/, tests/
• .gitignore covers results/, solver logs, *.tmp, venv, __pycache__
• Registry for algorithms and problems; explicit imports in myopt/__init__.py
• seeding.py is the only module that constructs generators; no np.random.seed anywhere
• write_json_atomic is the only way results reach disk
• Fast test tier passes: config rejection, registry contents, determinism, tiny brute-force
• README: install, run one campaign, run tests — three commands, copy-pasteable

Campaign config template (configs/base.yaml):

experiment_name: ils_baseline_v1     # becomes the results subdirectory
instances: [rand50_a, rand50_b, rand100_a]
algorithm:
  name: ils                          # must be a registered name
  params: {kick_strength: 4, max_iters: 20000}
seeds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9]
time_limit_s: 60.0
results_dir: results/raw

Per-run record field checklist (every record carries all of these):

• run_id, instance, seed, algorithm (name + resolved params)
• status (ok/error), objective, solution, runtime_s (+ iterations performed)
• Provenance: git_commit (with dirty flag), python, platform, package versions, timestamp_utc

Pre-submission reproducibility checklist:

• Working tree clean; campaign commit tagged (e.g. paper-table3-v1)
• Solver Seed/Threads recorded; lock file exported next to the tag
• Error-status records counted and reported, not silently dropped
• aggregate_results.py + make_figures.py regenerate every table and figure from results/raw/ alone
• Raw records of published numbers archived outside the working tree

Questions to Ask

• How many algorithms, problems, instances, and seeds will be combined — what is the total run count?
• Will any of these results appear in a paper or thesis, and which tables/figures are needed?
• What machines run this — laptop only, or also a cluster or a colleague's environment?
• How long is one run, and can a thousand-run campaign survive a crash at run 900?
• Where does randomness enter (moves, instance generation, solver), and is each source seeded and recorded?
• Is there existing code to migrate — notebooks, a flat script, or nothing yet?
• Do configs need composition and sweeps, or is one YAML per campaign enough?
• What result volume is expected — scalar records, full solutions, or per-iteration traces?

Related Skills

• pandas-experiment-management — when the per-run JSON records must become tidy result tables, aggregations over instances and seeds, and pivot tables for the paper.
• git-for-research-code — when commits, tags, and .gitignore rules must link every result table to the exact code version that produced it.
• solution-validation-testing — when the light test tier should grow into independent feasibility checkers, objective recomputation, and known-optimum regression tests.
• instance-generation-and-benchmarks — when data/instances/ needs standard benchmark parsers or seeded synthetic generators with controlled hardness.
• optuna-hyperparameter-tuning — when hand-edited parameter sweeps should become a proper tuning study fed by the same config and seeding system.