Ml setup

ML Environment Setup

Setup development environment with modern Python tooling (uv/pixi), install dependencies, and configure development tools.

Process

1. Detect Project State

First, check what's already configured:

# Check for existing package managers
ls pyproject.toml 2>/dev/null && echo "Found pyproject.toml (uv/pip)"
ls pixi.toml 2>/dev/null && echo "Found pixi.toml (pixi)"
ls requirements.txt 2>/dev/null && echo "Found requirements.txt (pip)"

2. Ask User Preferences

If no package manager is configured, ask the user:

Package Manager Choice:

• uv (recommended for pure Python projects)

  • Fast dependency resolution
  • Compatible with pip/PyPI
  • Good for projects without CUDA requirements

• pixi (recommended for ML projects with GPU)

  • Conda-based, handles CUDA/cuDNN automatically
  • Better for complex ML dependencies (PyTorch, TensorFlow)
  • Cross-platform reproducibility

• pip (traditional, not recommended for new projects)

  • Slower than uv
  • Manual CUDA setup required

3. Install Package Manager (if needed)

For uv:

# Install uv
curl -LsSf https://astral.sh/uv/install.sh | sh

# Verify installation
uv --version

For pixi:

# Install pixi
curl -fsSL https://pixi.sh/install.sh | sh

# Verify installation
pixi --version

4. Initialize Project Dependencies

With uv:

# Initialize pyproject.toml if not exists
uv init --name ml-project

# Add ML dependencies
uv add torch pytorch-lightning hydra-core
uv add --dev pytest pytest-cov ruff mypy

# Create virtual environment and install
uv sync

With pixi:

# Initialize pixi.toml if not exists
pixi init

# Add ML dependencies with CUDA
pixi add pytorch pytorch-cuda=12.1 pytorch-lightning hydra-core
pixi add --feature dev pytest ruff mypy

# Install environment
pixi install

5. Configure Development Tools

Setup ruff (linting and formatting):

# Create ruff.toml if not exists
cat > ruff.toml << 'EOF'
line-length = 100
target-version = "py310"

[lint]
select = ["E", "F", "I", "N", "UP", "ANN", "B", "LOG", "G"]
ignore = ["ANN101", "ANN102"]

[lint.per-file-ignores]
"__init__.py" = ["F401"]
"tests/*" = ["ANN", "S101"]
EOF

Setup mypy (type checking):

# Add mypy config to pyproject.toml
cat >> pyproject.toml << 'EOF'

[tool.mypy]
python_version = "3.10"
strict = true
ignore_missing_imports = true
EOF

Setup pytest:

# Add pytest config to pyproject.toml
cat >> pyproject.toml << 'EOF'

[tool.pytest.ini_options]
testpaths = ["tests"]
python_files = ["test_*.py"]
addopts = ["--cov=src", "--cov-report=html"]
EOF

6. Setup Pre-commit Hooks (Optional but Recommended)

# Install pre-commit
uv add --dev pre-commit  # or: pixi add --feature dev pre-commit

# Create .pre-commit-config.yaml
cat > .pre-commit-config.yaml << 'EOF'
repos:
  - repo: https://github.com/astral-sh/ruff-pre-commit
    rev: v0.8.4
    hooks:
      - id: ruff
        args: [--fix]
      - id: ruff-format

  - repo: https://github.com/pre-commit/pre-commit-hooks
    rev: v5.0.0
    hooks:
      - id: trailing-whitespace
      - id: end-of-file-fixer
      - id: check-yaml
      - id: check-added-large-files
EOF

# Install hooks
pre-commit install

7. Create Project Structure (if new project)

# Create standard ML project directories
mkdir -p src/{models,data,utils}
mkdir -p tests
mkdir -p configs/{model,data,trainer,logger,experiment}
mkdir -p notebooks
mkdir -p scripts

# Create __init__.py files
touch src/__init__.py
touch src/models/__init__.py
touch src/data/__init__.py
touch src/utils/__init__.py
touch tests/__init__.py

8. Validation

Run validation checks to ensure everything is setup correctly:

# Check package manager
if command -v uv &> /dev/null; then
    echo "✓ uv is installed"
    uv --version
elif command -v pixi &> /dev/null; then
    echo "✓ pixi is installed"
    pixi --version
fi

# Check ruff
uv run ruff check . || pixi run ruff check .
echo "✓ Ruff is configured"

# Check pytest
uv run pytest --collect-only || pixi run pytest --collect-only
echo "✓ Pytest is configured"

# Check Python version
python --version
echo "✓ Python environment is active"

9. Generate Documentation

Create README.md with setup instructions:

# ML Project

## Setup

### Prerequisites

- Python 3.10+
- [uv](https://astral.sh/uv) or [pixi](https://pixi.sh)

### Installation

**With uv:**

\`\`\`bash
uv sync
\`\`\`

**With pixi:**

\`\`\`bash
pixi install
\`\`\`

### Development

\`\`\`bash
# Run tests
uv run pytest  # or: pixi run pytest

# Lint code
uv run ruff check .  # or: pixi run ruff check .

# Format code
uv run ruff format .  # or: pixi run ruff format .

# Type check
uv run mypy src/  # or: pixi run mypy src/
\`\`\`

### Training

\`\`\`bash
# Run training
uv run python src/train.py  # or: pixi run python src/train.py
\`\`\`

Environment-Specific Notes

CUDA/GPU Setup

With pixi (automatic):

pixi add pytorch pytorch-cuda=12.1
# CUDA toolkit and drivers are handled automatically

With uv (manual):

# Install PyTorch with CUDA
uv add torch --index-url https://download.pytorch.org/whl/cu121

# Verify CUDA
uv run python -c "import torch; print(torch.cuda.is_available())"

macOS (Apple Silicon)

# With uv
uv add torch  # MPS support included by default

# With pixi
pixi add pytorch

Windows

# With uv (use PowerShell)
irm https://astral.sh/uv/install.ps1 | iex
uv add torch --index-url https://download.pytorch.org/whl/cu121

# With pixi
iwr -useb https://pixi.sh/install.ps1 | iex
pixi add pytorch pytorch-cuda=12.1

Troubleshooting

Issue: "command not found: uv"

Solution: Add uv to PATH

export PATH="$HOME/.local/bin:$PATH"
echo 'export PATH="$HOME/.local/bin:$PATH"' >> ~/.bashrc

Issue: "CUDA not available"

Solution: Verify CUDA installation

nvidia-smi  # Check GPU
python -c "import torch; print(torch.cuda.is_available())"

If using uv, install correct CUDA version:

uv add torch --index-url https://download.pytorch.org/whl/cu121

Issue: "Permission denied" during installation

Solution: Run without sudo (install to user directory)

# Don't use sudo with uv/pixi installers
curl -LsSf https://astral.sh/uv/install.sh | sh

Success Criteria

• Package manager installed (uv or pixi)
• Development dependencies installed
• Ruff, mypy, pytest configured
• Pre-commit hooks setup (optional)
• Project structure created
• All validation checks pass
• README.md generated

Your ML development environment is ready!