Python engineering

Modern Python Engineering Standards

Apply these standards whenever scaffolding, advising on, or generating Python code and configuration.

---

Core Philosophy

1. Ecosystem-First: Before writing custom logic, evaluate standardized solutions. Custom scripts are liabilities; community tools are assets.
2. Fail Fast: Catch errors at startup (typed config) and at commit time (pre-commit hooks), not in production.
3. src layout is mandatory — never flat layout for production projects.
4. Single quotes for code literals; double quotes for docstrings.
5. Never suggest pip, poetry, or virtualenv — always uv.
6. Astral-first toolchain: prefer uv + ruff + ty for the unified, high-performance developer experience.
7. Protocol-first typing: prefer typing.Protocol (structural subtyping) over ABCs for interfaces. Reserve ABCs for shared implementation only.
8. Async for I/O-bound services: reach for async/await on web services and I/O-bound code — not CPU-bound work or one-shot scripts. Use asyncio.TaskGroup over asyncio.gather().
9. Observable when it runs as a service: for anything long-running, logging alone is insufficient — instrument with OpenTelemetry traces, metrics, and correlated structured logs. Right-size it; a CLI or one-off script doesn't need distributed tracing.
10. Secure the supply chain: pin dependencies with hashes, audit for CVEs in CI, use Trusted Publishers for PyPI releases.

---

Modifying existing code (the edit lane)

Most work in an existing project is an edit, not a scaffold. When the task is to change a few files in a project that already has its layout, tooling, and CI, the scaffolding, Docker, observability, and CI sections below are not the relevant rules — skip them. Match what the surrounding code already does, and apply only the rules that govern the lines you touch (for a micro-edit — a one-line docstring or string change — even these are mostly moot: match the touched line's local form, run the project's gate, and stop):

• Match the local convention first — project config governs. Read 2–3 nearby files before editing. The project's existing patterns — even where they differ from this skill's greenfield defaults — are the contract for an edit; a one-file modernization that diverges from the rest of the module is noise, not improvement. Where the project states its own conventions (AGENTS.md, CLAUDE.md, ruff.toml, a style guide), those govern and this skill's defaults are the fallback — defer, don't override.
• Protocol-first typing for new interfaces. A new seam introduced inside existing code prefers typing.Protocol (structural) over an ABC, unless the surrounding code already commits to ABCs. See the Typing Philosophy section.
• @override semantics. When overriding a real base-class method, annotate it @override (PEP 698); do not add @override to a class that only structurally satisfies a Protocol — see the caveat in Typing Philosophy.
• Import hygiene. Keep imports at module top, grouped stdlib / third-party / first-party (ruff's isort handles ordering). When a format-on-save or autofix hook strips unused imports, add an import in the same edit that first references it — an "import now, use later" split loses the import to the hook between edits.
• Quoting and docstrings. Single quotes for code literals, double quotes for docstrings (the project's ruff config enforces this; an edit that fights it just gets reformatted).
• Don't widen the scope. An edit's diff is its scope. Adjacent cleanup, renames, and "while I'm here" refactors belong in a separate change — the same scope discipline data-engineering-discipline applies to a migration.

If the task is actually to modernize an inherited project's tooling (assess the current setup and bring it to standard), that is the broader scope the rest of this skill and scripts/doctor.py cover — run the doctor, then work through its findings.

---

The Canonical Stack

Layer	Tool	Notes
Package manager	uv	Replaces pip + virtualenv + poetry + pyenv
Build backend	uv_build	Default from uv init, PEP 621-compliant
Lint + Format	ruff	Single-quote enforced in both linter+fmt
Type checking	ty (primary)	Astral's Rust-based checker (beta, fast)
	mypy (stable alt)	Use if ty coverage is insufficient
Config mgmt	pydantic-settings	Typed env vars, SecretStr for secrets
Logging	structlog	JSON in prod, pretty in dev
Observability	opentelemetry	Traces + metrics; correlate with structlog
Testing	pytest	Unit (pure) + integration (I/O)
Property testing	hypothesis	For data processing, serialization, algos
Security audit	pip-audit	CVE scanning against OSV database in CI
CLI (if needed)	typer	Type-hint-driven argument parsing
Web (if needed)	fastapi	Interface layer only — keep logic in core/
HTTP client	httpx	Sync + async, HTTP/2; default for new code

A note on type checkers

ty (by Astral, the ruff/uv team) is the forward-looking choice: much faster than mypy (Astral reports 10-100× on large codebases), built-in language server, first-class intersection types, and advanced reachability analysis. It is currently in beta (0.0.x) and evolving rapidly toward a 1.0 release.

mypy remains the battle-tested, stable option for projects that need full typing ecosystem coverage today.

Recommendation: Use ty for new projects and active development (fast feedback loop). Keep mypy in CI as a secondary check if your project relies on mypy plugins (e.g., django-stubs, sqlalchemy-stubs). For VS Code / Cursor users, install the ty extension for the language server.

---

Typing Philosophy

Prefer Protocol over ABCs for interface definitions. Protocols enable structural subtyping (static duck typing) — any class that implements the required methods satisfies the Protocol without inheriting from it. This aligns with Python's duck-typing nature and keeps classes decoupled.

from typing import Protocol

class Repository(Protocol):
    """Any class with save() and get() satisfies this contract."""
    def save(self, entity: dict) -> str: ...
    def get(self, entity_id: str) -> dict | None: ...

# No inheritance needed — just implement the methods
class PostgresRepository:
    def save(self, entity: dict) -> str: ...
    def get(self, entity_id: str) -> dict | None: ...

Reserve ABCs only when you need: runtime isinstance() enforcement, shared method implementations via inheritance, or @abstractmethod guarantees.

Python 3.14 typing improvements

PEP 649 (deferred annotations): On Python 3.14+, annotations are lazily evaluated. Forward references work natively — no need for from __future__ import annotations. Drop that import on new 3.14+ projects.

Other modern typing patterns: use Self for fluent APIs, @override for explicit method overriding, ParamSpec for typed decorators, and TypeIs / TypeGuard for type narrowing in guards.

@override caveat (PEP 698). @override requires an actual base-class method to override. On a plain structural class that satisfies a Protocol without subclassing it — the Protocol-first default above — do not add @override: there is no base method, so the type checker flags it as an error. Structural conformance and @override are mutually exclusive; reach for @override only inside a real inheritance chain (an ABC subclass, or a class that explicitly subclasses its base).

→ Full architecture rationale: Read references/ecosystem_rationale.md

---

Project Layout Decision

Ask (or infer) whether the project is a Library/Data tool or an Application.

• Library layout → domain-centric modules exposed directly (src/mylib/datatools/, src/mylib/core/)
• Application layout → decouple logic from entry points (src/myapp/core/ + src/myapp/interface/)
• Tests → a top-level tests/ tree, never inside src/. Colocated src/.../tests/ modules ship inside the built wheel and pollute the installed package; for per-unit suites (datajobs, plugins) mirror the package under tests/<area>/<name>/. (doctor.py flags any test_*.py found under src/.)

The application layout follows the functional core / imperative shell pattern: core/ contains pure business logic (no I/O, fully testable), while interface/ contains thin adapters (FastAPI routes, CLI commands, database clients). Connect them via Protocol interfaces and constructor injection.

→ Full directory trees and the authoritative pyproject.toml template: Read references/project_templates.md

---

Scaffolding Protocol

When generating a project for a user:

1. Resolve names from the project name they provide:

   • PyPI: kebab-case (e.g., my-cool-tool)
   • Package: snake_case (e.g., src/my_cool_tool/)
   • Classes: PascalCase

2. Mandatory substitutions in pyproject.toml:

   • [project] name
   • [tool.ruff.lint.isort] known-first-party = ["actual_name"]

3. Strip irrelevant optional deps: if user doesn't mention data processing, remove datatools extras; if no CLI, remove typer; if no web, remove fastapi.

4. Initialization command: always uv init --lib <project-name> to get the src layout with uv_build from the start.

---

Dependency Management (PEP 735)

Use [dependency-groups] (PEP 735) for development dependencies — not [project.optional-dependencies], which is for end-user feature extras (pip install mylib[cli]). Dev deps are local-only and must not ship with the package. Group into lint / test / security, plus a dev group that includes them.

→ The authoritative [dependency-groups] block: Read references/project_templates.md

---

Ruff Single-Quote Enforcement

Critical: both the formatter and linter must be aligned or they fight.

[tool.ruff.format]
quote-style = "single"

[tool.ruff.lint.flake8-quotes]
inline-quotes = "single"
docstring-quotes = "double"
multiline-quotes = "double"

Ruff 0.8+ deltas: the TCH category is now TC (use "TC" in select); ANN101 / ANN102 were removed (don't list them in ignore); block suppression via # ruff: disable[RULE] / # ruff: enable[RULE] avoids per-line # noqa.

---

Configuration Management Pattern

Never read os.getenv() directly. Use a typed Settings(BaseSettings) from pydantic-settings, with SecretStr for secrets (masked in logs). Keep .env gitignored and commit .env.template to document required keys.

from pydantic import SecretStr
from pydantic_settings import BaseSettings, SettingsConfigDict


class Settings(BaseSettings):
    API_KEY: SecretStr
    model_config = SettingsConfigDict(env_file='.env', extra='ignore')


settings = Settings()

→ Why typed config (fail-fast at startup, validation): Read references/ecosystem_rationale.md

---

Observability

Observability has three pillars: structured logs, distributed traces, and metrics. Use structlog for logging and opentelemetry for traces/metrics. Correlate them by injecting trace IDs into log entries.

→ Full structlog + OpenTelemetry setup, stdlib bridge, FastAPI middleware, and pytest integration: Read references/observability.md

The minimum-viable configure_logging() (JSON in prod, pretty in dev) and the full trace-correlated setup both live in references/observability.md — call it once at application startup (CLI entry point or FastAPI lifespan).

---

Pre-commit Hooks

scripts/scaffold.py writes the canonical .pre-commit-config.yaml (trailing-whitespace, end-of-file-fixer, check-yaml, check-added-large-files, plus ruff + ruff-format at the revs pinned in stack.toml). Install with uv run pre-commit install. Enforcement is a ladder, not an assumption: on a harness with act-time hooks (Claude Code) this plugin formats each edit and blocks pip/poetry in uv projects as it happens; elsewhere the same rules hold at commit time via this pre-commit config (plus the exported check-uv-hygiene hook), and as advisory text where neither exists.

Note on ty in pre-commit: ty has no official pre-commit hook yet. Run it via uv run ty check src in CI or as a local script.

---

Key uv Commands

uv init --lib my-project        # Scaffold with src layout + uv_build
uv add <package>                # Add runtime dependency
uv add --dev <package>          # Add dev dependency (PEP 735)
uv add --group lint <package>   # Add to specific dependency group
uv sync                         # Install all deps from lockfile
uv sync --frozen                # CI: install without updating lockfile
uv run pytest                   # Run tests in project venv
uv run ruff check src           # Lint
uv run ruff format src          # Format
uv run ty check src             # Type check (ty)
uv run mypy src                 # Type check (mypy, if used)
uv python install 3.14          # Install a specific Python version
uv python pin 3.14              # Pin project to Python 3.14

---

AI-Assistant Configuration

Modern Python projects include configuration files for AI coding assistants. These are committed to the repo and provide persistent project context.

• CLAUDE.md — Read by Claude Code at conversation start. Include build/test commands, architecture decisions, code style preferences. Keep under ~50 concise instructions. Use /init to auto-generate.
• .cursor/rules/*.mdc — Cursor rules with frontmatter-based glob matching (e.g., scope rules to app/routers/*.py).
• .github/copilot-instructions.md — Repo-level instructions for GitHub Copilot Chat and Code Review.

→ Full templates and patterns: Read references/ai_config.md

---

Supply-Chain Security

Add these to every CI pipeline:

uv run pip-audit                  # Scan for known CVEs (OSV database)
uv run ruff check src --select S  # Bandit-equivalent security linting

For package publishers: use Trusted Publishers on PyPI (OIDC-based, no long-lived API tokens) and Sigstore attestations (automatic with the canonical GitHub Action). Pin all dependencies with cryptographic hashes via uv lock.

→ Full security setup and CI integration: Read references/security.md

---

Reference Files

Exhaustive templates and patterns — read on demand:

• project_templates.md — pyproject.toml master config, both directory layouts, GitHub Actions CI.
• ecosystem_rationale.md — why each tool, Protocol vs ABC, functional core / imperative shell, DI.
• observability.md — structlog + OpenTelemetry setup, trace correlation, FastAPI middleware.
• testing_and_qa.md — pytest, Hypothesis, mutation/snapshot testing, testcontainers, unit/integration split.
• docker_patterns.md — multi-stage Dockerfile with uv, CI/CD Docker caching.
• security.md — Trusted Publishers, Sigstore, pip-audit, dependency pinning.
• ai_config.md — CLAUDE.md, Cursor rules, Copilot instructions.
• currency_review.md — quarterly review protocol for keeping the skill current.

---

Scripts, freshness, and keeping current

last-reviewed: 2026-06-04. Pinned versions live in stack.toml — the single source of truth this skill cites instead of repeating numbers in prose. When exact current versions matter, verify against PyPI / context7 at the point of use; a pinned floor can lag the latest release.

Bundled scripts (scripts/):

python scripts/scaffold.py my-cool-tool  # new project to standard (uv init + canonical pyproject + pre-commit)
python scripts/doctor.py [path]          # audit an existing project against the standard
python scripts/check_versions.py         # compare stack.toml pins to latest on PyPI (--json for CI)

Freshness loop (Tier 3). check_versions.py detects drift (exits non-zero when a pin is behind a newer minor/major); a monthly CI cron opens a drift issue; the /refresh-stack command reviews each changelog and proposes the stack.toml + guidance updates for you to approve — mechanical bumps applied on approval, guidance edits never auto-applied. See references/currency_review.md for the full review checklist.