Deployment

deployment

Production deployment pattern for containerised web applications using Docker Compose. Prioritises safety (automatic backups before destructive steps, rollback-ready state, mandatory screen sessions for remote runs) over raw speed. This is not a zero-downtime pattern — it's a stop/rebuild/start cycle with health verification. Brief outages (5–30 s) are expected on updates.

This skill covers:

• Branch strategy and release management
• Change-aware deploy scripts (migrations, dependencies, static assets)
• Database safety (pre-change backups, rollback procedures)
• Remote SSH deployment with mandatory screen sessions
• Let's Encrypt SSL via certbot + nginx
• Health checks and deployment verification
• Environment management and .env handling
• Makefile targets for common operations

When to use

TRIGGER when: preparing or executing a production deploy for a Docker Compose application; setting up the initial deploy.sh + backup_db.sh + verify_deployment.sh toolchain; wiring a CI/CD pipeline that invokes the deploy scripts; handling rollback after a failed deploy.

SKIP for: local dev-container startup (docker compose up), single-container toy apps, Kubernetes-based deploys (different contract — use Helm/ArgoCD), PaaS targets (Heroku/Fly/Railway — they own their own deploy verb and these scripts will conflict).

Pattern

Branch strategy

Separate production from development:

• Use a deployment (or production) branch for releases.
• Merge stable commits from main into deployment explicitly — no auto-fast-forward.
• Branch protection on deployment: require review, disallow force-push.
• A clear commit boundary on the release branch = a clean rollback target.

git checkout deployment
git merge <stable-sha>
git push origin deployment

Service architecture

Typical Docker Compose stack:

Service	Role
web	Application container (Django, Rails, Node, …)
nginx	Reverse proxy, SSL termination, static file serving
db	PostgreSQL / MySQL
cache	Redis / Memcached
worker	Background jobs (Celery / Sidekiq / BullMQ)
certbot	Let's Encrypt renewals
storage (opt.)	MinIO / S3-compatible object store

Principles:

• Services communicate via internal Docker network only (no public ports except nginx).
• Persistent data on named volumes.
• All configuration via environment variables (no baked-in secrets).
• Each service declares healthchecks and dependencies.

Deploy scripts

Canonical toolchain lives in scripts/ (or tools/) at the repo root:

• deploy.sh — auto-detecting wrapper (first-time vs update)
• deploy_first_time.sh — initial deploy + data seed
• deploy_update.sh — change-aware update
• backup_db.sh — database snapshot
• restore_db.sh — counterpart restore
• verify_deployment.sh — post-deploy health checks

Reference implementations live in this skill's scripts/ directory — copy them into your project and customise.

Auto-detecting wrapper:

#!/bin/bash
# scripts/deploy.sh
SERVICES_RUNNING=$(docker compose ps | grep -qE "Up|running" && echo true || echo false)

if [ "$SERVICES_RUNNING" = "true" ]; then
    exec ./scripts/deploy_update.sh "$@"
else
    exec ./scripts/deploy_first_time.sh "$@"
fi

Project-root detection (survives symlinks and arbitrary cwd):

if [ -n "$PROJECT_ROOT" ]; then
    cd "$PROJECT_ROOT"
elif git rev-parse --git-dir > /dev/null 2>&1; then
    cd "$(git rev-parse --show-toplevel)"
else
    echo "Warning: using current directory as project root"
fi

Non-interactive mode

Deploy scripts must support non-interactive invocation for CI and screen sessions:

DEPLOY_NON_INTERACTIVE=1 ./tools/deploy.sh
# or
./tools/deploy.sh --yes

Why: screen allocates a TTY, so [ -t 0 ] returns true inside the session. A prompt in the deploy script then blocks forever with no stdin attached. Explicit non-interactive mode forces safe defaults regardless of TTY presence. CI runners have the same problem.

Change detection

The update script decides what work is needed by diffing current and previous deploy:

PREVIOUS=$(git rev-parse HEAD@{1} 2>/dev/null || echo "")

if [ -n "$PREVIOUS" ]; then
    HAS_MIGRATIONS=$(git diff "$PREVIOUS" HEAD --name-only | grep -q "migrations/" && echo true || echo false)
    HAS_DEPENDENCIES=$(git diff "$PREVIOUS" HEAD --name-only | grep -qE "requirements.*\.txt|Dockerfile|package(-lock)?\.json|Gemfile(\.lock)?" && echo true || echo false)
    HAS_STATIC=$(git diff "$PREVIOUS" HEAD --name-only | grep -qE "\.(css|js|scss|png|jpg|svg)" && echo true || echo false)
fi

# Dependency rebuild may touch static assets the grep missed — force a static rebuild.
[ "$HAS_DEPENDENCIES" = "true" ] && HAS_STATIC=true

Two layers: file diff + config fingerprint for expensive ops

File-diff change detection is necessary but not sufficient when an expensive post-deploy operation (full corpus reindex, CDN purge, search-engine wipe-and-rebuild, vector-index rebuild, ML-model warm-up) depends on a subset of the files that flip the diff. The first layer answers "did anything in this area change?"; you need a second layer to answer "did the thing the expensive op actually depends on change?"

Pattern:

1. Identify the inputs the expensive op truly depends on — usually env vars and a small set of config knobs, not the source files of the subsystem implementing it.
2. Compute a deterministic fingerprint over those inputs at deploy time (sourced from .env already in scope).
3. Persist the fingerprint to a marker file (.deploy_<topic>_shape) after the op succeeds.
4. On next deploy, compare current vs. stored fingerprint. If unchanged → skip the expensive op. If changed → run it. If marker is missing → run it (treat first-time / lost-marker as "run", since silently skipping is much worse than running unnecessarily).
5. Provide a FORCE_<TOPIC>=1 env override for the operator-driven cases the gate can't predict (env-only fixes, manual recovery).

SHAPE_FILE="$PROJECT_ROOT/.deploy_embedding_shape"
compute_embedding_shape() {
    # Fingerprint the inputs the expensive op depends on — provider, model,
    # dimension, API base, index version. Empty values included verbatim so
    # unset → set transitions register as a change.
    printf '%s|%s|%s|%s|%s|%s\n' \
        "${EMBEDDING_PROVIDER:-}" "${EMBEDDING_DIMENSION:-}" \
        "${EMBEDDING_MODEL_NAME:-}" "${GEMINI_EMBEDDING_MODEL:-}" \
        "${EMBEDDING_API_BASE_URL:-}" "${INDEX_VERSION:-}"
}

CURRENT=$(compute_embedding_shape)
PREVIOUS=$([ -f "$SHAPE_FILE" ] && cat "$SHAPE_FILE" || echo "")

if [ "$FORCE_REINDEX" = "1" ] || [ -z "$PREVIOUS" ] || [ "$CURRENT" != "$PREVIOUS" ]; then
    # Gate the marker write on the op's exit status so a failed reindex
    # leaves the PREVIOUS marker intact — next deploy retries. Without
    # explicit gating (or `set -e`), `echo > $SHAPE_FILE` on the next
    # line would run unconditionally and silently mask the failure.
    if run_expensive_reindex; then
        echo "$CURRENT" > "$SHAPE_FILE"
    else
        echo "❌ reindex failed — marker NOT written; next deploy will retry." >&2
        exit 1
    fi
else
    echo "🔍 Shape unchanged — skipping reindex."
fi

Anti-pattern this teaches against: gating an expensive post-deploy op on raw dependency-file diff (grep -qE "<subsystem>/" diff). The subsystem changed (new endpoint, refactor, test added) but the thing the op depends on didn't. Running anyway burns whatever the op costs — for a vector-index rebuild against a pay-per-token embedding API, that can be a rate-limit cliff and a real bill. The gate moves the decision from "did any file in the subsystem change?" to "did the op's actual contract change?", which is a much smaller surface.

Failure-mode discipline: write the marker only after the op succeeds. The if run_expensive_reindex; then ... fi shape above does this explicitly — without it (run_expensive_reindex; echo "$CURRENT" > "$SHAPE_FILE"), a failed reindex would still write the marker, and the next deploy would skip the retry, silently masking the failure. If the op is non-atomic (multi-stage), wrap only the final stage in the if so partial-success doesn't poison the marker.

Backup strategy

Database backup is mandatory before any schema change:

if [ "$HAS_MIGRATIONS" = "true" ]; then
    ./scripts/backup_db.sh   # pre-migration snapshot
fi

docker compose exec -T web python manage.py migrate

./scripts/backup_db.sh       # post-migration snapshot (clean rollback point)

Backup naming convention:

data/db_backup_YYYYMMDD_HHMMSS_<commit-sha>.sql.tar.gz

The commit sha in the filename lets you pair any backup with the exact code that produced the schema — critical for restore-plus-revert.

Remote deployment — mandatory screen

Remote deploys must run inside a screen session so they survive SSH disconnects, network blips, and terminal closures.

Canonical form:

ssh [email protected] << 'EOF'
cd /opt/myapp
SESSION="myapp-deploy-$(date -u +%Y%m%d-%H%M%S)"
LOG="deploy-$(date -u +%Y%m%d-%H%M%S).log"
screen -dmS "$SESSION" bash -c \
  "DEPLOY_NON_INTERACTIVE=1 ./tools/deploy.sh 2>&1 | tee $LOG"
echo "Session: $SESSION | Log: $LOG"
sleep 3 && tail -n 50 "$LOG"
EOF

Full recipe book — naming conventions, monitoring, attaching/detaching, cleanup, long-rebuild handling, troubleshooting — is in references/SCREEN_SESSIONS.md.

Host-key verification: never use -o StrictHostKeyChecking=no in automated paths. A MITM can then silently substitute for the deploy target and steal the deploy key. Populate known_hosts in advance (ssh-keyscan -H host >> ~/.ssh/known_hosts), or pin the expected fingerprint as a secret and verify it before connecting.

Deployment session tracking

Record each production deploy in the repo for later forensics:

• Path: docs/deployment/sessions/{host}_yyyymmdd-hhmm.md (e.g. myapp_com_20260417-1415.md)
• One file per deploy run, covering the whole lifecycle — pre-rollout plan, execution record, post-deploy verification, rollback target. Create the file with status: PENDING before the run and flip to ✅ / ❌ at the end. Do not fork the plan into a separate ROLLOUT_PLAN_*.md — the session file is the single artefact and the PENDING status handles the pre-rollout phase. A plan-doc-plus-session-file split produces the two-narrative drift every release; the single file stays in sync by construction.
• Template: maintain docs/deployment/sessions/_template.md with PENDING / SUCCESSFUL / FAILED status slots; copy per run.

This is the paper trail humans need when something goes wrong three months later and they're reconstructing the release timeline.

Environment management

All configuration comes from environment variables loaded from .env:

set -a
source .env
set +a

When .env changes, force-recreate containers that consume it:

docker compose up -d --force-recreate web worker

.env is never committed. Provide .env.example with all keys and safe defaults. Rotate secrets on staff changes.

SSL certificates — Let's Encrypt

docker-compose.yml:

certbot:
  image: certbot/certbot
  command: >
    certonly --webroot --webroot-path=/var/www/certbot
    --email $$CERTBOT_EMAIL --agree-tos --no-eff-email -d $$DOMAIN
  volumes:
    - certbot-etc:/etc/letsencrypt
    - certbot-var:/var/lib/letsencrypt
    - ./nginx/www:/var/www/certbot

nginx vhost:

server {
    listen 443 ssl http2;
    server_name ${DOMAIN};
    ssl_certificate     /etc/letsencrypt/live/${DOMAIN}/fullchain.pem;
    ssl_certificate_key /etc/letsencrypt/live/${DOMAIN}/privkey.pem;
    # proxy configuration…
}

Renewal runs on a cron/timer or via docker compose run --rm certbot renew — do not couple renewal to the main deploy flow.

Health checks and verification

After every deploy, verify. HEALTH_URL below is a project-supplied variable that points at a cheap JSON health endpoint when one exists (/api/health, /healthz, etc.) and falls back to / (landing page 200) when the project doesn't expose a dedicated probe. Never hard-code /api/health into a project's verify script without confirming the endpoint actually exists — a 404 from the liveness probe masks real signal.

docker compose ps                                       # containers healthy
curl -fsI https://${DOMAIN}                             # external HTTPS reachable
curl -fsS https://${DOMAIN}${HEALTH_URL:-/}             # project health endpoint (JSON if available; else landing page)
docker compose exec -T web python manage.py check       # framework-level sanity

Wrap these in verify_deployment.sh with a retry loop — services routinely take 10–30 s after container start to accept traffic, and failing on the first curl produces false-alarm rollbacks.

Note what these four commands have in common: every one of them runs on the deploy host, against the tier that was just recreated. The next section is about the checks they structurally cannot make.

Verify from the side that would experience the failure

Some defects are live and manifest right now and every honest check still comes back green — because each check is run from the side where the fault cannot appear by construction. No better assertion fixes that; only a different vantage does. Name the party who would experience the failure and reproduce the observation from their position; where that is impossible, downgrade your result to evidence about the producer only.

Discriminator: if you moved the same check, unmodified, to the other side of the boundary, would it fail? If yes, the check's subject was right and only its position was wrong — this section. If it would still pass wherever you ran it, the defect is elsewhere: either the check interrogates a stand-in for the thing the system consumes, or the environment it ran in could not express the fault (see shell / EVIDENCE_SCRIPTS_AND_FALSE_CLEANS.md).

Three unoccupied vantages recur in deploy work.

1 — The far end of a network path. A probe run on the box it tests addresses one of that host's own addresses: the packet is delivered locally, never crosses the external path, and under the usual host-firewall default of a blanket accept on the loopback interface it simply connects. Such a probe proves the port is published and nothing more. It cannot tell "filtered" from "reachable", and it may pass for a reason unrelated to the control it names — a bind scope rather than a firewall rule — so when that unrelated reason later changes (a publish widening from one interface to all), the same fail-closed line starts connecting and fails the deploy for the opposite reason. Choosing the vantage is two decisions, not one:

• an assertion that something is reachable must be probed from an allow-listed consumer, as a named step with its own credentials and a delivery-path remedy in the failure message;
• an assertion that something is blocked must be probed from an origin that is not allow-listed for that path.

What decides which one you are actually running is the source address the prober selects for that destination, not which host you sit on — a multi-homed box picks a private source for a private destination and a public source for a public one, so the same machine sits inside the allow-list for one target and outside it for another. Keep the local probe, but label it the producer check. Then: grep the verify script for probe targets and check them against the host's own address list; re-derive every such assertion when the enforcement mechanism moves (bind-scoping → host firewall is a rewrite of the check, not a no-op); and sweep siblings, because the same silhouette usually sits in the neighbouring check on the box's other local address. A fail-closed assertion that has never once fired is a warning sign, not reassurance.

2 — The tier the deploy did not recreate. A deploy that recreates only the request-serving tier leaves workers, schedulers and CLI entrypoints on the previous release indefinitely, doing old work correctly — a skew health checks cannot see, because they probe the tier that was updated. The same asymmetry hits configuration: background tiers read values the request path never consults, so a malformed value can crash-loop the queue while the site serves perfectly. Neither failure emits an error at the web tier. The shape follows from the topology (request tier + background tier), not from any framework or orchestrator, and restates unchanged for containers, sidecar schedulers, systemd units and managed task services. So verify a deploy by asserting a fact from the background tier — the running worker's image or release identity, a value the scheduler reads back — not just the web tier's health endpoint; and for every config value, name which execution paths consume it. Expect pre-production not to reproduce this, often for an accidental reason nobody lists as a difference (a queue tier held down until a bundled database is seeded means nothing was ever running to go stale). Pairs with references/MONITORING.md § scheduler liveness — there the background tier is dead behind a green stack, here it is alive on last release.

3 — The runtime the change was not tested in. When one source tree is deployed to more than one runtime, a change to any file a second runtime also consumes is a production change to that runtime too, even when it is verified correct in the runtime that prompted it. The discriminator is consumption, not "repo-level-ness": a file no second runtime reads is not a fleet change. The propagation mechanism is the non-obvious half and is what makes it bite — a dependency manifest reaches an older host because deploys re-resolve dependencies (lockfile not committed); a framework config default bites because the untested runtime holds warm state (caches, existing directories) the tested one lacks; a shared template bites because its content hash triggers a fleet-wide regeneration sweep on the next deploy. The review question is therefore not "is this fix correct?" but "which runtimes does this file reach, by what mechanism does the change take effect there, and what state do they hold that the tested runtime does not?" Prefer scoping the change into an artefact only the intended runtime reads — a build stage, a per-instance env file, an override (laravel / CONTAINER_BUILD_PLATFORM.md is a worked example: pin the platform in the build stage, not in the shared manifest). Shipping it shared is legitimate, but only after proving it is a no-op on every other runtime and writing that proof into the commit body. This one is caught in review every time so far — the guardrail is what is holding, not luck.

Common failure of all three: choose detection primitives whose absence fails loudly. A CLI flag missing from an unpinned older version returns an empty set, reproducing the very false negative you were fixing.

Rollback procedures

⚠️ Human-operated only. Rollback involves destructive git operations (reset --hard, force-push) and potentially destructive database operations (restoring a backup over live data). Per RULE_git-safety, AI agents must not execute these steps. Agents prepare the commands, show them, and wait for the human to run them.

Code rollback (requires temporarily relaxed branch protection on deployment):

git checkout deployment
git reset --hard <last-good-sha>
git push origin deployment --force-with-lease   # human-initiated only

--force-with-lease refuses to overwrite remote changes the operator hasn't seen, which is the minimum safety net on a destructive push.

Redeploy from rolled-back state:

git pull origin deployment
./scripts/deploy.sh

Database rollback:

./scripts/restore_db.sh data/db_backup_YYYYMMDD_HHMMSS_<sha>.sql.tar.gz
docker compose restart web worker

Paired rollback (code and data together): restore the DB backup whose <sha> suffix matches the code you're reverting to. Mismatched pairs cause schema drift and the app will fail in surprising ways.

Makefile targets

.PHONY: deploy deploy-first deploy-update backup-db restore-db
.PHONY: start stop restart logs health-check

deploy:          ## Auto-detect and run appropriate deploy
	./tools/deploy.sh

deploy-first:    ## First-time deploy + seeding
	./tools/deploy_first_time.sh

deploy-update:   ## Update existing deploy (change-aware)
	./tools/deploy_update.sh

backup-db:       ## Snapshot database
	./tools/backup_db.sh

restore-db:      ## Restore from backup: make restore-db FILE=...
	@test -n "$(FILE)" || { echo "Usage: make restore-db FILE=backup.tar.gz"; exit 1; }
	./tools/restore_db.sh "$(FILE)"

start:           ## Start all services
	docker compose up -d

stop:            ## Stop all services
	docker compose down

restart:         ## Restart all services
	docker compose restart

logs:            ## Tail logs
	docker compose logs -f

health-check:    ## Run verification suite
	./tools/verify_deployment.sh

CI/CD integration

For automated pipelines (GitHub Actions, GitLab CI), see references/CICD.md. Short rules:

• CI wraps deploy.sh — it does not reimplement the logic.
• Production deploys are manual-trigger only — never auto-deploy on main push.
• Database backup step runs before the deploy step if change-detection sees a migrations/ diff.
• SSH host keys are pinned via known_hosts, not bypassed with StrictHostKeyChecking=no.
• Rollback is not auto-triggered on pipeline failure — human-in-the-loop only.

When NOT to use these patterns

• Zero-downtime required — this pattern restarts services; brief outages are expected. Use blue/green or rolling deploys via Kubernetes, Nomad, or ECS.
• Horizontal scaling across hosts — single-host Docker Compose doesn't scale past one node. Graduate to an orchestrator.
• Stateless apps without a DB — the backup/rollback machinery is overhead; a simpler docker compose pull && up -d suffices.
• PaaS deploys — Heroku/Fly/Railway own the deploy verb; these scripts will conflict with the platform's model.

Troubleshooting

Database connection fails on start

• docker compose ps db — is the container up and healthy?
• Env vars in the web container match the db container?
• Network reachable: docker compose exec web nc -zv db 5432
• DB logs: docker compose logs db

Migration fails mid-deploy

• Inspect the failing migration file for logic errors.
• Check prior migrations all applied cleanly: docker compose exec web python manage.py showmigrations.
• Verify the DB user has DDL permission.
• Restore the pre-migration backup and investigate in a separate environment — never debug migrations on live data.

SSL certificate errors

• docker compose logs certbot — renewal output.
• Domain DNS points here: dig +short $DOMAIN.
• Ports 80/443 open at the host firewall (UFW) and cloud security group.
• Rate-limited? certbot certificates shows expiry; Let's Encrypt enforces renewal limits.

Permission errors during deploy

• Deploy user has write on the project directory: ls -la /opt/myapp.
• SSH key perms: chmod 600 ~/.ssh/id_rsa.
• Migrations created by Docker as root? Apply the ownership-bypass pattern from the django skill (chown from inside the container using the host UID/GID).

Health check fails post-deploy

• Service logs: docker compose logs <service>.
• App reachable from inside the container network? docker compose exec web curl -f localhost:8000${HEALTH_URL:-/}.
• OOM? docker stats and dmesg | tail on the host.

.env changes don't take effect

Containers cache env vars at start. After editing .env:

docker compose up -d --force-recreate <service>

Plain restart does not reload the environment.

Why it's generic

Applies to any Docker Compose web application because the pattern is rooted in:

• Container orchestration primitives (services, volumes, networks, healthchecks)
• Database safety during schema change (snapshot + restore, same for every RDBMS)
• Infrastructure automation via shell (no framework dependency)
• Change detection via git diff (language-agnostic)

The deploy shape is identical for Django, Rails, Express, FastAPI, Phoenix — only the service names in docker-compose.yml and the test/migration commands differ.

Example use cases

Framework-specific notes below are illustrative, not prescriptive.

Django — web (Daphne ASGI) + db (Postgres) + redis + celery + certbot. Migration safety and static collection are the high-risk steps; both are covered by change detection. See references/DJANGO_DEPLOYMENT.md for specifics.

Rails — web (Puma) + db (Postgres) + redis + sidekiq + certbot. rails db:migrate replaces manage.py migrate; the asset pipeline runs on dependency change.

Node.js / Next.js — web (node process manager) + db + certbot. Build step (npm run build) runs on dependency change; static output served via nginx.

References

• scripts/ — reference implementations (deploy.sh, backup_db.sh, verify_deployment.sh, setup_server.sh, harden_server.sh)
• scripts/README.md — per-script usage and customisation
• references/SCREEN_SESSIONS.md — mandatory reading for remote deploys; full screen recipes, monitoring, cleanup
• references/CICD.md — GitHub Actions, GitLab CI, secrets, approval gates
• references/MONITORING.md — Prometheus, Grafana, ELK
• references/DARK_DEPLOY_KILL_SWITCH.md — dark → shadow → live ladder; why a kill switch's OFF position must be faithful per call site; shadow OBSERVES but does not PROTECT (time in shadow has a running cost its own logs quantify); the staging unit is the ORGANISATION, not the tenant; rollback stops future writes but cannot unwind past ones
• references/DJANGO_DEPLOYMENT.md — Django-specific optimizations (migrations, collectstatic, ASGI)
• references/HARDENING.md — SSH, UFW, fail2ban, unattended upgrades (run before first deploy); fail2ban behind a proxy — a filter that reads forwarded-IP log lines bans the end user while the proxy path stays open, and application-level 4xx must never count toward a flood; systemd unit-sandbox version gates (SystemCallFilter=@system-service needs ≥ 240 — older systemd resolves it silently and the loader dies status=127; gate drop-ins on the capability, never on virt-type)
• references/CONTAINER_DNS_NSS.md — getaddrinfo shadowing public DNS inside containers; anchor case: sync_domains silent drop on fresh Debian VPS when hostname matches domain
• references/SHELL_INSTALLERS.md — authoring + review patterns for install/install-*.sh and remote-sudo / forced-command wrappers; installer-specific catalog (chown, marker blocks, opt-out sweep, target-user resolution) distilled from review-loop cycles across PRs #55/#58/#59, plus patterns 16–20 from br-docs IDEA-029 (ssh -t invisible sudo prompt, explicit-PTY vs forced-command keys, nologin breaks command=), with the language-general entries stubbed down into the shell layer
• references/ROOTLESS_DOCKER.md — rootless Docker deploys: non-login deploy shells (screen … bash -c, cron, systemd) miss the profile DOCKER_HOST and hit the dead rootful socket → idempotent deploy misreads "first-time". Fix with docker context use rootless (shell-independent) + a DOCKER_HOST auto-detect helper. Also: docker exec does NOT inherit the entrypoint's umask — exec-ed steps write group-read-only where the app's own writes are fine
• references/ROOTLESS_DOCKER_OPENVZ.md — getting rootless Docker to run at all on a stripped cgroup-v1 OpenVZ VPS (no per-user systemd, no cgroup delegation, no FUSE, no persistent tun): the root-owned docker-rootless.service system-unit recipe + five compensations (static crun --cgroup-manager=disabled, tmpfs-over-/sys/fs/cgroup, vfs storage, /dev/net/tun tmpfiles, socket ACL). Probe per box (symptom, not uname); reserve for nodes where systemctl --user won't start
• references/NGINX_TLS_REDIRECT_AND_CERTS.md — redirect vhost + Let's Encrypt on a busy proxy: apex served by default_server/first-ssl fallthrough (additive vhost, not replace; wins by server_name/SNI precedence), two-phase :80→issue→append-:443 (cert doesn't exist until issued), ACME-challenge location must stay a more-specific prefix than the 301 (longest-prefix, not file order) as a standing renewal precondition, renewer-unit varies (apt certbot.timer vs snap snap.certbot.renew.timer vs cron). Pairs with the shell maintenance-script contract
• references/TRAEFIK_EDGE_HARDENING.md — public Traefik v3 edge, no plugins: native dotfile-404 (high-priority catch-all router + ipAllowList+rejectStatusCode, short-circuits before a dummy-server noop that must fail closed), the .well-known carve-out (RE2 has no lookahead → && !PathPrefix; never block /.well-known/acme-challenge/ or HTTP-01 renewal dies → cert expires → edge down), per-router rate-limit (omit sourceCriterion; depth>0 trusts spoofable XFF), and the version-bump audit drill (latest-minor-only support, acme.json format unchanged v3.4–v3.7 → cert reused, %2E pre-routing normalization since v3.4.1, cert-reused-first gate + reviewed rollback), the empty-config abort (a programmatically-rendered fragment must emit bare {} when empty — {"http":{"routers":{},"services":{}}} is REJECTED and aborts the whole dynamic build, dropping every @file router estate-wide), and the rate-limit threat model (per-IP caps a single runaway IP, NOT a distributed botnet → set loose, a tight burst only 429s legit asset-heavy pages). Pairs with ROOTLESS_DOCKER.md and CONTAINER_SINGLE_FILE_MOUNT.md
• references/CONTAINER_SINGLE_FILE_MOUNT.md — delivering ONE rendered file into a consumer container without silent staleness: a single-file / volume.subpath mount is inode-bound, so an atomic-rename writer (write tmp; rename) is pinned to the original inode and the consumer never sees updates (fix: in-place O_TRUNC write, or mount the directory); a volume-at-a-directory shadows the image's committed content — named-volume copy-up populates once then freezes, so rebuilt-image files never arrive (nested explicit binds DO compose; overlay one file over a committed placeholder instead); and the reviewer heuristic that catches both — isolation-test the WRITE cycle, not just container startup (startup reads the inode once and passes; the bug is the second write). Pairs with TRAEFIK_EDGE_HARDENING.md §6 empty-config abort
• references/GITHUB_APP_DEPLOY_CREDENTIALS.md — replace personal SSH keys / PATs on servers with short-lived GitHub App installation tokens minted on-box (stateless git credential helper; openssl RS256 JWT → access_tokens, no jq/gh, no token at rest). The two-Apps-not-one invariant (an App's perms are the ceiling for any key-holder → read-only deploy box + write automation host need separate Apps), deploy (read-only) vs automation (write + gh-less PR) cases, the unprivileged deploy-user wiring (!$HOME shell-form + useHttpPath host-vs-path gotcha), and the gotchas (workflows:write for workflow files, Free-private repos don't enforce branch protection, agent hosts may be blocked from prod writes → human runs cutover). Pairs with ROOTLESS_DOCKER.md
• shell skill — base shell-language layer beneath this one (strict-mode hazards, quoting/input hygiene, traps/locking, maintenance-script contract); reach down for script mechanics instead of duplicating
• django skill — backend patterns that interact with deploy (migrations, collectstatic, ASGI)
• RULE_git-safety — rollback is a human-operated procedure
• Docker Compose docs