Server security

Server Security

A disciplined audit-then-harden workflow for a fresh or unsupervised Debian/Ubuntu server, reached over SSH. The skill covers the eight areas that block roughly 80% of automated attacks: system packages, a non-root sudo user, ed25519 SSH keys, sshd hardening, UFW, unattended-upgrades, fail2ban, and sudo + home permissions.

The skill is split into six phases. Do not skip phases. The audit phase must not mutate anything; mutations only happen in Phase 5 after the user explicitly approves the plan.

When parsing or fixing a specific check, consult references/checks.md for the precise commands and target values. When touching sshd_config, consult references/ssh-safety.md for the fail-safe rollback protocol — never edit sshd without it.

Phase 1 — Connect and detect environment

Ask the user for whatever connection details are missing. Required:

• Host (IP or DNS name)
• User to connect as (often root initially on a fresh server, or an existing deploy-style user)
• Port (default 22)
• Identity file (optional; SSH agent or default keys are fine)

Then verify reachability and capabilities. Use one ssh invocation that runs a small bundle of read-only commands and parse the output:

ssh -o ConnectTimeout=10 -o BatchMode=yes -p <port> <user>@<host> \
    'set -e; \
     . /etc/os-release && echo "OS=$ID VERSION=$VERSION_ID"; \
     id; \
     command -v sudo >/dev/null && echo "sudo=present" || echo "sudo=missing"; \
     sudo -n true 2>/dev/null && echo "sudo=passwordless" || echo "sudo=needs-password"; \
     uname -r'

From the output decide:

• OS check. If ID is not debian or ubuntu (or a derivative like linuxmint, pop), stop. Tell the user the skill currently targets Debian/Ubuntu and ask whether to continue at their own risk (in which case some commands may fail).
• Privilege check. Many audit commands require root reads (/etc/ssh/sshd_config.d/, sshd -T, ufw status, fail2ban-client). If the user has sudo but it requires a password, the skill must run with sudo and the user must be prepared to type their password — or grant passwordless sudo for the duration. If the connecting user has no sudo at all (and is not root), abort with a clear message.
• Connecting-as-root note. Connecting as root is normal on a brand-new VPS. The skill should plan to create a non-root user as part of the fix, then advise switching the SSH session to that user before disabling root login.

Speed tip. If you will run more than a few SSH commands, enable a ControlMaster session so each subsequent command does not pay the handshake cost:

mkdir -p ~/.ssh/cm && \
ssh -o ControlMaster=auto \
    -o ControlPath=~/.ssh/cm/%r@%h:%p \
    -o ControlPersist=10m \
    -p <port> <user>@<host> true

Subsequent ssh calls with the same ControlPath reuse the connection. Tear it down at the end with ssh -O exit <user>@<host>.

Phase 2 — Audit (read-only)

Run all eight checks. Read only — no mutations, no service restarts, no package installs. If a command requires root, prepend sudo (it will use the cached or passwordless sudo).

Collect the raw output and parse into a structured result per check. references/checks.md contains the exact detection command and the expected pass condition for each. The eight checks are:

1. System updates — apt list --upgradable count, age of /var/cache/apt/pkgcache.bin.
2. Non-root sudo user — at least one non-root user in sudo (or wheel on RHEL-derivatives, but we only support Debian/Ubuntu) with a populated ~/.ssh/authorized_keys.
3. SSH key strength — the keys in authorized_keys are ed25519 or rsa ≥ 3072 bits.
4. sshd hardening — effective values via sudo sshd -T match the target table in references/checks.md. Also enumerate /etc/ssh/sshd_config.d/*.conf in lexical order and flag any earlier-loading file that conflicts.
5. UFW firewall — active, default-deny incoming, default-allow outgoing, SSH port allowed.
6. Unattended-upgrades — package installed, enabled in 20auto-upgrades, security origin enabled in 50unattended-upgrades, automatic-reboot configured.
7. Fail2ban — service active, sshd jail enabled, ban policy reasonable.
8. Sudo and home permissions — sudo visudo -c clean, target user's home is mode 700.

For each check, record:

• status: pass / warn / fail / skipped
• current: what was observed (short string, e.g. PermitRootLogin=yes)
• target: what the article expects
• severity: info / low / medium / high
• notes: anything the user needs to know (e.g. "drop-in 00-cloud-init.conf sets PasswordAuthentication yes before our hardening file would load")

Phase 3 — Present report

Produce a single markdown report in this exact shape so the user can scan it in one pass:

# Server security audit — <host>

Connected as `<user>` to `<host>:<port>` on <ID> <VERSION>. Kernel `<uname -r>`.

## Summary
- ✅ Passing: N
- ⚠️ Warnings: N
- ❌ Failing: N

## Findings

| # | Area | Status | Current | Target | Severity |
|---|------|--------|---------|--------|----------|
| 1 | System updates | ⚠️ | 23 packages upgradable | up to date | low |
| 2 | Non-root sudo user | ✅ | `deploy` with ed25519 key | non-root + key | – |
| 3 | SSH PermitRootLogin | ❌ | `yes` | `no` | high |
| … |

## Details

### 3. SSH PermitRootLogin
Currently `PermitRootLogin yes` (set in `/etc/ssh/sshd_config` line 32). Allowing direct root logins exposes the most-targeted account on the server to credential attacks. Fix: drop `PermitRootLogin no` into `/etc/ssh/sshd_config.d/01-hardening.conf`. **Before disabling, verify a non-root sudo user with a working key — Phase 2 check #2.**

### …

For each failing or warning check, include a short Details paragraph explaining what was found and why it matters. Do not propose commands here — that comes in Phase 4.

Phase 4 — Confirm fix plan

Group fixes into two batches and present them separately.

Batch A — Safe fixes (low risk of lockout):

1. apt update && apt upgrade -y and install missing packages.
2. Create the non-root sudo user and install the SSH public key (if missing). 2b. Configure sudo authentication for that user — NOPASSWD drop-in or passwd. Ask the operator to pick one during the Phase 4 prompt; this is mandatory, not optional. See references/checks.md § 2 for the trade-offs.
3. UFW configuration.
4. Unattended-upgrades configuration.
5. Fail2ban configuration.
6. Home directory permissions, sudoers validation.

Batch B — SSH hardening (lockout risk if misconfigured): 7. Drop-in /etc/ssh/sshd_config.d/01-hardening.conf with the target values. 8. Restart ssh — but only via the rollback-guarded protocol in references/ssh-safety.md.

Show each batch as a numbered list. For each item, show the exact command(s) that will run. Ask the user, in one message, "Apply Batch A?" — yes/no/select-subset — and separately "Apply Batch B?" with an explicit note that this is the lockout-risk batch and the rollback protocol will run automatically.

If the user wants to cherry-pick, accept a list like A: 1,3,5 or natural language ("skip the firewall, do the rest"). Re-state what will run and ask once more for confirmation before doing anything.

Phase 5 — Apply fixes

Interactive commands. Anything that prompts the operator for input — passwd, an editor opened by visudo without -c, dpkg-reconfigure at its default priority, apt configfile diffs — cannot run through the agent's shell. The agent's stdin is not wired to the operator. When the skill needs one of these, tell the operator to run it in a separate terminal (or in their existing one) and wait for confirmation in chat before continuing. Never try to script around an interactive prompt with heredocs, expect, or by accepting the secret in chat to "automate" it — these either fail outright or leak the secret into the transcript.

Order matters. Run Batch A first, in this order, stopping the whole batch on the first failure:

1. System updates and packages

   sudo apt update
   sudo apt upgrade -y
   sudo apt install -y sudo curl wget gnupg ca-certificates ufw fail2ban unattended-upgrades
   

   apt upgrade -y can prompt about modified config files (/etc/...). Pass DEBIAN_FRONTEND=noninteractive plus -o Dpkg::Options::="--force-confold" to keep existing config files. If a kernel upgrade installed, note that a reboot is recommended (do not reboot inside the skill — surface it as a final action item).

2. Non-root user — account + key. Skip if a suitable user already exists with a working key.

   sudo adduser --gecos "" --disabled-password <name>
   sudo usermod -aG sudo <name>
   sudo install -d -m 700 -o <name> -g <name> /home/<name>/.ssh
   echo "<public-key>" | sudo install -m 600 -o <name> -g <name> /dev/stdin /home/<name>/.ssh/authorized_keys
   

   The public key must come from the user. If they do not have one, point them at ssh-keygen -t ed25519 -a 100 -C "<name>@<hostname>" on their local machine — never generate the private key on the server.

2b. Non-root user — sudo authentication. adduser --disabled-password puts ! in /etc/shadow — the account cannot use sudo until you give it a way to authenticate. This is a mandatory step before Batch B; do not skip it. The operator picked Option A or Option B during Phase 4 — apply the matching commands now (full rationale in references/checks.md § 2):

• Option A — NOPASSWD sudo:

  printf '%s ALL=(ALL) NOPASSWD:ALL\n' "<name>" \
      | sudo install -m 0440 -o root -g root /dev/stdin /etc/sudoers.d/90-<name>
  sudo visudo -c
  

• Option B — password-protected sudo: passwd is interactive and cannot run through the agent's shell. Do not try — no heredoc, no piping, no accepting the password in chat. Tell the operator to open a separate terminal, log in as the existing sudo user (or root), and run there:

  sudo passwd <name>
  

  Wait for the operator to confirm "done" in chat before continuing.

Verify before continuing. Ask the operator to open a second terminal: ssh -p <port> <name>@<host>, then run a sudo command in that session — sudo -n true for Option A, sudo true for Option B. Wait for the operator to confirm "got a shell + sudo works" in chat. Without this confirmation, Batch B is unsafe: the SSH rollback can only be cancelled from a session that can run sudo, and the existing root session may not survive the reload.

3. UFW.

   sudo ufw default deny incoming
   sudo ufw default allow outgoing
   sudo ufw allow <ssh-port>/tcp comment 'ssh'
   sudo ufw logging low
   yes | sudo ufw enable
   

   If the user already has other rules (web ports, etc.), preserve them — list current rules first with sudo ufw status numbered and confirm.

4. Unattended-upgrades. Configure non-interactively rather than running dpkg-reconfigure:

   sudo tee /etc/apt/apt.conf.d/20auto-upgrades >/dev/null <<'EOF'
   APT::Periodic::Update-Package-Lists "1";
   APT::Periodic::Unattended-Upgrade "1";
   APT::Periodic::AutocleanInterval "7";
   EOF
   

   Then patch /etc/apt/apt.conf.d/50unattended-upgrades to set Unattended-Upgrade::Automatic-Reboot "true";, Unattended-Upgrade::Automatic-Reboot-Time "04:00";, and Unattended-Upgrade::Remove-Unused-Kernel-Packages "true";. See references/checks.md for the sed/awk patterns.

5. Fail2ban.

   sudo cp -n /etc/fail2ban/jail.conf /etc/fail2ban/jail.local
   

   Write /etc/fail2ban/jail.d/sshd.local with the values from references/checks.md (bantime, findtime, maxretry, mode=aggressive). Then:

   sudo systemctl enable --now fail2ban
   sudo fail2ban-client status sshd
   

6. Home permissions and sudoers.

   sudo chmod 700 /home/<name>
   sudo visudo -c
   

Then Batch B — only proceed if Batch A items 2 and 2b both succeeded and the operator has confirmed, from a second terminal logged in as the deploy user, that both ssh and sudo work in that session. Without both confirmations, refuse to touch sshd; explain that disabling root login or password auth before key + sudo are verified in the deploy session will leave the operator unable to cancel the rollback if the root session dies during reload.

7. sshd hardening with rollback. Use the full protocol in references/ssh-safety.md. The short version:
   1. Write the proposed config to /tmp/01-hardening.conf.new and validate with sudo sshd -t -f /tmp/01-hardening.conf.new — abort on error.
   2. Back up any existing /etc/ssh/sshd_config.d/01-hardening.conf to /tmp/01-hardening.conf.bak.
   3. Schedule an auto-rollback in 5 minutes using a transient systemd-run unit that restores the backup (or deletes the new file if there was no backup) and restarts ssh.
   4. Move the new file into place and sudo systemctl reload ssh (reload, not restart — existing sessions stay alive either way, but reload is the documented hot-swap path).
   5. Tell the user, verbatim: "I have applied the SSH hardening. Open a new terminal and run ssh -p <port> <name>@<host> to confirm you can still log in. Reply once you see a shell prompt — I have 5 minutes before the config rolls back automatically."
   6. On confirmation, cancel the rollback unit.
   7. If the user reports they cannot log in, do not try to fix it — wait. The rollback will fire and restore working ssh.

Phase 6 — Verify and report

After Batch A and (optionally) Batch B, re-run the audit phase. Show a tiny diff: "Before: 3 failing, 2 warnings. After: 0 failing, 1 warning." Surface any item that did not improve and explain why.

Finish with a short action list of things the user must do off-server (or on-server, but outside this skill's scope):

• Confirm their local SSH config has Host <alias>\n HostName <host>\n User <name>\n IdentityFile ~/.ssh/<key> so future connections use the new user automatically.
• Don't leave the deploy user as "NOPASSWD sudo + no password set" forever. If Option A was chosen during step 2b, anyone who acquires the SSH private key gets instant root with no second factor. Run sudo passwd <name> to set a sudo password. Then either remove /etc/sudoers.d/90-<name> to require the password on every sudo (full second factor restored), or leave NOPASSWD for ergonomics — that is a trade-off the operator should make consciously. The one state to avoid is "non-root user with NOPASSWD and no password set": no second factor, and su - / any password-gated tool will not work either.
• Ubuntu locks the root account by default. With PermitRootLogin no and a locked root, the only way in is via the deploy user — which is exactly the desired state. If the operator ever genuinely needs su - root from deploy (rare), sudo passwd root sets a root password; otherwise leave root locked, because a locked root account combined with PermitRootLogin no is strictly better than any password-protected root.
• If a kernel was upgraded, schedule a reboot.
• Rotate any API keys or secrets that may have been pasted to the server during setup, and create per-project keys with spend limits in Anthropic / OpenAI / etc. consoles if the server runs LLM agents.

Non-goals

• Application-layer hardening (nginx, docker, postgres, app configs) — out of scope for v1; see references/checks.md for what is not covered.
• OS reinstall / image rebuild. The skill assumes a running server you can SSH into.
• IDS, SIEM, audit logging beyond fail2ban, and full CIS-benchmark compliance — also out of scope.
• Generating SSH keypairs on the server — keys must be generated on the user's local machine; only the public half ever touches the server.