Coder k3s

Coder on k3s

End-to-end skill for running Coder (self-hosted dev environments) on a k3s cluster over Tailscale.

Cluster Context

• Coder server: <CODER_SERVER_HOST> (NodePort 31557), namespace coder
• Access URL: http://<CODER_SERVER_IP>:31557 (Tailscale direct) or SSH tunnel → http://localhost:8081
• SSH tunnel: ssh -L 8081:<CODER_SERVER_IP>:31557 <USER>@<CODER_SERVER_HOST> -N
• PostgreSQL: coder-db pod in coder namespace on <CODER_SERVER_HOST>
• Helm values: ~/coder/helm-values.yaml
• Templates dir: ~/coder/workspace-template/ (ai-workspace), ~/coder/workspace-template-minimal/ (minimal-ubuntu)

When to Use

• Installing or upgrading Coder via Helm
• Authenticating the coder CLI (token-based, non-interactive)
• Creating or pushing workspace templates
• Debugging workspace pod failures (502, connection refused, bad gateway)
• Configuring GPU vs CPU node scheduling in templates
• Setting up per-user namespace isolation
• Integrating workspaces with LiteLLM or other cluster services

---

Install & Upgrade

See ./references/install.md for full procedure.

Quick upgrade:

helm upgrade coder coder-v2/coder -n coder -f ~/coder/helm-values.yaml

---

CLI Authentication

Coder CLI requires a session token — browser auth doesn't work over SSH.

# 1. Open UI → Settings → Tokens → + New token
# 2. Run:
ssh <USER>@<CODER_SERVER_HOST> "coder login http://<CODER_SERVER_IP>:31557 --token <token>"

Verify:

ssh <USER>@<CODER_SERVER_HOST> "coder list"

---

Template Management

See ./references/templates.md for authoring guide.

Push an update

scp -r ~/coder/workspace-template <USER>@<CODER_SERVER_HOST>:/tmp/
ssh <USER>@<CODER_SERVER_HOST> "coder templates push ai-workspace --directory /tmp/workspace-template/ --yes"

Update existing workspace to new template version

ssh <USER>@<CODER_SERVER_HOST> "coder update <owner>/<workspace>"

List templates

ssh <USER>@<CODER_SERVER_HOST> "coder templates list"

---

Workspace Storage — What Persists vs What Doesn't

Each workspace has a PVC mounted at /home/coder. Only this directory survives a workspace stop/start.

Location	Persists?	Notes
/home/coder/**	✅ Yes	PVC — survives pod deletion
~/.local, ~/.config, ~/.bashrc	✅ Yes	Inside /home/coder
System packages (apt install)	❌ No	Container filesystem — gone on stop
/usr/local/**, /opt/**	❌ No	Ephemeral container layer

Making installed tools persist

Option 1 — Install to home dir (immediate, no template change):

# Example: install a binary to ~/.local/bin (persists)
curl -fsSL https://example.com/install.sh | INSTALL_DIR=/home/coder/.local/bin bash
echo 'export PATH="$HOME/.local/bin:$PATH"' >> ~/.bashrc

Option 2 — Add to startup_script (reinstalls on every start, reproducible):

startup_script = <<-EOT
  # Pixi package manager — install once, cached on PVC
  if [ ! -f /home/coder/.pixi/bin/pixi ]; then
    curl -fsSL https://pixi.sh/install.sh | bash
  fi
  echo 'export PATH="/home/coder/.pixi/bin:$PATH"' >> /home/coder/.bashrc
  
  # System tools reinstalled on every start (fast with apt cache)
  sudo apt-get install -y -qq curl git vim python3-pip
EOT

Option 3 — Custom Docker image (fastest startup, most reproducible): Replace codercom/enterprise-base:ubuntu with a custom image that has your tools pre-baked.

---

Workspace Operations

# List all workspaces
ssh <USER>@<CODER_SERVER_HOST> "coder list"

# Create workspace from template
ssh <USER>@<CODER_SERVER_HOST> "coder create --template=ai-workspace --org=coder myworkspace"

# Delete workspace
ssh <USER>@<CODER_SERVER_HOST> "coder delete <owner>/<workspace> --yes"

# Get workspace pod
kubectl get pods -A | grep coder- | grep -v '^coder '

# Exec into workspace pod
kubectl exec -n coder-<owner> workspace-<name> -- bash

restart vs update — important distinction

Command	Template version used	When to use
coder restart <ws> --yes	Same version as currently running	Bounce the pod (no template change)
coder update <ws>	Latest published template version	After pushing a template fix — does NOT accept --yes

After pushing a template fix with coder templates push, always use coder update (not coder restart) to pick up the change:

# Push fix
scp -r ~/coder/workspace-template-minimal <USER>@<CODER_SERVER_HOST>:/tmp/
ssh <USER>@<CODER_SERVER_HOST> "coder templates push minimal-ubuntu --directory /tmp/workspace-template-minimal/ --yes"

# Apply to a running workspace (no --yes flag available)
ssh <USER>@<CODER_SERVER_HOST> "coder update <owner>/<workspace>"

Monitor until healthy:

ssh <USER>@<CODER_SERVER_HOST> "coder logs <owner>/<workspace> --follow"
# Confirm: HEALTHY: true
ssh <USER>@<CODER_SERVER_HOST> "coder list"

---

Troubleshooting

See ./references/troubleshoot.md for all known issues.

Quick checks:

# Check startup script log
kubectl exec -n coder-<owner> workspace-<name> -- tail -50 /tmp/coder-startup-script.log

# Check if code-server is listening
kubectl exec -n coder-<owner> workspace-<name> -- ss -tlnp | grep 13337

# Check coder server logs
kubectl logs -n coder -l app.kubernetes.io/name=coder --tail=50