Ansible configuration management

Ansible Configuration Management

Overview

Terraform builds the house; Ansible furnishes it. Configuration management turns fresh VMs into known-good servers: packages, users, services, files, hardening — described as code, applied idempotently, converging to the declared state.

Idempotency is the contract. Running a playbook twice must change nothing the second time (changed=0). A playbook that isn't idempotent isn't configuration management — it's a fragile script with YAML syntax.

Converge, don't script. Declare the end state (package: state=present), don't orchestrate steps (shell: apt-get install ...). Ansible modules are idempotent; raw command/shell is not.

When to Use

• Configuring OS packages, users, services, files on Linux (or Windows) hosts
• OS hardening (CIS-style controls: sshd, firewall, kernel params)
• Writing or reviewing playbooks, roles, inventories
• Building golden VM images with Packer
• Bootstrapping instances with cloud-init
• Replacing manual "server setup wiki pages" with code

Project Layout

ansible/
├── ansible.cfg
├── inventories/
│   ├── production/
│   │   ├── hosts.yml
│   │   └── group_vars/
│   │       ├── all.yml
│   │       └── webservers.yml
│   └── staging/
├── playbooks/
│   ├── site.yml              # the full converge
│   └── webservers.yml
├── roles/
│   ├── common/               # users, sshd, ntp, agents
│   ├── nginx/
│   └── app/
└── requirements.yml          # pinned external collections/roles

# ansible.cfg — sane defaults
[defaults]
inventory = inventories/production
roles_path = roles:~/.ansible/roles
retry_files_enabled = False
stdout_callback = yaml
forks = 20

[privilege_escalation]
become = True
become_method = sudo

Idempotent Tasks: Right and Wrong

# RIGHT — declarative modules
- name: Install nginx
  ansible.builtin.apt:
    name: nginx
    state: present
    update_cache: true

- name: Configure nginx
  ansible.builtin.template:
    src: nginx.conf.j2
    dest: /etc/nginx/nginx.conf
    owner: root
    group: root
    mode: "0644"
    validate: nginx -t -c %s          # validate before replacing!
  notify: Reload nginx

- name: Nginx running and enabled
  ansible.builtin.service:
    name: nginx
    state: started
    enabled: true

# WRONG — shell for everything; changes every run, no validation
- shell: apt-get install -y nginx                    # not idempotent-aware
- shell: echo "worker_processes 4;" >> /etc/nginx/nginx.conf   # corrupts on re-run

Handler pattern (restart only when config actually changed):

# roles/nginx/handlers/main.yml
- name: Reload nginx
  ansible.builtin.service:
    name: nginx
    state: reloaded

Handlers fire once at play end, only if notified, only when the notifying task changed. Services don't flap on no-op runs.

The Rules

1. Modules over command/shell. If you must use command/shell, make it idempotent with creates:/removes: or changed_when: based on real checks.
2. Check mode must work. --check (dry run) previews changes; design tasks to support it (check_mode: false only for read-only fact gathering).
3. Every file has explicit owner/group/mode. Default permissions are environment-dependent; explicit is auditable.
4. Secrets via Ansible Vault or external lookup — never plaintext vars:

ansible-vault encrypt inventories/production/group_vars/all/vault.yml
ansible-playbook site.yml --ask-vault-pass   # or --vault-password-file in CI from a secrets manager

5. Tags for selective runs: --tags nginx for iteration, but the full site.yml converge is the canonical path.
6. Pin external content: requirements.yml with exact versions; ansible-galaxy install -r requirements.yml in CI.
7. Facts before changes: gather what you branch on (ansible_os_family, ansible_distribution_major_version); don't assume.
8. Lint everything: ansible-lint and yamllint in pre-commit and CI.

Testing Roles with Molecule

Untested roles rot. Molecule spins a throwaway container/VM, applies the role, and asserts the result:

# roles/nginx/molecule/default/molecule.yml
scenario:
  test_sequence:
    - lint
    - destroy
    - create
    - converge
    - idempotence      # applies the role twice; second run must be clean
    - verify
    - destroy

# roles/nginx/molecule/default/verify.yml
- name: Verify nginx
  hosts: instance
  tasks:
    - name: Nginx is installed
      ansible.builtin.package:
        name: nginx
      check_mode: true
      register: pkg
      failed_when: pkg is changed

    - name: Port 80 is listening
      ansible.builtin.wait_for:
        port: 80
        timeout: 5

The idempotence step is the killer feature: it proves the contract automatically.

Hardening Baseline (what "configured" means)

A converged server has, at minimum:

• sshd: PermitRootLogin no, PasswordAuthentication no, key-only access, explicit AllowUsers/AllowGroups
• Unattended security upgrades enabled (deliberate, tested — not blind nightly kernel jumps on prod)
• Firewall default-deny inbound, explicit allows only (ufw/nftables/firewalld)
• NTP synchronized (chrony/systemd-timesyncd)
• Centralized logging agent installed (see observability-and-alerting)
• Unneeded packages and services removed/disabled
• Service users are unprivileged and shell-less (shell: /usr/sbin/nologin)
• Kernel params for the workload (sysctl managed as code)
• fail2ban or equivalent where SSH is internet-reachable (it shouldn't be — VPN/bastion)

Use established hardening collections (e.g., devsec.hardening) as a baseline, reviewed and pinned — not as unexamined magic.

Packer and cloud-init: Images and First Boot

Packer builds golden images so instances boot pre-hardened, pre-monitored, fast:

# image.pkr.hcl — build once, promote to accounts/regions
source "amazon-ebs" "base" {
  source_ami_filter {
    filters = { name = "ubuntu/images/hvm-ssd/ubuntu-jammy-22.04-amd64-server-*" }
    owners  = ["099720109477"]   # Canonical only
    most_recent = true
  }
  ssh_username = "ubuntu"
}

build {
  sources = ["source.amazon-ebs.base"]
  provisioner "ansible" {
    playbook_file = "ansible/playbooks/image-hardening.yml"
  }
  post-processor "manifest" {}   # records AMI ID → feed to Terraform
}

cloud-init handles per-boot identity (hostname, users, SSH keys, small joins), not full configuration:

#cloud-config
hostname: web-01
users:
  - name: deploy
    groups: [sudo]
    shell: /bin/bash
    ssh_authorized_keys:
      - ssh-ed25519 AAAA... 
package_update: true
runcmd:
  - /opt/bootstrap/register-with-fleet.sh

The split: cloud-init for identity, Packer for content, Ansible for everything else. Don't fight the tools' grain.

Execution Model

• Playbooks run from CI/CD or a controlled runner, against inventory from source control — not from a laptop against production.
• Production runs are limited batches (--limit web[0-1] style canary) or serial: 1 rolling plays, with verification between batches.
• Every run is logged; CI keeps the output attached to the run.

Common Rationalizations

Rationalization	Reality
"shell: is faster to write than finding the module"	It's faster once. Then it runs every converge, breaks on re-run, and can't be validated or checked. The module pays for itself on the second run.
"Idempotency is nice-to-have"	Non-idempotent playbooks can't be scheduled, can't be trusted, and can't be diffed. changed=0 on re-run is the definition of done.
"Testing roles is over-engineering"	Molecule's idempotence step alone catches the most common class of playbook bugs. Untested roles are scripts with extra steps.
"I'll encrypt secrets later"	Later means after the repo was cloned to three laptops and a CI log. Vault from the first secret, not after the first incident.
"Manual hotfix on the box, I'll reconcile later"	The next converge silently reverts your fix — or worse, half-reverts it. Fix in code, converge, done.
"Hardening slows down debugging, turn it off"	A server you can debug but anyone can enter is not a trade worth making. Debug through logs and sudo, not through PermitRootLogin yes.

Red Flags

• Playbooks dominated by command/shell/raw
• changed results on every re-run of a converged host
• Plaintext passwords/keys in group_vars or host_vars
• validate: missing on template tasks that write service configs
• Roles without molecule/ (or any test)
• gather_facts: false with tasks that assume OS details
• Servers configured "by the wiki page" instead of by converge
• Inventory hardcoded in someone's local hosts file

Verification

• ansible-lint and yamllint pass clean
• Full converge runs green: ansible-playbook site.yml
• Immediate re-run reports changed=0 on every host (idempotency proven)
• --check mode completes and shows a meaningful diff
• Molecule tests pass for every role, including the idempotence step
• All secrets encrypted with Ansible Vault or pulled from an external secrets manager at runtime
• Every templated config uses validate: before replacing the live file
• Runs execute from CI/controlled runners with logged output — not from personal laptops against production