Shell scripting automation

Shell Scripting and Automation

Overview

Automation is the DevOps engine, and a bad script is worse than no script: it fails halfway, leaves systems in unknown states, and lies about it with exit code 0. Every script you write is production code — it gets reviewed, tested, and maintained like one.

Idempotency is the contract. Running a script twice must produce the same state as running it once. Scripts that depend on "what happened before" are incidents waiting for a retry.

Fail loudly, fail early, fail safe. A script that silently continues after an error is a corrupted state generator.

When to Use

• Automating any task performed more than twice
• Writing Bash scripts for Linux operations
• Building Python CLI tools for complex automation (APIs, data transformation, validation)
• Writing PowerShell for Windows Server, Active Directory, IIS, or .NET estates
• Reviewing existing scripts for safety

Choosing the Tool

Is it mostly orchestrating existing commands on Linux?
└── Yes, and < ~100 lines, no complex data structures → Bash

Does it parse JSON/YAML, call APIs, need retries, tests, or validation?
└── Yes → Python (typer/click + requests + pydantic + pytest)

Does it touch Windows Server, AD, IIS, Exchange, or .NET?
└── Yes → PowerShell (7+ where possible; WinRM/PSRemoting for remote)

Do not write JSON-parsing logic in Bash with grep/sed. That's what jq (acceptable) or Python (better) is for.

Bash: The Mandatory Template

Every Bash script starts like this. No exceptions:

#!/usr/bin/env bash
set -Eeuo pipefail
IFS=$'\n\t'

# What each safety flag buys you:
#   -e          exit on any command failure
#   -E          ERR traps propagate into functions/subshells
#   -u          treat unset variables as errors (no more rm -rf "$EMPTY_VAR/")
#   -o pipefail a failed pipe stage fails the pipeline (no more lost errors)

log() { printf '[%s] %s\n' "$(date -u +%FT%TZ)" "$*" >&2; }
die() { log "ERROR: $*"; exit 1; }

cleanup() {
  # Always runs (success, failure, Ctrl+C): remove temp files, release locks
  rm -f "$TMPFILE" 2>/dev/null || true
}
trap cleanup EXIT
trap 'die "Failed at line $LINENO"' ERR

usage() {
  cat <<EOF
Usage: $(basename "$0") [--dry-run] <environment>

Deploys the thing. Idempotent: safe to re-run.
EOF
  exit "${1:-0}"
}

DRY_RUN=0
while [[ $# -gt 0 ]]; do
  case "$1" in
    --dry-run) DRY_RUN=1; shift ;;
    -h|--help) usage 0 ;;
    -*) usage 1 ;;
    *) break ;;
  esac
done

[[ $# -ge 1 ]] || usage 1
ENV="$1"
[[ "$ENV" =~ ^(dev|staging|prod)$ ]] || die "Invalid environment: $ENV"

main() {
  TMPFILE="$(mktemp)"
  log "Starting against ${ENV} (dry_run=${DRY_RUN})"
  # ... work here ...
  log "Done"
}

main "$@"

Bash: The Idempotency Patterns

# Create-or-update, never create-and-fail
mkdir -p /opt/app                       # -p: no error if it exists
install -m 0644 -o app -g app config.toml /etc/app/config.toml

# Package installation: check first (or use the package manager's own idempotency)
dpkg -s nginx &>/dev/null || apt-get install -y nginx

# Line-in-file: replace or append, never blind-append
grep -q '^maxconn ' /etc/haproxy/haproxy.cfg \
  && sed -i 's/^maxconn .*/maxconn 4096/' /etc/haproxy/haproxy.cfg \
  || echo 'maxconn 4096' >> /etc/haproxy/haproxy.cfg

# Services: enable --now is idempotent
systemctl enable --now nginx

# Downloads: verify, don't re-download
if [[ ! -f app.tar.gz ]] || ! sha256sum -c app.tar.gz.sha256; then
  curl -fsSL -o app.tar.gz "$URL"
  sha256sum -c app.tar.gz.sha256
fi

Destructive operations need guards:

# Never: rm -rf "$DIR/"   with an unquoted or possibly-empty variable
# Always:
[[ -n "$WORKDIR" && "$WORKDIR" == /tmp/build-* ]] || die "Refusing unsafe WORKDIR='$WORKDIR'"
rm -rf -- "$WORKDIR"

Bash: The Safety Rules

1. Quote every expansion: "$var", not $var. Word-splitting bugs are the #1 Bash footgun.
2. Use [[ ]] not [ ] for conditionals — safer pattern matching, no pathname expansion surprises.
3. Never parse ls — use globs or find -print0 | while IFS= read -r -d ''.
4. Validate all input, including positional params, before acting on it.
5. Use mktemp for temp files — never predictable paths in /tmp (symlink attacks).
6. Return meaningful exit codes: 0 success, 1 runtime error, 2 usage error.
7. Lint everything: shellcheck -x script.sh is part of CI, not optional.
8. Format consistently: shfmt -i 2 -ci -w . in pre-commit.
9. No secrets on the command line — they're visible in ps, shell history, and logs. Read from files with 0600 perms or from a secrets manager.

Python: When Bash Ends

#!/usr/bin/env python3
"""Sync DNS records from the zone file to the provider API.

Idempotent: creates missing records, updates drifted ones, never deletes.
"""
from __future__ import annotations

import sys
import typer
import requests
from pydantic import BaseModel, Field

app = typer.Typer(help="Idempotent DNS record sync.")

class DnsRecord(BaseModel):
    name: str
    type: str = Field(pattern="^(A|AAAA|CNAME|TXT)$")
    value: str
    ttl: int = 300

@app.command()
def sync(zone: str, dry_run: bool = False) -> None:
    """Reconcile desired records with the provider."""
    desired = load_desired(zone)            # validated with DnsRecord
    current = fetch_current(zone)           # via requests, with retries
    plan = diff(desired, current)           # create/update/none
    for change in plan:
        typer.echo(f"{change.action}: {change.record.name}")
        if not dry_run and change.action != "none":
            apply_change(change)
    typer.echo(f"Done: {plan.summary()}")

if __name__ == "__main__":
    app()

Python automation standards:

• typer (or click) for CLIs — real --help, real validation
• pydantic for anything crossing a boundary (API payloads, YAML configs)
• requests with explicit timeout= and retry logic (urllib3 Retry or tenacity)
• pytest tests for the pure logic (diffing, transformation) — automation deserves tests too
• Structured logging (logging module), never bare print for operational output
• Same contract as Bash: idempotent, meaningful exit codes, --dry-run for anything that changes state

PowerShell: The Windows Estate

#Requires -Version 7.0
[CmdletBinding(SupportsShouldProcess)]
param(
    [Parameter(Mandatory)]
    [ValidateSet('dev','staging','prod')]
    [string]$Environment
)

Set-StrictMode -Version Latest
$ErrorActionPreference = 'Stop'

# Idempotent: check before change
$feature = 'Web-Server'
if ((Get-WindowsFeature -Name $feature).InstallState -ne 'Installed') {
    Install-WindowsFeature -Name $feature -IncludeManagementTools
}

# Remote administration via PowerShell Remoting (WinRM)
Invoke-Command -ComputerName web01, web02 -ScriptBlock {
    Get-Service -Name W3SVC | Select-Object PSComputerName, Status
} -Credential (Get-Credential)

PowerShell standards: Set-StrictMode, $ErrorActionPreference = 'Stop', SupportsShouldProcess (-WhatIf) on anything destructive, approved verbs (Get-, Set-, New-, Remove-), and PSScriptAnalyzer in CI.

Testing Scripts

Scripts get tested like code:

• Bash: shellcheck + bats for behavior tests on critical scripts
• Python: pytest with mocked HTTP (responses library)
• PowerShell: PSScriptAnalyzer + Pester
• All: a --dry-run/-WhatIf mode exercised in CI, so the planning path never rots

Common Rationalizations

Rationalization	Reality
"It's a throwaway script"	Throwaway scripts get copy-pasted into cron jobs and run for three years. Write it properly or don't write it.
"set -e is annoying, my script fails for no reason"	It fails because of a reason you were ignoring. A script that continues after an error produces corrupt state with exit code 0.
"Quoting everything is ugly"	Unquoted expansions are how rm -rf $DIR/ deletes a filesystem when DIR is empty. Ugliness is cheaper than recovery.
"I'll parse the API output with grep"	JSON field order and whitespace are not a contract. Use jq or Python — grep-on-JSON is a time bomb.
"Idempotency is overkill for a one-off"	One-offs get retried after they half-fail at 2 AM. The retry is when non-idempotency hurts.
"Testing a shell script is over-engineering"	The script manages production systems. It deserves at least shellcheck and a dry-run mode.

Red Flags

• Scripts without set -Eeuo pipefail (Bash) / Set-StrictMode (PowerShell)
• Unquoted variables anywhere
• Secrets as positional arguments or hardcoded literals
• rm -rf with unvalidated variables
• Scripts that blind-append to config files (run it twice, broken config)
• No --dry-run on state-changing scripts
• Exit code 0 on failure paths ("error handling" that prints and continues)
• Cron jobs pointing at scripts in someone's home directory

Verification

• shellcheck / PSScriptAnalyzer / pytest pass clean
• Script succeeds when run twice in a row (idempotency proven, not assumed)
• --dry-run mode exists and is exercised for state-changing scripts
• Meaningful exit codes: 0 success, non-zero on any failure
• All variables quoted; all inputs validated before use
• No secrets in arguments, literals, or logs
• Error paths tested: what happens when the API is down / the disk is full / the lock is held?
• The script is in version control, reviewed like production code