Terraform infrastructure as code

Terraform Infrastructure as Code

Overview

Terraform is how infrastructure stops being a pile of console clicks and becomes reviewable, repeatable, versioned code. The discipline matters more than the syntax: state is sacred, plans are reviewed, and blast radius is deliberately small.

The plan is the contract. You never apply what you haven't read. terraform plan output is the single most important artifact in the workflow — reviewed like a diff, attached to the PR, verified after apply.

Small states, small blast radius. One giant state file means every change risks everything. Split state by environment, by component lifecycle, and by team ownership.

When to Use

• Provisioning any cloud resource (networks, VMs, clusters, databases, DNS)
• Writing or reviewing Terraform/OpenTofu code
• Designing or refactoring modules
• Configuring or migrating state backends
• Investigating drift between code and reality
• Planning imports of manually-created resources

The Non-Negotiable Workflow

1. BRANCH       git checkout -b feat/add-staging-vpc
2. WRITE        terraform fmt && terraform validate (pre-commit hooks run these)
3. PLAN         terraform plan -out=tfplan        ← read every line
4. REVIEW       PR with the plan output pasted    ← a human reads it too
5. MERGE        CI re-runs plan on main
6. APPLY        terraform apply tfplan            ← apply the REVIEWED plan file,
                in CI/CD, never from a laptop against shared environments
7. VERIFY       plan again → "No changes"         ← proof, not assumption

terraform apply without a reviewed plan is a red-alert practice. The plan file binds what was reviewed to what is executed.

State: The Rules That Are Never Broken

terraform {
  backend "s3" {
    bucket         = "org-terraform-state"
    key            = "networking/prod/terraform.tfstate"
    region         = "eu-west-1"
    encrypt        = true
    dynamodb_table = "terraform-state-lock"   # or use_lockfile on modern S3 backends
    kms_key_id     = "alias/terraform-state"
  }
}

1. Remote state always. S3+lock / Azure Blob+lease / GCS / Terraform Cloud / HCP Terraform. Local state is for throwaway learning only.
2. Encrypted at rest, access-controlled, versioned (state history is your undo button and your audit log).
3. Locking enabled — two concurrent applies corrupt state. The lock is not optional.
4. terraform.tfstate NEVER in Git. It's in .gitignore from the first commit. State contains plaintext secrets (RDS passwords, private keys) by design.
5. State access is narrower than apply access. Reading state exposes secrets; grant it deliberately.

Repository Layout: Small Blast Radius

infra/
├── modules/                    # reusable, versioned, independently tested
│   ├── vpc/
│   ├── eks/
│   └── rds/
├── environments/
│   ├── prod/
│   │   ├── networking/         # own state file: networking/prod
│   │   ├── data/               # own state file: data/prod  (databases — slow, careful)
│   │   └── apps/               # own state file: apps/prod   (changes often)
│   └── staging/
│       ├── networking/
│       ├── data/
│       └── apps/

Why three states per environment: app changes happen daily; database changes are rare and terrifying; network changes are rarest. If the app state is corrupted or misapplied, the database state is untouched. Match state boundaries to lifecycle, ownership, and risk — HashiCorp's own adopt-phase guidance.

Cross-state references use terraform_remote_state data sources with read-only access, or explicit outputs passed as variables. No wild cross-reads of everything.

Module Discipline

module "vpc" {
  source  = "git::https://github.com/org/infra-modules.git//vpc?ref=v2.3.1"  # pinned version

  name        = "prod"
  cidr_block  = "10.20.0.0/16"
  az_count    = 3

  tags = local.standard_tags
}

• Modules for repeated patterns: networks, clusters, databases, IAM roles. Copy-pasted 200-line resource blocks across environments are where drift is born.
• Modules are versioned (?ref=v2.3.1) — upgrading a module is a deliberate, reviewable change, not a surprise.
• Modules expose variables, not magic. No hardcoded environment names inside a shared module.
• Don't over-modularize. A module wrapping a single resource adds indirection, not value. Module-ize what repeats or what needs a standard (tagging, encryption, logging) enforced.

Code Quality Gates (pre-commit + CI)

terraform fmt -check -recursive     # formatting
terraform validate                  # syntax + internal consistency
tflint                              # provider-aware linting (unused vars, deprecated attrs)
trivy config .                      # or checkov/tfsec — misconfig scanning of the IaC itself

These run in pre-commit hooks AND in CI on every PR. The IaC scan is as mandatory as the unit tests — an open security group is easier to catch in HCL than in a cloud audit.

Safe Apply Practices

• terraform plan -out=planfile → review → terraform apply planfile. The reviewed artifact is what executes.
• Applies run in CI/CD with OIDC cloud credentials (see ci-cd-pipeline-engineering), not from laptops. Humans plan; pipelines apply.
• Destructive changes get explicit attention. Any plan showing destroy gets a named reviewer and a confirmation. For databases and stateful resources, prefer prevent_destroy:

resource "aws_db_instance" "main" {
  # ...
  lifecycle {
    prevent_destroy = true
  }
}

• -target is a scalpel for emergencies, not a workflow. Targeted applies create state that doesn't match the code; reconcile immediately after.
• moved and import blocks over terraform state mv/rm CLI surgery — the change is in code, reviewable, and repeatable.
• Removed-but-keep resources use the removed block (Terraform 1.7+) instead of state rm.

Drift Management

Drift — reality diverging from code — is guaranteed to happen. Managed, not prevented:

1. Scheduled detection: nightly CI job runs terraform plan -detailed-exitcode; exit code 2 = drift = alert.
2. Triage: is the drift a legitimate out-of-band change (console fix during an incident) or vandalism/error?
3. Reconcile deliberately: either import the change into code (adopt it) or re-apply the code (revert it). Decide — don't let it linger.
4. Emergency console changes are logged (see the non-negotiable rules) and reconciled into code within the same day.

Secrets in Terraform

• Terraform never stores secrets you give it — it reads them at apply time from the secrets manager (data sources: aws_secretsmanager_secret_version, Vault provider, etc.) and writes them to the managed service.
• State still contains sensitive values (resource attributes, generated passwords). That's why state is encrypted and access-controlled.
• Mark sensitive variables sensitive = true; still treat state as a secret.
• Prefer provider-generated secrets stored directly into a secrets manager (e.g., RDS manage_master_user_password = true) over generating in Terraform.

Common Rationalizations

Rationalization	Reality
"It's a tiny change, I'll just apply from my laptop"	Laptop applies bypass review, the lock discipline, and the audit trail — and they run with your credentials. Pipelines apply; humans review plans.
"The plan is 40 lines, I'll skim it"	The line you skimmed past was force_destroy = true on the bucket with the backups. Read every line; for big plans use terraform show -json and tooling.
"One big state is simpler"	Until a bad apply on a test VPC locks and corrupts the state that also owns production DNS. Small states = small blast radius.
"I'll fix the drift later"	Drift compounds. The next plan is now a diff of two changes: yours and the mystery one. Reconcile now.
"We can commit state, the repo is private"	State contains plaintext secrets. Private repos get cloned to laptops, exported, and leaked. Never.
"Modules are overkill here"	For one resource, yes. For the third copy of the same VPC-with-subnets-and-NAT pattern, the module is overdue. Module-ize repetition, not singularity.
"prevent_destroy gets in the way"	It's supposed to. It's the one thing standing between a typo and an unrecoverable database.

Red Flags

• terraform.tfstate anywhere in Git history
• apply run from a laptop against shared environments
• Plans reviewed by nobody, or "LGTM" without the plan output in the PR
• A single state file containing everything the company runs
• Hardcoded secrets in .tf files or terraform.tfvars in Git
• -target used routinely
• Drift alerts that fire nightly and are ignored nightly
• force_destroy = true / disabled deletion protection on stateful production resources
• Module sources pinned to main / unpinned

Verification

• terraform fmt, validate, tflint, and a config scanner (trivy/checkov) pass in CI
• State is remote, encrypted, locked, versioned — and not in Git (verified in .gitignore and history)
• The plan output is attached to the PR and reviewed by a human
• Apply runs in CI/CD with OIDC credentials, using the reviewed plan file
• Post-apply terraform plan returns "No changes" (or documented expected drift)
• Destructive changes were explicitly acknowledged; stateful resources have prevent_destroy
• Drift detection job exists and its alerts have an owner
• State is split by environment and component lifecycle — one bad apply can't touch unrelated infrastructure