Secrets management and identity

Secrets Management and Identity

Overview

Secrets are the most dangerous material you handle: invisible when managed well, catastrophic when leaked. The rules are simple and absolute: secrets are centralized, encrypted, short-lived where possible, rotated, and never — ever — in code.

Identity before secrets. The modern answer to "how does this workload prove who it is" is platform identity (OIDC, workload identity, managed identities) — not a stored credential at all. Every secret you can replace with a short-lived identity token is a secret that can't leak.

Base64 is encoding, not encryption. A base64'd secret in Git is a plaintext secret in Git.

When to Use

• Storing or retrieving any password, token, API key, or private key
• Deciding where a new secret lives
• Setting up OIDC / workload identity for CI or workloads
• Rotating or revoking credentials
• Responding to a suspected secret leak
• Reviewing .env handling, Kubernetes Secrets usage, or pipeline secrets

The Hierarchy of Secret Storage

BEST   │ No secret at all — platform identity (OIDC, managed identity, workload identity)
       │ Short-lived tokens, minted on demand, scoped, auto-expiring
───────┼────────────────────────────────────────────────────────────────────
GOOD   │ Central secrets manager — HashiCorp Vault, AWS Secrets Manager,
       │ Azure Key Vault, GCP Secret Manager
       │ Access via identity, dynamic secrets, rotation, audit log
───────┼────────────────────────────────────────────────────────────────────
OK     │ Encrypted-in-Git — SOPS (+ age/KMS), sealed-secrets, External Secrets
       │ Operator syncing from a manager into K8s Secrets
───────┼────────────────────────────────────────────────────────────────────
NEVER  │ Plaintext in code, config files, CI variables, Dockerfiles,
       │ base64 in Git, real .env files committed, screenshots, chat

Rule Zero: Secrets Never in Code

What "in code" means (all forbidden):

• Literals in source files, config files, or IaC (password = "hunter2")
• Real .env files committed to Git (only .env.example with dummy values)
• Kubernetes Secrets as base64 YAML in Git
• CI/CD YAML files (env: AWS_SECRET_ACCESS_KEY: AKIA...)
• Dockerfiles, docker-compose files, Makefile variables
• Git history at all — once committed, assume compromised even after deletion

Prevention layers, in order:

1. gitleaks (or trufflehog) in pre-commit hooks — catches it before the commit exists
2. Server-side secret scanning (GitHub/GitLab secret scanning + push protection) — catches what local hooks miss
3. Regular history scans — because hooks get bypassed (--no-verify exists)
4. A documented leak-response runbook — because prevention eventually fails

Leak Response (memorize the order)

1. REVOKE/ROTATE the credential FIRST. Not after the meeting. Now.
2. Assess blast radius: audit logs for the identity — what did it touch?
3. Purge from history (git filter-repo / BFG) — AFTER revocation, as cleanup.
4. Root cause: how did it get committed? Fix the process gap (hooks? training? tooling?)
5. Record the incident; feed the postmortem (blameless — see incident-response skill).

Deleting the commit without rotating is security theater: GitHub caches, forks, and clones have already seen it.

Platform Identity (the best secret is no secret)

CI/CD → Cloud: OIDC federation (detailed in ci-cd-pipeline-engineering). GitHub/GitLab mints a signed token; the cloud exchanges it for minutes-lived, narrowly-scoped credentials. No stored keys.

Kubernetes workloads → anything:

• Workload Identity (EKS IRSA, AKS Workload Identity, GKE Workload Identity): a K8s ServiceAccount maps to a cloud IAM role. The pod gets cloud credentials without any stored secret.
• Vault Agent / CSI driver: pods authenticate to Vault with their ServiceAccount; Vault renders secrets into the pod at runtime.
• External Secrets Operator: syncs secrets from Vault/cloud managers into K8s Secrets — control plane stays outside the cluster, rotation propagates automatically.

# ESO: Git holds a REFERENCE, the manager holds the VALUE
apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
  name: api-db
  namespace: shop
spec:
  refreshInterval: 1h
  secretStoreRef:
    name: vault
    kind: ClusterSecretStore
  target:
    name: api-secrets            # the K8s Secret ESO creates and maintains
  data:
    - secretKey: db-password
      remoteRef:
        key: shop/prod/api
        property: db_password

Secrets Manager Operations

Vault/cloud manager baseline rules:

1. Auth via identity only — no shared root tokens; humans via SSO, machines via platform auth (K8s SA, IAM, managed identity).
2. Dynamic secrets preferred — Vault database engines mint per-consumer credentials with TTLs; a leaked one expires and identifies its consumer.
3. Path layout mirrors blast radius: env/team/service — policies grant services exactly their path.
4. Audit log enabled and shipped to the security account/SIEM. Every secret read is attributable.
5. Rotation is scheduled and automated where the downstream supports it (databases, cloud keys). Rotation runbooks for the rest. Compromised or unused credentials revoked immediately, not at the next cycle.
6. Break-glass procedure documented — sealed Vault recovery, manager outage behavior (apps cache last-known; never fail-open to hardcoded fallbacks).

SOPS: Secrets in Git, Done Right

When Git-native secret storage is the right fit (small teams, GitOps flow):

# Encrypt with age or cloud KMS — keys in the KMS, not in the repo
sops --encrypt --age age1ql3z7hjy54pw3hyww5ayyfg7zqgvc7w3j2elw8zmr3052csa7qzs4qv86mz \
  secrets.prod.yaml > secrets.prod.enc.yaml

# Edit without manual decrypt/encrypt dance
sops secrets.prod.enc.yaml

Rules:

• Only the encrypted file is committed.
• Decryption keys live in KMS/age key files with 0600 perms, distributed via the secrets manager — chicken-and-egg resolved at the platform layer, never in the repo.
• .sops.yaml in the repo pins which files/keys — make encryption the default, not a remembered step.
• GitOps controllers (Flux/Argo CD with plugins) decrypt in-cluster; humans and CI decrypt via identity.

Least Privilege: Humans and Machines

1. Human access: SSO + MFA everywhere (Git, cloud, VPN, secrets manager, admin panels). Admin rights are separate accounts, used only for admin tasks, session-logged.
2. Machine access: one identity per service per environment. Never share a deploy key between repos/services — revocation becomes surgery.
3. Pipeline access: never a personal account. A pipeline running as "Dave (admin)" is Dave's credentials in every log, unrevocable when Dave leaves without breaking the pipeline.
4. Permissions from evidence: start minimal, expand from access-denied logs and access advisors. Never start broad "temporarily."
5. Regular access review: quarterly, who has what, who still needs it, what's unused for 90 days → revoked.

Common Rationalizations

Rationalization	Reality
"The repo is private, the secret is safe"	Private repos get cloned, forked, exported, and breached. Git hosting is not a secret store.
"I'll rotate it next sprint"	Rotation delayed is rotation forgotten. Compromised credentials rotate now; scheduled rotation is automated, not a ticket that ages.
"Base64 hides it well enough"	`echo <b64>
"OIDC is too much setup for this small project"	The setup is an hour. The leaked static key is an incident, a rotation, and an audit. OIDC is cheaper.
"We deleted the file from Git, we're clean"	History, forks, caches, and CI artifacts retain it. Revoke first, clean history second.
"One shared deploy key for all services is simpler"	One leak compromises everything, and rotation breaks everything at once. Per-service identities bound both risks.
"The vault adds latency; cache secrets in env vars at boot"	Long-lived env-var secrets defeat rotation. Use short TTLs with agent-side caching and re-render, or accept refresh-interval sync (ESO) — with rotation propagating automatically.

Red Flags

• gitleaks findings treated as false positives and dismissed
• .env files in Git history (check: git log --all --full-history -- .env)
• Base64 Kubernetes Secrets committed to repos
• CI pipelines authenticating with personal tokens or long-lived cloud keys
• A "shared" admin credential in a team password manager
• Secrets manager with no audit log configured
• Rotation documented as "when someone remembers"
• Production and development sharing any credential

Verification

• Zero secrets in code, config, CI YAML, Dockerfiles, or Git history — verified by gitleaks detect --redact on full history
• Pre-commit and server-side secret scanning active with push protection
• All machine-to-cloud auth via OIDC/workload identity — no static keys in CI
• Every secret lives in a manager (Vault/cloud) or is SOPS-encrypted with keys outside the repo
• Audit logging on for the secrets manager; every read attributable to an identity
• Rotation schedule exists and is automated for supported systems; leak-response runbook includes revoke-first ordering
• Humans: SSO + MFA everywhere; separate admin accounts; quarterly access review evidence
• Each service/environment has its own identity; revoking any one breaks exactly one thing
• Kubernetes secrets arrive via ESO/Vault/CSI — nothing sensitive in Git, even base64'd