Supply chain security

Supply Chain Security

Overview

Your production risk is not just your code — it's every dependency, base image, build tool, and pipeline step between a developer's keyboard and a running pod. Supply chain security answers three questions about every artifact: what's in it (SBOM), where did it come from (provenance/signing), and is it allowed to run (policy).

Scan at every phase, not once. Dependencies, source code, secrets, Dockerfiles, images, IaC, the cluster, and the running application each get their own scanner. A single scan at a single phase is a single point of failure.

Trust nothing mutable. Tags move, branches move, registries get compromised. Pin by SHA, verify signatures, and let policy engines enforce what humans forget.

When to Use

• Adding security scanning to a pipeline or registry
• Generating or consuming SBOMs
• Signing and verifying container images
• Reviewing dependency update automation (Renovate/Dependabot)
• Enforcing admission policies in Kubernetes
• Responding to a new CVE affecting your stack
• Auditing third-party actions, base images, or Helm charts

The Multi-Phase Scanning Matrix

Phase	What	Tool examples	Gate?
Pre-commit	Secrets, lint	gitleaks, hadolint, shellcheck	Yes (locally)
Dependency	Known CVEs in libraries	Trivy, Dependabot/Renovate alerts, pip-audit, npm audit	Yes
Source	SAST (injection, unsafe patterns)	Semgrep, CodeQL, SonarQube	Yes (tuned rules)
IaC	Misconfigured Terraform/K8s YAML	Trivy config, Checkov, tfsec	Yes
Image	OS + app CVEs, misconfig	Trivy image (before push), registry scanning	Yes, blocks push
Cluster	Running-workload CVEs, policy drift	Trivy Operator, admission policies	Continuous
Runtime	Anomalous behavior	Falco	Detect + alert
DAST	Running web app	OWASP ZAP (baseline scan)	Scheduled

Gates fail the pipeline on HIGH/CRITICAL findings; anything lower reports and trends. Tuning matters more than volume: a scanner that pages daily on noise gets muted, and then it's decorative.

SBOM: Know What You Ship

# Generate at build time, attach to the artifact
syft ghcr.io/org/api@sha256:abc123 -o spdx-json > sbom.spdx.json

# Scan the SBOM later — new CVEs get found against OLD artifacts
grype sbom:sbom.spdx.json

Rules:

1. Every published artifact has an SBOM (SPDX or CycloneDX), generated in CI, stored with the artifact (registry as OCI artifact, or release assets).
2. SBOMs answer "are we affected by CVE-2026-XXXX?" in seconds (query the SBOM inventory) instead of weeks (audit every service by hand).
3. Scan SBOMs continuously: a clean artifact today is a vulnerable artifact the day the next Log4Shell drops.

Signing and Provenance

# Keyless signing with Cosign (OIDC identity from CI — no keys to manage)
cosign sign ghcr.io/org/api@sha256:abc123

# SLSA-style provenance attestation: which repo, workflow, and commit built this
cosign attest --predicate provenance.json --type slsaprovenance ghcr.io/org/api@sha256:abc123

# Verify before deploy (and in admission control)
cosign verify ghcr.io/org/api@sha256:abc123 \
  --certificate-identity-regexp 'https://github.com/org/api/.github/workflows/ci.yml@.*' \
  --certificate-oidc-issuer https://token.actions.githubusercontent.com

The chain you're building: commit SHA → CI run (OIDC identity) → signed digest → admission controller verifies signature before the pod starts. An artifact that didn't come through YOUR pipeline cannot run in YOUR cluster. GitHub artifact attestations provide this with less wiring; Cosign gives you vendor-neutral control.

Dependency Update Automation

Updates are security work, but unreviewed auto-updates are supply-chain roulette:

# renovate.json — controlled automation
{
  "extends": ["config:recommended"],
  "schedule": ["before 6am on Monday"],
  "automerge": false,
  "vulnerabilityAlerts": { "schedule": ["at any time"] },
  "packageRules": [
    { "matchUpdateTypes": ["patch"], "automerge": true, "automergeType": "pr" },
    { "matchUpdateTypes": ["major"], "reviewers": ["team:platform"], "minimumReleaseAge": 7 }
  ]
}

Rules:

• Security patches fast-track (immediate PRs), with CI as the gate.
• Major updates wait (minimumReleaseAge lets the ecosystem canary fresh releases — many supply-chain attacks are caught within days of publication).
• Pin Docker base images and GitHub Actions by digest/SHA — Renovate updates those too, so pinning doesn't mean fossilizing.
• Lockfiles committed, CI installs from lockfile (npm ci, pip install --require-hashes).

Policy as Code (admission enforcement)

Policies make the other skills' rules self-enforcing. Kyverno example:

apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: require-signed-images
spec:
  validationFailureAction: Enforce     # start as Audit, promote to Enforce
  background: false
  rules:
    - name: verify-cosign-signature
      match:
        any:
          - resources:
              kinds: [Pod]
      verifyImages:
        - imageReferences: ["ghcr.io/org/*"]
          attestors:
            - entries:
                - keyless:
                    subject: "https://github.com/org/*"
                    issuer: "https://token.actions.githubusercontent.com"

The minimum policy set (Audit → Enforce):

1. Images only from approved registries, referenced by digest
2. No privileged containers, no root, no allowPrivilegeEscalation
3. Required labels (owner, env) and resources (requests/limits)
4. No :latest, no default namespace, no hostPath mounts
5. Signed images for Tier-1 workloads

Always roll out in Audit mode first, read the violations, fix legit workloads, then enforce. Enforcing blind is how you break production with a YAML file.

Runtime Detection (Falco)

Prevention fails; detection is the backstop. Falco watches syscalls and K8s audit events for the classics: shell spawned in a container, unexpected outbound connections, package tools running in prod images, sensitive file access, crypto miners. Rules tuned to your baseline; alerts routed to the on-call path (see observability-and-alerting).

CVE Response Workflow

New critical CVE published
   │
   ▼
1. Are we affected?  → query SBOM inventory (not Slack archaeology)
   │
   ├── No  → record the check, done
   └── Yes →
2. Is it reachable/exploitable in OUR usage? (VEX, internet-facing? data path?)
3. Patch: update dependency/base image → pipeline rebuilds, scans, signs → promote digest
4. Verify: scanner clean on the new digest; admission policy sees new signature
5. Record: affected versions, fix version, time-to-remediate (feeds metrics)

Common Rationalizations

Rationalization	Reality
"Scanners are too noisy to gate on"	Tune the rules; don't drop the gate. Gate on HIGH/CRITICAL, report the rest. A muted scanner protects nothing.
"We scanned at build time, we're covered"	New CVEs are published daily against old artifacts. Re-scan the registry/SBOMs continuously.
"Pinning by SHA makes updates hard"	That's what Renovate is for — it bumps SHAs and digests with PRs you can review. Mutable references are unpatchable and unauditable.
"Signing images is over-engineering for us"	The first time a tampered or mistaken image tries to run and the cluster refuses it, it pays for itself. Keyless Cosign is an afternoon of setup.
"Auto-merging all updates saves time"	A compromised package version with a postinstall script auto-merged to main is the canonical supply-chain attack. Patches auto-merge; majors get eyes and age.
"Policies in Enforce from day one"	Enforce-blind breaks workloads and teaches the team to hate the policy engine. Audit first, fix, then enforce.

Red Flags

• No SBOM for production artifacts (can't answer "are we affected?" in minutes)
• Images deployed with no signature or provenance
• Scanners present but configured to never fail the build
• Dependencies pinned by mutable tags ("v4", "latest", "main")
• Secrets detected in history scans, treated as routine
• Auto-merge on all dependency updates including majors
• Admission policies stuck in Audit forever (or absent)
• No one owns CVE response; "we patch when we get to it"

Verification

• Every phase of the scanning matrix runs and gates: pre-commit, dependency, SAST, IaC, image, cluster, runtime, DAST
• Every published artifact has an SBOM stored alongside it; SBOM re-scanning is scheduled
• Images signed (Cosign keyless or attestations); verification enforced at admission for production
• Third-party actions, base images, and charts pinned by SHA/digest with automated update PRs
• Dependency updates automated with security fast-track and major-update review
• Policy engine enforcing (after Audit phase): approved registries, digest-only, non-root, resources required
• Runtime detection (Falco or equivalent) alerts to on-call with a documented triage path
• A recent CVE drill: from announcement to "we are/aren't affected" answer in under a day, using the SBOM inventory