Kubernetes application delivery

Kubernetes Application Delivery

Overview

Kubernetes is a powerful production platform and an excellent way to build an unmaintainable mess. The difference is discipline: every workload declares its resources, its health contract, and its security posture — no exceptions, or the scheduler and the network can't protect you.

Kubernetes is the last step of the journey, not the first. You should already know how to run the app on a Linux VM and in a container before orchestrating it here (see the learning path). K8s adds failure modes; it doesn't remove the need for fundamentals.

Declarative, or it didn't happen. kubectl apply from a laptop is for local clusters. Real clusters are reconciled from Git (see gitops-and-progressive-delivery).

When to Use

• Writing or reviewing Deployments, StatefulSets, Jobs, CronJobs, Services, Ingress
• Setting resource requests/limits and autoscaling
• Configuring probes, PodDisruptionBudgets, namespaces
• Writing RBAC, NetworkPolicies, security contexts
• Packaging with Helm or customizing with Kustomize
• Setting up ingress controllers, cert-manager, External Secrets Operator

The Production-Grade Workload Checklist

A workload is production-ready only when ALL of these are true:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: api
  namespace: shop                    # 1. Namespaced — never 'default'
  labels:
    app.kubernetes.io/name: api
    app.kubernetes.io/version: "1.4.2"
spec:
  replicas: 2                        # 2. More than one replica for anything user-facing
  revisionHistoryLimit: 5
  strategy:
    rollingUpdate:
      maxSurge: 1
      maxUnavailable: 0              # 3. Never drop capacity during rollout
  selector:
    matchLabels:
      app.kubernetes.io/name: api
  template:
    metadata:
      labels:
        app.kubernetes.io/name: api
        app.kubernetes.io/version: "1.4.2"
    spec:
      serviceAccountName: api        # 4. Dedicated SA, not 'default'
      securityContext:
        runAsNonRoot: true           # 5. Pod-level security
        runAsUser: 10001
        fsGroup: 10001
        seccompProfile:
          type: RuntimeDefault
      containers:
        - name: api
          image: ghcr.io/org/api@sha256:abc123...   # 6. Digest-pinned, never :latest
          ports:
            - containerPort: 8080
          resources:                 # 7. Requests AND limits — mandatory
            requests:
              cpu: 100m
              memory: 128Mi
            limits:
              cpu: 500m
              memory: 256Mi
          securityContext:
            allowPrivilegeEscalation: false
            readOnlyRootFilesystem: true
            capabilities:
              drop: ["ALL"]
          envFrom:
            - configMapRef:
                name: api-config     # 8. Config externalized
          env:
            - name: DB_PASSWORD      # 9. Secrets from a real source (ESO/Vault),
              valueFrom:             #    never literals, never base64-in-git
                secretKeyRef:
                  name: api-secrets
                  key: db-password
          startupProbe:              # 10. Startup probe protects slow starters
            httpGet: { path: /healthz, port: 8080 }
            failureThreshold: 30
            periodSeconds: 2
          readinessProbe:            # 11. Readiness: can I take traffic?
            httpGet: { path: /readyz, port: 8080 }
            periodSeconds: 5
          livenessProbe:             # 12. Liveness: am I deadlocked?
            httpGet: { path: /healthz, port: 8080 }
            periodSeconds: 10
          lifecycle:
            preStop:
              exec: { command: ["sh", "-c", "sleep 5"] }  # 13. Graceful shutdown
          volumeMounts:
            - name: tmp
              mountPath: /tmp
      volumes:
        - name: tmp
          emptyDir: {}               # writable scratch space for readOnlyRootFilesystem
      terminationGracePeriodSeconds: 30

Probes deserve their own paragraph:

• readinessProbe — removes the pod from Service endpoints when failing. This is your traffic gate. Failure here is normal (dependencies down, draining).
• livenessProbe — restarts the container when failing. Only for detecting deadlocks/unrecoverable states. A liveness probe that fails on dependency failure causes cascading restarts — never include dependencies in liveness.
• startupProbe — gives slow-starting apps time before liveness kills them. Without it, legacy apps enter CrashLoopBackOff.

Requests vs limits:

• requests = scheduling and QoS guarantee. Set from observed usage (metrics, not guesses).
• limits = eviction/kill thresholds. Memory limits cause OOMKills; CPU limits cause throttling. Missing memory limit on a leaky pod kills the node's other workloads.
• No requests = scheduler packs pods blindly = node overcommit = the 2 AM page.

Stateful and Batch Workloads

• StatefulSets for stable identity/storage (databases, queues) — but strongly consider managed databases instead (see cloud-platform-foundations). Running stateful systems on K8s well is expert-level work.
• Jobs/CronJobs: set activeDeadlineSeconds, backoffLimit, ttlSecondsAfterFinished, and concurrency policy (Forbid for non-reentrant jobs). A cronjob without a deadline is a zombie generator.

RBAC: Least Privilege in the Cluster

# Bind namespace-scoped Roles, not cluster-admin. Ever.
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: api
  namespace: shop
rules:
  - apiGroups: [""]
    resources: ["configmaps"]
    verbs: ["get", "list", "watch"]

Rules:

1. No cluster-admin bindings for workloads or CI. Cluster-admin in a pipeline token is a full-cluster compromise kit.
2. Every app gets its own ServiceAccount; automountServiceAccountToken: false unless it calls the API.
3. Human access via SSO-integrated groups, read-only by default, escalation logged.
4. CI/GitOps identities scoped to the namespaces they manage.

NetworkPolicies: Default-Deny East-West

# Start every namespace with default-deny, then allow explicitly
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: default-deny-all
  namespace: shop
spec:
  podSelector: {}
  policyTypes: [Ingress, Egress]
---
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: api-allow
  namespace: shop
spec:
  podSelector:
    matchLabels:
      app.kubernetes.io/name: api
  policyTypes: [Ingress, Egress]
  ingress:
    - from:
        - podSelector:
            matchLabels:
              app.kubernetes.io/name: frontend
      ports: [{ port: 8080 }]
  egress:
    - to:
        - namespaceSelector:
            matchLabels:
              kubernetes.io/metadata.name: kube-system
      ports: [{ port: 53, protocol: UDP }]   # DNS
    - to:
        - podSelector:
            matchLabels:
              app.kubernetes.io/name: postgres
      ports: [{ port: 5432 }]

Without NetworkPolicies, any compromised pod can reach every other pod and the metadata service. Flat networks are incident multipliers. (Requires a CNI that enforces them: Cilium, Calico, etc.)

Helm and Kustomize: The Two Packaging Patterns

Helm — package + parameterize (the app store model):

chart/
├── Chart.yaml          # name, version (chart), appVersion (the app)
├── values.yaml         # defaults — every knob documented
├── templates/
└── values-prod.yaml    # environment overlays (kept in your env repo)

Rules: pin chart versions (helm dependency update with a lockfile), template reviews include a rendered diff (helm template output in the PR), never helm upgrade prod from a laptop — the pipeline/GitOps controller does it.

Kustomize — patch a base per environment (the overlay model):

├── base/                 # the environment-agnostic manifests
│   ├── deployment.yaml
│   └── kustomization.yaml
└── overlays/
    ├── staging/          # replicas: 1, debug log level
    └── production/       # replicas: 3, PDB, strict policies

Choose one model per repo. Helm for third-party software distribution, Kustomize for first-party environment overlays is a common, sane split. Argo CD supports both natively.

The Standard Add-On Set (install in this order)

Add-on	Purpose	Without it
ingress-nginx / Traefik / Gateway API	HTTP(S) entry to the cluster	LoadBalancer per service = cost + sprawl
cert-manager	Automated TLS cert lifecycle	Manual cert renewals = expiry outages
external-secrets-operator	Sync secrets from Vault/cloud managers into K8s Secrets	Secrets base64'd in Git
metrics-server + Prometheus stack	Observability (see observability-and-alerting)	Flying blind
Kyverno / OPA Gatekeeper	Policy enforcement (non-root, required labels, digest-only images)	Every rule above is a polite request

Policy as code is how the checklist enforces itself. Example Kyverno rule: reject images not referenced by digest; reject pods without requests/limits; require runAsNonRoot. Audit mode first, enforce after the noise is understood.

Cluster Tiers (know which one you're in)

• Learning: K3s, Minikube, kind. Break things freely.
• Production: managed control plane (EKS/AKS/GKE), multiple nodes across AZs, etcd backups handled by the provider, upgrade channels planned.
• Production readiness is about the control plane's availability, upgrade strategy, and your ability to recover — not about whether pods start. Kubernetes' own docs draw this line sharply; respect it.

Common Rationalizations

Rationalization	Reality
"Limits are optional, requests are what matter"	A memory leak without limits takes down the node's other pods. Missing limits convert one bad workload into a platform incident.
"The app is fast, it doesn't need probes"	Probes aren't for the happy path — they're for the 3 AM deadlock and the dependency outage. Unprobed pods blackhole traffic silently.
":latest is fine, the tag didn't change"	:latest defeats rollbacks, digest verification, and the question "what is actually running." Pin by digest; let the GitOps controller roll forward deliberately.
"NetworkPolicies break things, skip them"	They break unwanted things — that's the point. Start default-deny with audit-friendly allows (DNS!), test in staging, then enforce.
"One ServiceAccount for all apps is simpler"	One compromised pod then owns every app's permissions. Per-app SAs cost nothing and bound the blast radius.
"Kubernetes for our 3-person team and 2 services"	Honest answer: probably not yet. A VM + Compose + good CI covers that scale. Adopt K8s when its problems (scheduling, self-healing, declarative ops at scale) are your problems.

Red Flags

• Workloads in the default namespace
• Containers running as root, privileged: true, or allowPrivilegeEscalation: true
• Images referenced by :latest or mutable tags in production namespaces
• No requests/limits anywhere (or limits copy-pasted at 10x reality)
• Secrets as plain env literals or base64-in-Git
• cluster-admin bound to CI service accounts
• No NetworkPolicies, flat cluster network
• Liveness probes that check database connectivity (cascading restart generator)
• kubectl apply history on production from personal laptops

Verification

• Every workload has requests and limits set from observed metrics
• Readiness, liveness, and startup probes defined; rollout verified with kubectl rollout status and zero dropped requests
• Pods run non-root with readOnlyRootFilesystem and dropped capabilities (policy engine enforces it)
• Images pinned by digest; deployments traceable to a Git commit
• Namespace has default-deny NetworkPolicy plus explicit allows
• RBAC: per-app ServiceAccounts, no cluster-admin anywhere outside platform components
• Secrets synced from an external manager (ESO) — nothing sensitive in Git
• PDBs protect multi-replica workloads during voluntary disruption (test: kubectl drain a node in staging)
• TLS automated via cert-manager; ingress reachable and redirecting to HTTPS
• All of the above reconciled from Git — kubectl diff against the repo shows no drift