Kubernetes operations

Kubernetes Operations — Agent Skill

Production-grade Kubernetes operations for AI agents. Generates secure, scalable, multi-cloud K8s manifests and Helm charts with built-in failure-mode prevention, compliance validation, and structured output contracts.

---

Core Workflow — 7-Step Failure-Mode Prevention

Every K8s response MUST follow this sequence. Skip a step → risk a failure mode.

Context → Diagnose → Reference → Design → Validate → Output → Rollback

Step	Action	Deliverable
1. Context	Identify target platform (EKS/GKE/AKS/OpenShift/k3s), cluster version, namespace, existing workloads, constraints	Context summary
2. Diagnose	Check for failure-mode exposures in the ask (see §Failure Modes)	Risk assessment
3. Reference	Pull current API versions, platform-specific defaults, security standards (PSS, CIS, NSA/CISA)	Reference baseline
4. Design	Build manifest/chart/policy with least-privilege, resource bounds, network isolation	Artifact draft
5. Validate	Run mental validation against all 8 failure modes, check API deprecations, verify security context	Validation pass/fail
6. Output	Emit structured output with assumptions, tradeoffs, and rollback instructions per §Output Contract	Final artifact + contract
7. Rollback	Provide kubectl delete / helm uninstall / kubectl rollout undo instructions	Rollback plan

---

Failure Modes — The 8 Ways K8s Goes Wrong

These are the failure modes this skill prevents. Every manifest review and generation MUST check all eight.

FM-1: Insecure Workloads

Symptom: Container running as root, privileged mode, hostPath mounts, no securityContext. Prevention: runAsNonRoot: true, readOnlyRootFilesystem: true, drop ALL capabilities, add only required ones. Detection: kubectl get pods -o json | jq '.items[].spec.containers[].securityContext'

FM-2: Resource Starvation

Symptom: No requests/limits, unbounded memory growth, CPU throttling, OOMKilled. Prevention: Always set requests = limits for Guaranteed QoS on critical workloads. Use LimitRange at namespace level. Detection: kubectl top pods --namespace=<ns> and check for OOMKilled in pod status.

FM-3: Network Exposure

Symptom: Missing NetworkPolicies, services exposed as LoadBalancer unnecessarily, no TLS termination. Prevention: Deny-all ingress by default, explicit NetworkPolicy allowlists, TLS via cert-manager. Detection: kubectl get netpol --all-namespaces and kubectl get svc --all-namespaces | grep LoadBalancer

FM-4: Privilege Sprawl

Symptom: ClusterRoleBindings to cluster-admin, overly broad RBAC, service accounts with secrets access. Prevention: Least-privilege RBAC, per-namespace Roles, service account token audiences restricted. Detection: kubectl get clusterrolebindings -o json | jq '.items[] | select(.roleRef.name=="cluster-admin")'

FM-5: Fragile Rollouts

Symptom: No health probes, no PDB, rolling update with single replica, no revision history. Prevention: Readiness + liveness probes, PodDisruptionBudget, revisionHistoryLimit: 10, minReadySeconds. Detection: kubectl get deployment -o json | jq '.items[] | select(.spec.replicas==1 and .spec.strategy.type=="RollingUpdate")'

FM-6: API Drift

Symptom: Deprecated extensions/v1beta1, apps/v1beta2, policy/v1beta1 PDB. Prevention: Always check kubectl api-resources for current API version. Use apps/v1 for Deployments, policy/v1 for PDB. Detection: kubectl get --raw /apis | jq -r '.groups[].preferredVersion.groupVersion'

FM-7: GitOps Divergence — NEW

Symptom: Manual kubectl apply bypasses GitOps pipeline, cluster state drifts from Git, unrecorded changes. Prevention: All changes through Git → Flux/ArgoCD reconciliation. kubectl for read-only and emergencies only. Detection: flux get kustomizations -A or argocd app diff <app> to detect drift.

FM-8: Multi-Cloud Skew — NEW

Symptom: EKS-specific annotations on GKE, AKS ingress class mismatch, OpenShift SCC violations. Prevention: Platform-conditional manifest generation. Check cloud provider before emitting manifests. Detection: Validate against provider-specific kubectl api-resources and admission webhooks.

---

Platform Guidance

Amazon EKS

• Ingress: AWS Load Balancer Controller (kubernetes.io/ingress.class: alb)
• Storage: EBS CSI driver (gp3 default)
• Auth: IAM Roles for Service Accounts (IRSA) — eks.amazonaws.com/role-arn annotation
• Networking: VPC CNI, security groups for pods
• Patching: Bottlerocket OS recommended

Google GKE

• Ingress: GKE Ingress Controller / Gateway API (networking.gke.io/managed-certificates)
• Storage: Compute Engine persistent disk CSI (pd-standard, pd-ssd)
• Auth: Workload Identity Federation (iam.gke.io/gcp-service-account annotation)
• Networking: Dataplane V2, NetworkPolicy logging via Cloud Logging
• Patching: GKE Autopilot for managed nodes

Azure AKS

• Ingress: Application Gateway Ingress Controller (AGIC) or NGINX
• Storage: Azure Disk CSI / Azure Files CSI
• Auth: Microsoft Entra Workload ID (azure.workload.identity/use: "true")
• Networking: Azure CNI with NetworkPolicy via Calico
• Patching: AKS node image auto-upgrade

Red Hat OpenShift

• Ingress: OpenShift Router (HAProxy-based, route.openshift.io)
• Security: Security Context Constraints (SCC) — restricted-v2 as default, drop ALL caps mandatory
• Storage: OpenShift Data Foundation / any CSI
• Auth: OAuth integrated, no IRSA equivalent — use serviceAccount annotations
• Patching: Cluster Version Operator (CVO) managed

---

Security

Pod Security Standards (PSS)

restricted (default recommended)
  - runAsNonRoot: true
  - seccompProfile: RuntimeDefault
  - capabilities.drop: ["ALL"]
  - readOnlyRootFilesystem: true (where feasible)
  - allowPrivilegeEscalation: false

Namespace labeling:

apiVersion: v1
kind: Namespace
metadata:
  name: production
  labels:
    pod-security.kubernetes.io/enforce: restricted
    pod-security.kubernetes.io/enforce-version: latest
    pod-security.kubernetes.io/audit: restricted
    pod-security.kubernetes.io/warn: restricted

RBAC Least-Privilege Patterns

apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  namespace: app-ns
  name: app-deployer
rules:
- apiGroups: ["apps", ""]
  resources: ["deployments", "pods", "services", "configmaps"]
  verbs: ["get", "list", "watch", "create", "update", "patch"]
- apiGroups: [""]
  resources: ["pods/log"]
  verbs: ["get", "list"]
# NO delete, NO secrets access, NO cluster-wide verbs

NetworkPolicy — Deny-All + Allowlist

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: default-deny-all
spec:
  podSelector: {}
  policyTypes:
  - Ingress
  - Egress

OWASP Kubernetes Top 10 (Rapid Reference)

1. Insecure workload configurations → FM-1
2. Supply chain vulnerabilities → Image scanning (Trivy/Grype)
3. Overly permissive RBAC → FM-4
4. Lack of centralized policy enforcement → OPA/Gatekeeper, Kyverno
5. Inadequate logging & monitoring → FM-8 Observability
6. Broken authentication → Service account token hygiene
7. Missing network segmentation → FM-3
8. Secrets management failures → §Secret Management
9. Misconfigured cluster components → CIS benchmarks
10. Outdated/vulnerable K8s components → FM-6

NSA/CISA Kubernetes Hardening Guidance

• Scan container images for vulnerabilities
• Run containers as non-root
• Use network segmentation (NetworkPolicies)
• Encrypt secrets at rest (etcd encryption)
• Enable audit logging
• Use service mesh for mTLS (Istio/Linkerd)
• Regularly update cluster components

CIS Kubernetes Benchmarks

• 1.1: API server — AlwaysPullImages admission plugin
• 1.2: Scheduler — profiling disabled
• 4.1: Worker node — kubelet --protect-kernel-defaults=true
• 5.1: RBAC — minimize cluster-admin bindings
• 5.2: Pod Security Standards — restricted enforcement
• 5.3: Network Policies — namespaced default deny

---

Helm Chart Generation & Review

Chart Structure

mychart/
├── Chart.yaml          # name, version, appVersion, dependencies
├── values.yaml         # Default values with documentation
├── values/             # Multi-environment overrides (values-prod.yaml, values-staging.yaml)
├── templates/
│   ├── _helpers.tpl    # Reusable template functions
│   ├── deployment.yaml
│   ├── service.yaml
│   ├── ingress.yaml
│   ├── hpa.yaml
│   ├── pdb.yaml
│   ├── serviceaccount.yaml
│   ├── networkpolicy.yaml
│   └── NOTES.txt       # Post-install instructions
├── templates/tests/    # Helm test pod definitions
├── crds/               # Custom Resource Definitions
└── README.md

values.yaml Patterns

# ALWAYS document every value
replicaCount: 3

image:
  repository: nginx
  tag: "1.25"
  pullPolicy: IfNotPresent
  # pullSecrets for private registries
  pullSecrets: []

# Security context as a named block — reusable
securityContext:
  runAsNonRoot: true
  runAsUser: 1000
  runAsGroup: 1000
  fsGroup: 1000
  seccompProfile:
    type: RuntimeDefault
  capabilities:
    drop: ["ALL"]

# Resource defaults — MUST be set
resources:
  requests:
    cpu: 100m
    memory: 128Mi
  limits:
    cpu: 500m
    memory: 256Mi

# Autoscaling
autoscaling:
  enabled: true
  minReplicas: 3
  maxReplicas: 10
  targetCPUUtilizationPercentage: 80
  targetMemoryUtilizationPercentage: 80

# Probes with sensible defaults
probes:
  liveness:
    httpGet:
      path: /healthz
      port: 8080
    initialDelaySeconds: 30
    periodSeconds: 10
  readiness:
    httpGet:
      path: /ready
      port: 8080
    initialDelaySeconds: 5
    periodSeconds: 5

# Pod disruption budget
pdb:
  enabled: true
  minAvailable: 1

# Ingress with TLS
ingress:
  enabled: true
  className: nginx
  annotations:
    cert-manager.io/cluster-issuer: letsencrypt-prod
  hosts:
    - host: app.example.com
      paths:
        - path: /
          pathType: Prefix
  tls:
    - secretName: app-tls
      hosts:
        - app.example.com

# Network policy
networkPolicy:
  enabled: true
  ingressAllow:
    - namespaceSelector:
        matchLabels:
          kubernetes.io/metadata.name: ingress-nginx
  egressAllow:
    - to:
        - namespaceSelector: {}
          podSelector:
            matchLabels:
              app: database
      ports:
        - port: 5432
          protocol: TCP

# Service account with annotations for cloud IAM
serviceAccount:
  create: true
  annotations:
    eks.amazonaws.com/role-arn: "arn:aws:iam::123456789012:role/app-role"  # EKS IRSA
    # iam.gke.io/gcp-service-account: "[email protected]"  # GKE WI
    # azure.workload.identity/use: "true"  # AKS WI

# Tolerations & node affinity
tolerations: []
affinity:
  podAntiAffinity:
    preferredDuringSchedulingIgnoredDuringExecution:
      - weight: 100
        podAffinityTerm:
          labelSelector:
            matchExpressions:
              - key: app.kubernetes.io/name
                operator: In
                values: ["myapp"]
          topologyKey: kubernetes.io/hostname

# Topology spread constraints — zone-level HA
topologySpreadConstraints:
  - maxSkew: 1
    topologyKey: topology.kubernetes.io/zone
    whenUnsatisfiable: ScheduleAnyway
    labelSelector:
      matchLabels:
        app.kubernetes.io/name: myapp

nodeSelector: {}
# platform-specific:
# EKS: nodeSelector: { "eks.amazonaws.com/capacityType": "ON_DEMAND" }
# GKE: nodeSelector: { "cloud.google.com/gke-nodepool": "default-pool" }

Dependency Management

# Chart.yaml
dependencies:
  - name: redis
    version: "18.x.x"
    repository: "https://charts.bitnami.com/bitnami"
    condition: redis.enabled
  - name: postgresql
    version: "15.x.x"
    repository: "https://charts.bitnami.com/bitnami"
    condition: postgresql.enabled

Multi-environment deployment:

# Staging
helm upgrade --install myapp ./mychart -f values.yaml -f values/staging.yaml -n staging

# Production
helm upgrade --install myapp ./mychart -f values.yaml -f values/prod.yaml -n production

---

GitOps Integration

Flux CD Pattern

apiVersion: source.toolkit.fluxcd.io/v1
kind: GitRepository
metadata:
  name: myapp
  namespace: flux-system
spec:
  interval: 1m
  url: https://github.com/org/myapp-deploy
  ref:
    branch: main
---
apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata:
  name: myapp
  namespace: flux-system
spec:
  interval: 5m
  path: ./overlays/production
  prune: true
  sourceRef:
    kind: GitRepository
    name: myapp
  healthChecks:
    - apiVersion: apps/v1
      kind: Deployment
      name: myapp
      namespace: production
  postBuild:
    substitute:
      ENV: production

ArgoCD Pattern

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: myapp
  namespace: argocd
spec:
  project: default
  source:
    repoURL: https://github.com/org/myapp-deploy
    targetRevision: main
    path: overlays/production
  destination:
    server: https://kubernetes.default.svc
    namespace: production
  syncPolicy:
    automated:
      prune: true
      selfHeal: true
    syncOptions:
      - CreateNamespace=true
    retry:
      limit: 5
      backoff:
        duration: 30s
        factor: 2
        maxDuration: 5m

---

Resource Management

Requests & Limits

Workload Type	CPU Request	CPU Limit	Memory Request	Memory Limit
Burstable API	100m	500m	128Mi	256Mi
Guaranteed DB	500m	500m	2Gi	2Gi
Background Job	50m	200m	64Mi	128Mi

# Pod with Guaranteed QoS (requests == limits)
resources:
  requests:
    cpu: "500m"
    memory: "2Gi"
  limits:
    cpu: "500m"
    memory: "2Gi"

HPA (Horizontal Pod Autoscaler)

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: myapp-hpa
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: myapp
  minReplicas: 3
  maxReplicas: 20
  metrics:
  - type: Resource
    resource:
      name: cpu
      target:
        type: Utilization
        averageUtilization: 70
  - type: Resource
    resource:
      name: memory
      target:
        type: Utilization
        averageUtilization: 80
  behavior:
    scaleDown:
      stabilizationWindowSeconds: 300
      policies:
      - type: Percent
        value: 50
        periodSeconds: 60
    scaleUp:
      stabilizationWindowSeconds: 0
      policies:
      - type: Percent
        value: 100
        periodSeconds: 15

VPA vs HPA Decision Matrix

• HPA: Stateless workloads, request-driven scaling, predictable patterns
• VPA: Stateful workloads, right-sizing after profiling, JVM heap tuning
• Both: Use VPA in recommendation mode (updateMode: "Off") with HPA for execution

Priority Classes

apiVersion: scheduling.k8s.io/v1
kind: PriorityClass
metadata:
  name: high-priority
value: 1000000
globalDefault: false
preemptionPolicy: PreemptLowerPriority
description: "Critical production workloads"
---
# System-critical (never evicted): 2000000000
# Production high: 1000000
# Production default: 100000
# Batch / non-critical: -1

---

Cost Optimization

Right-Sizing

• Use kubectl top pods + VPA recommender to profile actual usage
• Set requests at P95 of observed usage, not guesstimates
• Run kubecost or opencost for namespace-level cost allocation

Spot Instances

# EKS
spec:
  nodeSelector:
    eks.amazonaws.com/capacityType: SPOT
  tolerations:
  - key: "eks.amazonaws.com/capacityType"
    operator: "Equal"
    value: "SPOT"
    effect: "NoSchedule"

# GKE
spec:
  nodeSelector:
    cloud.google.com/gke-spot: "true"
  tolerations:
  - key: "cloud.google.com/gke-spot"
    operator: "Equal"
    value: "true"
    effect: "NoSchedule"

# AKS
spec:
  nodeSelector:
    kubernetes.azure.com/scalesetpriority: spot
  tolerations:
  - key: "kubernetes.azure.com/scalesetpriority"
    operator: "Equal"
    value: "spot"
    effect: "NoSchedule"

Cluster Autoscaler

• Set min/max on node groups, not on individual deployments
• Use cluster-autoscaler.kubernetes.io/safe-to-evict: "true" on pods that can move
• Apply PDBs so autoscaler respects availability during scale-down

Workload Rightsizing Rules

1. Never run without limits. Use LimitRange to enforce defaults
2. Burstable for dev/staging. Guaranteed for production databases and stateful sets
3. Preemptible nodes for batch. Tolerations + nodeSelectors
4. HPA over static replicas. Let metrics drive scaling decisions

---

Observability

Prometheus Metrics

# Pod annotations for Prometheus scraping
metadata:
  annotations:
    prometheus.io/scrape: "true"
    prometheus.io/port: "8080"
    prometheus.io/path: "/metrics"

# ServiceMonitor (Prometheus Operator)
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: myapp
spec:
  selector:
    matchLabels:
      app: myapp
  endpoints:
  - port: metrics
    interval: 30s
    path: /metrics

Structured Logging

# Sidecar or daemonset pattern — emit JSON to stdout
# Containers should log to stdout/stderr in JSON format:
# {"level":"info","ts":"2026-06-18T02:06:00Z","msg":"request","method":"GET","path":"/api","duration_ms":42,"status":200}

# Use Fluent Bit or Vector as DaemonSet for log collection:
apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: fluent-bit
  namespace: logging
spec:
  selector:
    matchLabels:
      app: fluent-bit
  template:
    metadata:
      labels:
        app: fluent-bit
    spec:
      serviceAccountName: fluent-bit
      containers:
      - name: fluent-bit
        image: fluent/fluent-bit:3.1
        volumeMounts:
        - name: varlog
          mountPath: /var/log
        - name: varlibdockercontainers
          mountPath: /var/lib/docker/containers
          readOnly: true

OpenTelemetry

# Instrument with OTel SDK, configure via OTEL_EXPORTER_OTLP_ENDPOINT
env:
- name: OTEL_EXPORTER_OTLP_ENDPOINT
  value: "http://otel-collector.observability:4317"
- name: OTEL_SERVICE_NAME
  value: "myapp"
- name: OTEL_RESOURCE_ATTRIBUTES
  value: "deployment.environment=production,cloud.provider=aws"

# OpenTelemetry Collector — sidecar pattern
# OR use the OpenTelemetry Operator for auto-instrumentation

Golden Signals Dashboard (Grafana / Datadog / New Relic)

• Latency: P50, P95, P99 of request duration
• Traffic: Requests per second
• Errors: 5xx rate + error budget burn rate
• Saturation: CPU throttle %, memory pressure, goroutine count

Alert Rules (Prometheus)

groups:
- name: app
  rules:
  - alert: HighErrorRate
    expr: rate(http_requests_total{status=~"5.."}[5m]) / rate(http_requests_total[5m]) > 0.01
    for: 5m
    labels:
      severity: critical
    annotations:
      summary: "Error rate > 1% for {{ $labels.app }}"
  - alert: PodRestarting
    expr: rate(kube_pod_container_status_restarts_total[15m]) > 0
    for: 5m
    labels:
      severity: warning

---

Secret Management

External Secrets Operator (ESO)

apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
  name: myapp-secrets
spec:
  refreshInterval: 1h
  secretStoreRef:
    name: aws-secretsmanager  # or gcp-secretmanager, azure-keyvault
    kind: ClusterSecretStore
  target:
    name: myapp-secrets
    creationPolicy: Owner
  data:
  - secretKey: DATABASE_URL
    remoteRef:
      key: prod/myapp/database-url
  - secretKey: API_KEY
    remoteRef:
      key: prod/myapp/api-key

Sealed Secrets

# Encrypt a secret for GitOps
kubectl create secret generic mysecret --from-literal=password=s3cret --dry-run=client -o yaml \
  | kubeseal --controller-namespace kube-system --format yaml > sealed-secret.yaml

apiVersion: bitnami.com/v1alpha1
kind: SealedSecret
metadata:
  name: mysecret
  namespace: production
spec:
  encryptedData:
    password: AgBy8hCKF...encrypted_base64...

Secret Management Decision Tree

Scenario	Solution
Cloud-native, existing secrets manager	External Secrets Operator
GitOps, no external dependency	Sealed Secrets
Dynamic secrets (DB creds, PKI)	Vault + Vault CSI Provider
Simple, single-cluster	Kubernetes Secrets + etcd encryption
Multi-cluster, audit trail	Vault with Kubernetes auth method

---

Multi-Cloud Deployment Patterns

Conditional Manifests (Helm)

# templates/ingress.yaml
{{- if eq .Values.cloudProvider "aws" }}
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  annotations:
    alb.ingress.kubernetes.io/scheme: internet-facing
    alb.ingress.kubernetes.io/target-type: ip
spec:
  ingressClassName: alb
  # ...
{{- else if eq .Values.cloudProvider "gcp" }}
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  annotations:
    kubernetes.io/ingress.class: gce
    networking.gke.io/managed-certificates: myapp-cert
  # ...
{{- end }}

Node Selectors & Taints/Tolerations

# EKS — Spot node pool
nodeSelector:
  eks.amazonaws.com/capacityType: SPOT
tolerations:
- key: "eks.amazonaws.com/capacityType"
  operator: "Equal"
  value: "SPOT"
  effect: "NoSchedule"

# GKE — GPU node pool
nodeSelector:
  cloud.google.com/gke-accelerator: nvidia-l4
tolerations:
- key: "nvidia.com/gpu"
  operator: "Exists"
  effect: "NoSchedule"

# AKS — Memory-optimized node pool
nodeSelector:
  agentpool: memoryoptimized

Cloud-Agnostic Abstraction

# Use Kubernetes-native primitives, not cloud-specific ones
# Good: Ingress + cert-manager (works everywhere with proper controller)
# Bad: Cloud-specific LoadBalancer annotations hardcoded

# For cloud-specific features, use Helm template functions:
{{- define "app.cloudConfig" -}}
{{- if eq .Values.cloudProvider "aws" }}
serviceAccountAnnotations:
  eks.amazonaws.com/role-arn: {{ .Values.aws.roleArn }}
{{- else if eq .Values.cloudProvider "gcp" }}
serviceAccountAnnotations:
  iam.gke.io/gcp-service-account: {{ .Values.gcp.serviceAccount }}
{{- else if eq .Values.cloudProvider "azure" }}
podLabels:
  azure.workload.identity/use: "true"
{{- end }}
{{- end }}

---

Output Contract

Every response that produces K8s artifacts MUST include:

Required Sections

1. Assumptions: What you assumed about the environment
2. Tradeoffs: What you chose and why (e.g., Guaranteed vs Burstable QoS)
3. Rollback Instructions: Exact kubectl delete / helm uninstall / kubectl rollout undo commands

### Assumptions
- Target cluster: EKS 1.30, namespace: `production`
- You have cert-manager and AWS Load Balancer Controller installed
- IRSA is configured for service account IAM roles

### Tradeoffs
- Chose Guaranteed QoS for the database pod (slight over-provision, but predictable performance)
- Used Burstable QoS for the API (cost-effective, acceptable for stateless workloads)
- Disabled privilege escalation even though it breaks some debugging tools — security over convenience

### Rollback
\`\`\`bash
kubectl delete -f manifest.yaml
# OR for Helm:
helm uninstall myapp -n production
# OR to undo last rollout:
kubectl rollout undo deployment/myapp -n production
\`\`\`

---

DO / DON'T

✅ DO

# DO: Complete security context
securityContext:
  runAsNonRoot: true
  runAsUser: 1000
  seccompProfile:
    type: RuntimeDefault
  capabilities:
    drop: ["ALL"]
  readOnlyRootFilesystem: true
  allowPrivilegeEscalation: false

# DO: Both probes
livenessProbe:
  httpGet:
    path: /healthz
    port: 8080
readinessProbe:
  httpGet:
    path: /ready
    port: 8080

# DO: Resource requests AND limits
resources:
  requests:
    cpu: 100m
    memory: 128Mi
  limits:
    cpu: 500m
    memory: 256Mi

# DO: Namespace labels for PSS
metadata:
  labels:
    pod-security.kubernetes.io/enforce: restricted

# DO: Use current API versions
apiVersion: apps/v1
apiVersion: networking.k8s.io/v1
apiVersion: policy/v1

# DO: Pod anti-affinity for HA
affinity:
  podAntiAffinity:
    requiredDuringSchedulingIgnoredDuringExecution:
    - labelSelector:
        matchLabels:
          app: myapp
      topologyKey: kubernetes.io/hostname

# DO: Topology spread for zone HA
topologySpreadConstraints:
- maxSkew: 1
  topologyKey: topology.kubernetes.io/zone
  whenUnsatisfiable: ScheduleAnyway

# DO: Provide PDB
apiVersion: policy/v1
kind: PodDisruptionBudget
spec:
  minAvailable: 1

❌ DON'T

# DON'T: No security context
spec:
  containers:
  - name: app
    image: myapp:latest
    # securityContext is MISSING — FM-1

# DON'T: Running as root
securityContext:
  runAsUser: 0  # NO — FM-1

# DON'T: Privileged container
securityContext:
  privileged: true  # NO — FM-1, NEVER in production

# DON'T: hostPath mounts without extreme caution
volumes:
- name: dangerous
  hostPath:
    path: /var/run/docker.sock  # NO — FM-1, container escape risk

# DON'T: No resource limits
resources: {}  # NO — FM-2

# DON'T: No health probes — FM-5
# No livenessProbe or readinessProbe defined

# DON'T: Deprecated API versions — FM-6
apiVersion: extensions/v1beta1  # DEPRECATED since 1.16

# DON'T: Single replica without PDB — FM-5
spec:
  replicas: 1
  # No PodDisruptionBudget defined

# DON'T: Overly broad RBAC — FM-4
rules:
- apiGroups: ["*"]
  resources: ["*"]
  verbs: ["*"]  # NO — just use cluster-admin if you need this

# DON'T: No NetworkPolicy — FM-3
# Default deny NetworkPolicy must exist in every namespace

# DON'T: Hardcoded cloud provider specifics — FM-8
annotations:
  eks.amazonaws.com/role-arn: "..."  # OK if target is known EKS, NOT OK as generic manifest

# DON'T: kubectl apply for GitOps-managed resources — FM-7
# Use Git → Flux/ArgoCD path instead

---

Quick Reference Cards

Minimal Production-Ready Deployment Checklist

• securityContext with runAsNonRoot: true, allowPrivilegeEscalation: false, capabilities drop ALL
• readinessProbe AND livenessProbe defined
• resources.requests AND resources.limits set for all containers
• HPA with minReplicas >= 2 for stateless workloads
• PodAntiAffinity or TopologySpreadConstraints for HA
• PodDisruptionBudget with minAvailable: 1 or maxUnavailable: 1
• ServiceAccount with cloud IAM annotations (if cloud managed)
• NetworkPolicy — deny-all + explicit allowlist
• PSS namespace labels (enforce: restricted)
• No deprecated API versions (kubectl api-resources verified)
• Secrets externalized (ESO/SealedSecrets/Vault), NOT in plain ConfigMap
• Prometheus scrape annotations or ServiceMonitor

Troubleshooting Quick Commands

# Pod won't start
kubectl describe pod <pod> -n <ns>
kubectl logs <pod> -n <ns> --previous

# Resource issues
kubectl top pods -n <ns>
kubectl get events -n <ns> --sort-by='.lastTimestamp'

# RBAC issues
kubectl auth can-i --list --as=system:serviceaccount:<ns>:<sa> -n <ns>

# Network issues
kubectl run tmp --rm -it --image=nicolaka/netshoot -n <ns> -- /bin/bash
# Then: curl, nc, dig, tcpdump from inside cluster

# Drift detection (Flux)
flux get kustomizations -A --status-selector ready=false

# Drift detection (ArgoCD)
argocd app diff <app>

# API version check
kubectl api-resources --verbs=list -o wide
kubectl explain deployment --api-version=apps/v1

---

Evaluation

See evals/eval_cases.json for trigger match test cases and near-miss negatives.

References

• references/k8s-security-hardening.md — Full security reference: PSS, RBAC, NetworkPolicies, OWASP Top 10
• references/k8s-failure-modes.md — All 8 failure modes with detection and remediation
• references/helm-patterns.md — Helm chart architecture, multi-env patterns, dependency management

Scripts

• scripts/validate-k8s-manifest.sh — Validate K8s YAML against schemas (kubeval/kubeconform)
• scripts/security-scan-k8s.sh — Security scan for K8s manifests (kubesec, privileged checks)