Kubernetes

Kubernetes

Purpose

Deploy workloads that survive node failures, scale predictably, and fail visibly. Most Kubernetes incidents trace back to three things: wrong resource limits, wrong probes, and a rollout that had no way to be judged healthy.

When to Use

• Writing or reviewing Kubernetes manifests.
• Debugging a pod that will not start, keeps restarting, or is being evicted.
• Configuring autoscaling, rollouts, or disruption budgets.
• Diagnosing networking or DNS problems inside a cluster.

Capabilities

• Workload configuration: Deployments, StatefulSets, Jobs, CronJobs.
• Resource requests, limits, and quality-of-service classes.
• Liveness, readiness, and startup probes.
• Rollout strategies, PodDisruptionBudget, and graceful shutdown.
• Networking: Services, Ingress, NetworkPolicy, cluster DNS.
• Debugging: events, logs, ephemeral containers.

Inputs

• The workload, its resource profile, and its startup behavior.
• The cluster's constraints: node sizes, available classes, policies.
• The symptom, if debugging: pod status, events, exit code.

Outputs

• Manifests with explicit requests, limits, and probes.
• A rollout that is safe under disruption.
• A root cause with the evidence that identified it.

Workflow

1. Set requests and limits deliberately — Requests determine scheduling; limits determine throttling and killing. A pod with no requests is scheduled anywhere and evicted first.
2. Configure the three probes distinctly — Startup probe protects a slow boot. Readiness controls traffic. Liveness restarts a hung process. Conflating them causes a slow-starting pod to be killed forever.
3. Handle SIGTERM — Kubernetes sends SIGTERM, waits terminationGracePeriodSeconds, then SIGKILL. A process that ignores SIGTERM drops in-flight requests on every deploy.
4. Protect the rollout — maxUnavailable, maxSurge, and a PodDisruptionBudget so a node drain cannot take the last replica.
5. Debug from the events — kubectl describe pod before kubectl logs. The reason is usually in the events, not the application output.

Best Practices

• A CPU limit throttles; a memory limit kills. Setting a CPU limit equal to the request wastes burst capacity for no reliability gain — in most cases, set CPU requests and omit CPU limits.
• Memory limits should be set. Without one, a leaking pod takes the whole node with it.
• A liveness probe that hits a dependency will restart your pod when the dependency is down, converting a partial outage into a total one. Liveness checks the process; readiness checks the dependencies.
• imagePullPolicy: Always with a :latest tag makes rollbacks impossible. Pin an immutable digest or a version tag.
• Never store secrets in a ConfigMap. Use a Secret, and preferably an external secret store.
• kubectl exec into a distroless container will fail. Use kubectl debug with an ephemeral container.

Examples

A Deployment with correct probes and graceful shutdown:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: orders-api
spec:
  replicas: 3
  strategy:
    rollingUpdate: { maxSurge: 1, maxUnavailable: 0 }   # never dip below capacity
  template:
    spec:
      terminationGracePeriodSeconds: 45                 # > the longest in-flight request
      containers:
        - name: api
          image: registry.example.com/orders-api@sha256:9f2c...   # immutable
          resources:
            requests: { cpu: 250m, memory: 512Mi }       # scheduling
            limits:   { memory: 1Gi }                    # OOM ceiling; no CPU limit
          startupProbe:                                  # allows up to 60s to boot
            httpGet: { path: /healthz, port: 8080 }
            failureThreshold: 30
            periodSeconds: 2
          readinessProbe:                                # gates traffic; checks deps
            httpGet: { path: /readyz, port: 8080 }
            periodSeconds: 5
          livenessProbe:                                 # restarts a hung process only
            httpGet: { path: /healthz, port: 8080 }
            periodSeconds: 10
            failureThreshold: 3
          lifecycle:
            preStop:
              exec:
                command: ["sleep", "10"]                 # let the LB deregister first
---
apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: orders-api
spec:
  minAvailable: 2
  selector:
    matchLabels: { app: orders-api }

Diagnosing the standard failures:

kubectl describe pod orders-api-7d4f -n prod | sed -n '/Events/,$p'
# OOMKilled       -> memory limit too low, or a leak. Check actual usage, then raise or fix.
# CrashLoopBackOff-> read the previous container's logs: kubectl logs --previous
# ImagePullBackOff-> registry auth or a tag that does not exist
# Pending         -> no node satisfies the requests; check `kubectl describe node`

Notes

• The preStop sleep is not superstition: the endpoint removal and the SIGTERM race each other. Sleeping for a few seconds lets the load balancer stop sending traffic before the process starts shutting down.
• readinessProbe failing removes the pod from the Service but does not restart it — which is exactly right when a dependency is temporarily unavailable.
• Vertical Pod Autoscaler in recommendation mode is the fastest way to discover what your requests should actually be, without letting it change them for you.