Kubernetes operator

Build operators that reconcile correctly

An operator is a reconcile loop, not a script. Most operator bugs are not Kubernetes bugs — they are reconcile-loop bugs: missing finalizers, blocking calls, no requeue on transient errors, status drift, RBAC over-grants.

The reconcile mental model

observe(actual) → desired = read(spec) → diff(actual, desired) → act → update(status)
                                                                         ↓
                                                                  requeue / done

Idempotent, not imperative. The reconcile function must be callable any number of times with the same outcome. It must not assume what happened on the previous call.

Common operator bugs

Bug	Prevention
Blocking HTTP calls in reconcile	Use async clients or move to a goroutine
No requeue on transient error	Always return ctrl.Result{}, err for transient; RequeueAfter for polling
Missing finalizer	Add finalizer on creation; remove only after cleanup complete
Mutating spec instead of status	Spec is user-owned; status is controller-owned
No status subresource	Status updates without subresource trigger spec reconcile → loop
RBAC over-grant	Principle of least privilege; use ClusterRole only when namespace-scope is insufficient
No leader election	Multi-replica deploy without leader election → split-brain

Operator capability levels (OLM)

Level 1 (Basic install) → Level 2 (Seamless upgrades) → Level 3 (Full lifecycle) → Level 4 (Deep insights) → Level 5 (Auto pilot)

Start at Level 1 and promote only when the lower levels are tested and stable.

Steps

1. Read the lore + existing operators. search_lore for CRD conventions, RBAC policy, and framework choice. Match the existing toolchain (kubebuilder vs operator-sdk vs KOPF vs metacontroller).
2. Design the CRD API surface. spec = desired state (user-owned); status = observed state (controller-owned). Define conditions as standard status fields. Validate with admission webhooks for types that have invariants.
3. Implement the reconcile loop. Read → diff → act → update status. Every return is either Result{}, nil (done, no requeue) or Result{}, err / Result{RequeueAfter: d}, nil (retry/poll).
4. Add finalizers. Register on object creation; execute cleanup in the finalizer block; remove only after cleanup succeeds.
5. RBAC. Generate from controller-gen annotations; scope to namespace where possible; document every ClusterRole grant.
6. Leader election. Enable for any operator that will run with >1 replica.
7. Verify. Deploy to a local cluster (kind/minikube); create/update/delete a CR; confirm status conditions reflect the current state; test error injection (unavailable dependency); record evidence.

Build / Test

• go vet + golangci-lint clean.
• Unit tests for the reconcile function with a fake client — test every return path.
• E2E tests against a real cluster (kind): create, update (in-place upgrade), delete with finalizer cleanup.
• controller-gen generates CRD + RBAC manifests — commit generated files, do not hand-edit them.

Review checklist

• Reconcile is idempotent — safe to call N times with the same result.
• Every error path requeues — transient errors return err; expected polls use RequeueAfter.
• Finalizer present — and cleanup is tested on deletion.
• Status uses conditions — standard Kubernetes condition format (Type, Status, Reason, Message).
• RBAC minimal — no ClusterRole without justification; generated by controller-gen.
• Leader election enabled for multi-replica deploys.

Capture lore

Framework choice, CRD naming conventions, cluster version, and RBAC policy are high-value lore — call suggest_lore with tags: [kubernetes, operator, crd].