K8s aiops

k8s AIops

Disclaimer: This is a community-maintained open-source project and is not affiliated with, endorsed by, or sponsored by the Cloud Native Computing Foundation, the Kubernetes project, or k3s/Rancher. "Kubernetes" and "k3s" are trademarks of their respective owners. Source code is publicly auditable at github.com/AIops-tools/K8s-AIops under the MIT license.

Governed Kubernetes operations — 55 MCP tools, every one wrapped with the bundled @governed_tool harness: a local unified audit log under ~/.k8s-aiops/, a token/runaway budget guard, undo-token recording, and a descriptive risk-tier label on every audit row. Works with any kubeconfig-reachable cluster (standard Kubernetes, k3s, EKS, GKE, AKS). Run k8s-aiops init for a friendly onboarding wizard that registers your kube contexts as named targets.

Standalone: the governance harness is bundled in the package (k8s_aiops.governance) — k8s-aiops has no external skill-family dependency. Coverage focuses on common operations and is not yet exhaustive.

What This Skill Does

Category	Tools	Count	Read or Write
Pods	list, get, logs, describe, delete	5	4 read / 1 write
Deployments	list, get, scale, rollout restart, delete	5	2 read / 3 write
Rollout	status, history, undo, pause, resume, set-image	6	2 read / 4 write
StatefulSets	list, get, scale	3	2 read / 1 write
DaemonSets	list, get	2	2 read
ReplicaSets	list	1	1 read
Jobs / CronJobs	job list/get/delete, cronjob list/get	5	4 read / 1 write
Services / Ingress / Endpoints	service list, ingress list/get, endpoints list	4	4 read
Config / Secrets	configmap list/get, secret list (names/keys only)	3	3 read
Storage	pvc list/get, pv list, storageclass list	4	4 read
Nodes	list, describe, cordon, uncordon, drain	5	2 read / 3 write
Namespaces	list, create, delete	3	1 read / 2 write
Metrics (top)	pod, node	2	2 read
Cluster	cluster_info, api_resources	2	2 read
Events	list	1	1 read
Diagnostics / RCA	pod-health, workload-readiness	2	2 read

Quick Install

uv tool install k8s-aiops
k8s-aiops init            # friendly wizard: register your kube contexts as targets
k8s-aiops doctor          # or skip init — works with your current kube-context too

When to Use This Skill

• List/inspect pods, deployments, services, nodes, namespaces and recent events
• Read a pod's recent log lines to diagnose a crash loop
• Run a read-only RCA sweep (diagnose pod-health / diagnose workload-readiness) to find the root cause worst-first
• Scale a deployment up/down, or trigger a rolling restart
• Delete a stuck pod (a controller recreates it) or a deployment
• Cordon a node before maintenance, then uncordon it after

Do NOT use when the target is not a Kubernetes cluster (hypervisor VM lifecycle, backup products, or cloud-provider consoles are out of scope for this skill).

Related Skills — Skill Routing

If the user wants…	Use
Kubernetes pods / deployments / nodes	k8s-aiops (this skill)
Hypervisor VM lifecycle (power, snapshot, migrate)	a hypervisor ops skill
Backup & restore	a backup ops skill

Common Workflows

Diagnose a crash-looping pod and restart its deployment

1. k8s-aiops pod list -n prod → find the pod with high restarts / non-Running phase
2. k8s-aiops pod logs <pod> -n prod --tail 200 → read the recent logs for the crash cause
3. k8s-aiops events -n prod → check for FailedScheduling / image-pull events
4. k8s-aiops deployment restart <deploy> -n prod → roll the deployment after fixing the cause
5. Failure branch: if logs/events show an RBAC 403, the kube context lacks the verb — run kubectl auth can-i get pods -n prod and switch to a context with adequate RBAC; the skill never retries a denied auth.

Triage an unhealthy namespace with RCA, then act on the worst finding

1. k8s-aiops diagnose pod-health -n prod → worst-first findings; a critical CrashLoopBackOff on prod/api cites restarts=9 and the exact kubectl logs … --previous action
2. k8s-aiops diagnose workload-readiness -n prod → confirm the blast radius: e.g. Deployment web ready 0/3 (critical, under-replicated)
3. k8s-aiops pod logs api-<hash> -n prod --tail 200 --previous-equivalent via k8s-aiops pod describe api-<hash> -n prod → read the crash cause the RCA pointed you at
4. k8s-aiops deployment restart web -n prod → roll the deployment once the root cause is fixed
5. Failure branch: if diagnose returns an RBAC 403, the kube context cannot list pods/deployments in that namespace — run kubectl auth can-i list pods -n prod and switch to a context with adequate RBAC; the RCA tools are read-only and never retry a denied auth.

Drain a node for maintenance, safely reversible

1. k8s-aiops node list → identify the node and confirm it is Ready/schedulable
2. k8s-aiops node cordon <node> --dry-run → preview, then k8s-aiops node cordon <node> (double confirm) — records an inverse uncordon_node undo descriptor
3. After maintenance: k8s-aiops node uncordon <node> → re-enable scheduling
4. Failure branch: if doctor shows the cluster unreachable, fix the kubeconfig context (kubectl config get-contexts) before retrying — cordon is never issued against an unauthenticated session.

Usage Mode

Scenario	Recommended	Why
Local/small models	CLI	fewer tokens than MCP
Cloud models (Claude, GPT)	Either	MCP gives structured JSON I/O
Automated pipelines	MCP	type-safe parameters, audited

MCP environment caveat: MCP clients spawn the server with a CLEAN environment — shell exports may not reach it. Set K8S_AIOPS_HOME, K8S_AUDIT_APPROVED_BY, K8S_AUDIT_RATIONALE (and KUBECONFIG when the kubeconfig is not at ~/.kube/config) in the MCP server config's env block, not just in your terminal.

MCP Tools (55 — 39 read, 16 write)

Category	Tools	R/W
Pods	pod_list, pod_get, pod_logs, pod_describe	Read
	delete_pod	Write
Deployments	deployment_list, deployment_get	Read
	scale_deployment, rollout_restart_deployment, delete_deployment	Write
Rollout	rollout_status, rollout_history	Read
	rollout_undo_deployment, rollout_pause, rollout_resume, set_deployment_image	Write
StatefulSets	statefulset_list, statefulset_get	Read
	scale_statefulset	Write
DaemonSets / ReplicaSets	daemonset_list, daemonset_get, replicaset_list	Read
Jobs / CronJobs	job_list, job_get, cronjob_list, cronjob_get	Read
	delete_job	Write
Services / Ingress	service_list, ingress_list, ingress_get, endpoints_list	Read
Config / Secrets	configmap_list, configmap_get, secret_list (names/keys only)	Read
Storage	pvc_list, pvc_get, pv_list, storageclass_list	Read
Nodes	node_list, node_describe	Read
	cordon_node, uncordon_node, drain_node	Write
Namespaces	namespace_list	Read
	create_namespace, delete_namespace	Write
Metrics (top)	node_top, pod_top	Read
Cluster	cluster_info, api_resources	Read
Events	event_list	Read
Diagnostics / RCA	pod_health_rca, workload_readiness_rca	Read
Undo	undo_list	Read
	undo_apply	Write

Security — secrets: secret_list returns secret names, types, and key NAMES only. Secret VALUES are never read, returned, or logged, and there is deliberately no tool that returns secret values.

Dry-run previews: every write tool takes dry_run: bool = False. A dry run returns a {"dryRun": true, "wouldX": ...} preview without touching the cluster, and no undo descriptor is recorded for a preview.

Harness features that light up: write tools with a clean inverse pass an undo= lambda so the harness records an inverse descriptor (with _undo_id) to the undo store — scale_deployment/scale_statefulset record a scale-back to their returned previous_replicas, set_deployment_image records a restore to the captured previous_image, cordon_node ↔ uncordon_node and rollout_pause ↔ rollout_resume are mutual inverses, and create_namespace records a delete_namespace. drain_node records a partial uncordon_node inverse (cordon is reversible; evictions are not). delete_* and rollout_undo_deployment declare no undo. risk_level=high: delete_deployment, delete_job, delete_namespace, drain_node, rollout_undo_deployment. undo_list (read) lists recorded reversible writes whose undo tokens have not been applied yet, and undo_apply (write) executes a recorded inverse — itself governed, single-use, and supports dry_run. All 55 tools are audit-logged under ~/.k8s-aiops/ and pass through the budget/runaway guard, each recorded with a descriptive risk-tier label. pod_top/node_top return a clear "metrics-server not installed" message (not an error) when metrics-server is absent. Avoid tight poll loops (re-listing pods every second) — the runaway breaker backs this up.

CLI Quick Reference

k8s-aiops init                                            # interactive onboarding wizard
k8s-aiops pod list [-n <ns>] [-t <target>]
k8s-aiops pod get <name> [-n <ns>]
k8s-aiops pod describe <name> [-n <ns>]                   # status, container states, events
k8s-aiops pod logs <name> [-n <ns>] [--tail 200] [-c <container>]
k8s-aiops pod delete <name> [-n <ns>] [--dry-run]        # double confirm
k8s-aiops deployment list|get|scale|restart|delete ...    # scale/restart: single confirm + --dry-run; delete: double confirm
k8s-aiops rollout status|history|pause|resume <name> [-n <ns>]
k8s-aiops rollout set-image <name> <container> <image> [-n <ns>]
k8s-aiops rollout undo <name> [--to-revision N] [--dry-run]   # double confirm
k8s-aiops statefulset list|get|scale ...
k8s-aiops daemonset list|get ...
k8s-aiops job list|get|delete ...                         # delete: double confirm
k8s-aiops cronjob list|get ...
k8s-aiops service list [-n <ns>]
k8s-aiops ingress list|get [-n <ns>]
k8s-aiops configmap list|get [-n <ns>]
k8s-aiops secret list [-n <ns>]                           # names/keys only — never values
k8s-aiops storage pvc-list|pvc-get|pv-list|class-list
k8s-aiops top pod|node                                    # requires metrics-server
k8s-aiops node list|describe
k8s-aiops node cordon|drain <name> [--dry-run]           # double confirm
k8s-aiops node uncordon <name>
k8s-aiops namespace list|create
k8s-aiops namespace delete <name> [--dry-run]            # double confirm
k8s-aiops cluster-info
k8s-aiops api-resources
k8s-aiops events [-n <ns>]
k8s-aiops diagnose pod-health [-n <ns>] [-l <selector>]         # read-only RCA: crashloop/imagepull/OOM/unschedulable/restarts
k8s-aiops diagnose workload-readiness [-n <ns>]                 # read-only RCA: ready<desired / stuck rollouts
k8s-aiops doctor
k8s-aiops mcp                                             # start MCP server (stdio)

See references/cli-reference.md for the full command list.

Troubleshooting

"Could not load kubeconfig … context not found"

The named context does not exist in your kubeconfig. Run kubectl config get-contexts and set the target's context: to a listed name (or omit it to use current-context).

"Authentication/authorization failed (401/403)"

The kube context lacks the RBAC verb for the resource. Check with kubectl auth can-i <verb> <resource> -n <ns> and switch to a context/ServiceAccount with adequate roles. For EKS/GKE/AKS, confirm the exec-plugin (aws/gcloud/az CLI) is installed and logged in.

"Resource not found (404)"

The pod/deployment/node name or namespace is wrong, or the object was deleted. List the parent collection first (pod list, deployment list, node list) to get a current name. Remember most commands default to the default namespace unless -n is given.

"Conflict (409)"

The object changed concurrently (or already exists). Re-read it and retry the write.

Logs are empty or truncated

pod logs returns the trailing --tail lines (default 100); raise --tail. For a multi-container pod, pass -c <container> or the API returns an error naming the available containers.

Audit & Safety

All operations are automatically audited via the bundled @governed_tool decorator (k8s_aiops.governance):

• Every tool call logged to ~/.k8s-aiops/audit.db (local SQLite audit DB; relocate with K8S_AIOPS_HOME)
• Budget / runaway guard caps cumulative tool calls and wall-time, and trips on tight poll/retry loops — a safety backstop, not authorization
• Undo store records inverse descriptors for reversible writes (scale → previous replicas; cordon ↔ uncordon)
• Each write carries a descriptive risk-tier label into its audit row — a label, not a gate; K8S_AUDIT_APPROVED_BY / K8S_AUDIT_RATIONALE are optional annotations recorded when set, never required

Authorization is not this tool's job. There is no read-only switch, policy file, or approval gate. Whether a write is permitted is the agent's judgement or the RBAC of the kubeconfig context you connect with — give it a read-only ServiceAccount and writes fail at the apiserver, the place that owns the permission.

The harness is bundled in the package — no external dependency, no manual setup. See references/setup-guide.md for security details.

Driving these tools with a smaller / local model? See references/agent-guardrails.md — which guardrails the tool now enforces for you, plus a ready-to-paste system prompt.

Contributing & feature requests

Coverage is intentionally focused. Missing a device, action, or feature you need? Open an issue or pull request at github.com/AIops-tools/K8s-AIops — feature requests, contributions, and comments are all welcome.

License

MIT — github.com/AIops-tools/K8s-AIops