Kelly agent observability

Skill mr-kelly/skills/skills/kelly-agent-observability

Agent Skills by mr-kelly

Install

npx -y skills add mr-kelly/skills --skill kelly-agent-observability

Assembled from the repository path, not quoted from the project. Check it against their README if it does not work.

One thing to look at

3 stars3 stars. Stars are a popularity signal and not a quality one, but at this level it is likely that nobody has read this closely except its author, and you would be relying on your own review.

What its author says it does

Copied from the file, not written here

Agent Fleet Observability Desk App-in-Skill — local mock dashboard visualizing a fleet of LLM agents running behind a shared AI gateway for your organization. Use when the user invokes $kelly-agent-observability or /kelly-agent-observability, wants to review agent health, call volume, latency, error rates, cost, or trace-level chain breaks across an agent fleet. This is a demo/reference dashboard over generated mock telemetry, not a live monitoring integration.

SKILL.md

5.6 KB, as published. Nobody here has run it

Agent Fleet Observability Desk

Overview

Use this skill as a local operator surface over a MOCK fleet of LLM agents running behind a shared AI gateway for a generic organization. It aggregates per-agent call volume, latency (p50/p95), error/failure rate, and cost into one file-backed App-in-Skill dashboard with a Fleet Overview, a sortable Agent Health table, and a Trace Detail view that highlights exactly where a chain of tool calls broke.

Default interaction mode: App UI. Unless the user explicitly asks for chat-only handling, seed mock telemetry if missing, start/reuse the local app with app/start.sh, and give the actual local URL. Use chat-only mode only when the user says "纯聊天", "chat only", "no UI", or similar.

This is a demo/reference dashboard: all data is generated locally by scripts/generate_fleet_data.ts. There is no real gateway, no real agents, and no external network calls anywhere in this skill.

App UI Screenshots

<table> <tr> <td width="50%"><img src="assets/screenshots/overview.webp" alt="Fleet overview"></td> <td width="50%"><img src="assets/screenshots/agent-health.webp" alt="Agent health table"></td> </tr> <tr> <td>Fleet Overview Total calls, total cost, degraded/critical/healthy agent counts, and a per-agent sparkline card grid.</td> <td>Agent Health Sortable table with call volume, p50/p95 latency, error rate, cost, and a status badge per agent.</td> </tr> <tr> <td width="50%"><img src="assets/screenshots/trace-detail.webp" alt="Trace detail with chain break"></td> <td width="50%"><img src="assets/screenshots/overview.zh-CN.webp" alt="Fleet overview in Chinese"></td> </tr> <tr> <td>Trace Detail Ordered step timeline for one trace; the step where the chain broke is visually flagged.</td> <td>Chinese UI Full zh-CN chrome via the language toggle or <code>lang=zh</code>.</td> </tr> </table>

Boundary

Local, read-only mock dashboard. The app reads and writes local files only.
The only mutating endpoint is POST /api/handoffs, which appends a human-in-the-loop acknowledgement or investigation note to a local app/.data/handoffs.jsonl file. It never calls any remote system, agent gateway, or third-party API.
Never reference any real company, brand, or production system. Agent names are generic archetypes (Booking Assistant, Support Triage, etc.) for a generic organization ("your organization", "the fleet").

Local App

Start the dashboard with:

skills/kelly-agent-observability/app/start.sh

First run installs hono and @hono/node-server, then seeds mock telemetry into app/.data/fleet.json if it doesn't already exist. The frontend is zero-build vanilla JS/HTML/CSS. The app uses local HTTP on 127.0.0.1, preferring port 3100 through 4100, or KELLY_AGENT_OBS_UI_PORT when set.

To regenerate mock telemetry at any time:

node skills/kelly-agent-observability/scripts/generate_fleet_data.ts

Demo Mode

?demo=1 opens a deterministic, fully offline mock fleet (8 agents, 48h of hourly buckets, ~16 traces per agent) for documentation and screenshots, without touching app/.data/fleet.json.
?demo=agents and ?demo=trace select named mock scenes (agent health table, a broken trace).
lang=en or lang=zh forces UI chrome language for screenshots.
Demo API responses never read or write local files.

UI language: support English and Chinese chrome with Auto default.

Data Model

Read references/fleet-schema.md before editing the app, scripts, or the generator. Primary local files:

app/.data/fleet.json: generated mock telemetry (agents, per-agent metrics, hourly buckets, traces). Regenerated by scripts/generate_fleet_data.ts.
app/.data/handoffs.jsonl: append-only human-in-the-loop handoff log written by POST /api/handoffs.

Views

#/overview: total calls (24h), total cost (today), degraded/critical/ healthy agent counts, and a per-agent card grid with a call-volume sparkline.
#/agents: sortable Agent Health table (call volume, p50/p95 latency, error rate, cost, status badge).
#/agents/<agent_id>: agent detail — metrics, sparkline, recent traces, and a handoff form (acknowledge / needs investigation).
#/traces/<trace_id>: trace detail — ordered step timeline with the chain-break step visually flagged, and a handoff form.
#/handoffs: history of recorded handoffs.
#/settings: data provider and status-threshold summary.

Safety

Never invent a real incident or real company data; this is illustrative mock telemetry only.
Keep all writes local and file-based (app/.data/handoffs.jsonl). No external network calls anywhere in this skill.
Redact nothing sensitive is collected here — there are no credentials in this skill.

Execution reports

Re-read the active provider's decisions immediately before any approved execution. Record each concrete operation, target, status, timestamp, and error in the provider-backed execution report; keep app actions local-only.