Kelly agent observability

Agent Fleet Observability Desk

Overview

Use this skill as a local operator surface over a MOCK fleet of LLM agents running behind a shared AI gateway for a generic organization. It aggregates per-agent call volume, latency (p50/p95), error/failure rate, and cost into one file-backed App-in-Skill dashboard with a Fleet Overview, a sortable Agent Health table, and a Trace Detail view that highlights exactly where a chain of tool calls broke.

Default interaction mode: App UI. Unless the user explicitly asks for chat-only handling, seed mock telemetry if missing, start/reuse the local app with app/start.sh, and give the actual local URL. Use chat-only mode only when the user says "纯聊天", "chat only", "no UI", or similar.

This is a demo/reference dashboard: all data is generated locally by scripts/generate_fleet_data.ts. There is no real gateway, no real agents, and no external network calls anywhere in this skill.

App UI Screenshots

<table> <tr> <td width="50%"><img src="assets/screenshots/overview.webp" alt="Fleet overview"></td> <td width="50%"><img src="assets/screenshots/agent-health.webp" alt="Agent health table"></td> </tr> <tr> <td><strong>Fleet Overview</strong><br>Total calls, total cost, degraded/critical/healthy agent counts, and a per-agent sparkline card grid.</td> <td><strong>Agent Health</strong><br>Sortable table with call volume, p50/p95 latency, error rate, cost, and a status badge per agent.</td> </tr> <tr> <td width="50%"><img src="assets/screenshots/trace-detail.webp" alt="Trace detail with chain break"></td> <td width="50%"><img src="assets/screenshots/overview.zh-CN.webp" alt="Fleet overview in Chinese"></td> </tr> <tr> <td><strong>Trace Detail</strong><br>Ordered step timeline for one trace; the step where the chain broke is visually flagged.</td> <td><strong>Chinese UI</strong><br>Full zh-CN chrome via the language toggle or <code>lang=zh</code>.</td> </tr> </table>

Boundary

• Local, read-only mock dashboard. The app reads and writes local files only.
• The only mutating endpoint is POST /api/handoffs, which appends a human-in-the-loop acknowledgement or investigation note to a local app/.data/handoffs.jsonl file. It never calls any remote system, agent gateway, or third-party API.
• Never reference any real company, brand, or production system. Agent names are generic archetypes (Booking Assistant, Support Triage, etc.) for a generic organization ("your organization", "the fleet").

Local App

Start the dashboard with:

skills/kelly-agent-observability/app/start.sh

First run installs hono and @hono/node-server, then seeds mock telemetry into app/.data/fleet.json if it doesn't already exist. The frontend is zero-build vanilla JS/HTML/CSS. The app uses local HTTP on 127.0.0.1, preferring port 3100 through 4100, or KELLY_AGENT_OBS_UI_PORT when set.

To regenerate mock telemetry at any time:

node skills/kelly-agent-observability/scripts/generate_fleet_data.ts

Demo Mode

• ?demo=1 opens a deterministic, fully offline mock fleet (8 agents, 48h of hourly buckets, ~16 traces per agent) for documentation and screenshots, without touching app/.data/fleet.json.
• ?demo=agents and ?demo=trace select named mock scenes (agent health table, a broken trace).
• lang=en or lang=zh forces UI chrome language for screenshots.
• Demo API responses never read or write local files.

UI language: support English and Chinese chrome with Auto default.

Data Model

Read references/fleet-schema.md before editing the app, scripts, or the generator. Primary local files:

• app/.data/fleet.json: generated mock telemetry (agents, per-agent metrics, hourly buckets, traces). Regenerated by scripts/generate_fleet_data.ts.
• app/.data/handoffs.jsonl: append-only human-in-the-loop handoff log written by POST /api/handoffs.

Views

• #/overview: total calls (24h), total cost (today), degraded/critical/ healthy agent counts, and a per-agent card grid with a call-volume sparkline.
• #/agents: sortable Agent Health table (call volume, p50/p95 latency, error rate, cost, status badge).
• #/agents/<agent_id>: agent detail — metrics, sparkline, recent traces, and a handoff form (acknowledge / needs investigation).
• #/traces/<trace_id>: trace detail — ordered step timeline with the chain-break step visually flagged, and a handoff form.
• #/handoffs: history of recorded handoffs.
• #/settings: data provider and status-threshold summary.

Safety

• Never invent a real incident or real company data; this is illustrative mock telemetry only.
• Keep all writes local and file-based (app/.data/handoffs.jsonl). No external network calls anywhere in this skill.
• Redact nothing sensitive is collected here — there are no credentials in this skill.

Execution reports

Re-read the active provider's decisions immediately before any approved execution. Record each concrete operation, target, status, timestamp, and error in the provider-backed execution report; keep app actions local-only.