Scan health

scan-health: Service Health Monitor

The "eyes" of the harness. Checks configured HTTP endpoints for availability and response time, compares against baseline metrics, and generates alerts when thresholds are breached.

• Checks configured HTTP endpoints for availability and response time
• Compares against baseline metrics stored in service_health.json
• Generates alerts when thresholds are breached
• READ-ONLY: no code changes, no PRs — writes only to agent/state/service_health.json

---

Safety Rules

1. HALT file — Check config.safety.halt_file before any work. If present, print reason and stop.
2. Read-only — Only writes to agent/state/service_health.json. No external API modifications.
3. Graceful degradation — Record failures without crashing. Missing config → skip with a message.

---

Step 0: Safety

Check the HALT file at config.safety.halt_file. If it exists, print: HALT: <reason>. scan-health aborted. and exit.

Check config.health_endpoints. If missing or empty: No health endpoints configured. Skipping. — exit 0.

Validate the list before proceeding:

• Deduplicate — remove duplicate URLs (keep first occurrence).
• Reject malformed URLs — entries that do not start with http:// or https:// are skipped with a warning: [WARN] Skipping malformed URL: <url>.
• Cap at 20 endpoints — if more than 20 remain after dedup, check only the first 20 and warn: [WARN] Endpoint list truncated to 20 (had <N>).

---

Step 0.5: Lens Load (Adaptive Context)

Read agent/state/service_health/lens.json. If missing or unparseable, proceed with base behavior only — this step is purely additive.

If lens exists:

• focus_items (confidence >= 0.6): Prioritize these endpoints/metrics first. Allocate extra time and retries to high-priority items.
• learned_thresholds (confidence >= 0.6): Override default thresholds. For example, if the lens says endpoint X has a learned threshold of 800ms instead of the default 1500ms, use 800ms for anomaly detection on that endpoint.
• discovered_signals (actionable=true): Include these as additional diagnostic checks beyond the base behavior. For example, if the lens says "deploy failures correlate with health degradation", check recent deploy status first.

If lens is corrupt (invalid JSON, missing schema_version): Log: "[WARN] Lens file corrupt, using base behavior." Continue normally. Do NOT crash.

---

Step 1: Baseline Load

Read agent/state/service_health.json. If missing, create with initial structure:

{
  "schema_version": "1.0.0",
  "last_run": null,
  "run_count": 0,
  "status": "unknown",
  "alerts": [],
  "baseline": { "endpoints": [] }
}

Extract previous per-endpoint avg_response_ms, last_status, consecutive_failures for use in anomaly detection.

---

Step 2: Endpoint Checks

For each URL in config.health_endpoints:

curl -s -o /dev/null -w "%{http_code} %{time_total}" --max-time <timeout_seconds> <url>

Record: url, status_code, response_time_ms, timestamp. Timeout: config.health_check_timeout_seconds (default 10, valid range 2-30).

Method support: if the endpoint object includes a method field (e.g., "method": "POST"), pass it to curl via -X {method}. Default: GET.

curl -s -o /dev/null -w "%{http_code} %{time_total}" -X {method} --max-time <timeout_seconds> <url>

Status code note: curl's %{http_code} returns a three-digit string. On connection refused, DNS failure, or timeout it returns 000 (not 0). Treat both 0 and 000 as network failure — always compare numerically (e.g., parseInt(code) === 0) rather than by string equality.

Response time: curl's %{time_total} is in seconds (float). Convert to milliseconds: response_time_ms = time_total * 1000.

Retry once on network failure (status 0 / 000 / connection refused). Skip retry on timeout (saves waiting another full timeout period).

If curl unavailable, fall back to wget --server-response --timeout=<timeout_seconds>. If both unavailable, record status_code: 0, error: "no_http_client".

---

Step 3: Anomaly Detection

Condition	Alert type	Severity
status_code 5xx or 0/000 (single endpoint)	endpoint_down	warning
status_code 5xx or 0/000 (2+ endpoints)	multiple_endpoints_down	critical
status_code 3xx (without expected_status override)	endpoint_redirect	info
status_code 403 (without expected_status override)	endpoint_forbidden	warning
response_time_ms > 1500	slow_response	warning
response_time_ms > 2x baseline avg	response_degradation	warning

Expected status override: if the endpoint object includes expected_status (e.g., 301), treat that status code as healthy (equivalent to 2xx). This allows monitoring redirect-based or custom health endpoints. When expected_status is set, suppress the corresponding alert type (e.g., endpoint_redirect for 3xx). If the response does NOT match expected_status, generate an unexpected_status alert with severity warning.

Baseline comparison on failure — when an endpoint is DOWN (HTTP 000, connection refused, or timeout with no response), include baseline context in the alert detail so the user understands what changed:

Previous baseline: {avg_response_ms}ms / 200 OK (from {samples} samples)
Current: CONNECTION REFUSED
Change: endpoint_down (was healthy)

If no baseline exists for the endpoint (first run), omit the "Previous baseline" line and note Change: endpoint_down (no prior baseline).

Alert shape: { "severity": "warning", "type": "...", "endpoint": "...", "detail": "..." }

Increment consecutive_failures per endpoint on 5xx or status 0/000 (timeout/connection refused). Reset to 0 on any 2xx response. Do NOT increment on 3xx or 403 (endpoint is reachable).

---

Step 4: State Update

Write to agent/state/service_health.json:

{
  "schema_version": "1.0.0",
  "last_run": "<ISO 8601>",
  "run_count": "<N>",
  "status": "healthy | degraded | critical",
  "consecutive_failures": 0,
  "alerts": [{ "severity": "warning", "type": "slow_response", "endpoint": "...", "detail": "..." }],
  "baseline": {
    "endpoints": [{ "url": "...", "avg_response_ms": 130, "last_status": 200, "consecutive_failures": 0 }]
  }
}

Top-level consecutive_failures = the MAX of the per-endpoint consecutive_failures values, written every run. This is the variable evolve's 4-B chain trigger evaluates (consecutive_failures >= 3) — trigger conditions read top-level state fields, so the nested per-endpoint counters alone would leave the trigger undecidable. Also: when config.health_endpoints is empty, write status: "no_endpoints" (readers must treat it as neither healthy nor critical — informational only).

Status: healthy = all 2xx, no warning/critical alerts (info alerts are OK). degraded = any warning alert. critical = any critical alert. Update avg_response_ms using an exponential moving average: new_avg = old_avg * 0.8 + current_response_ms * 0.2. On the first run for an endpoint (no prior baseline), set avg_response_ms = current_response_ms directly.

EMA on failure: when status_code is 0/000 (connection refused, DNS failure, timeout), do NOT update avg_response_ms — the response time is meaningless (either ~0ms for connection refused or the full timeout duration). Keep the previous avg_response_ms intact so it remains a reliable baseline of healthy performance.

Note: This skill does NOT write to agent/state/service_health/lens.json. Lens updates (learning thresholds, promoting signals, adjusting focus) happen exclusively in evolve's Reflect phase (Step 5-E).

---

Step 5: Report

scan-health — <ISO timestamp>
Overall status: healthy | degraded | critical

| Endpoint                 | Status | Response (ms) | Baseline (ms) | Alert          |
|--------------------------|--------|---------------|---------------|----------------|
| https://example.com/     |  200   |     142       |     130       | —              |
| https://example.com/api  |  503   |     —         |     120       | endpoint_down  |

If alerts exist: Alerts: N warning, N critical. Consider running /dev-cycle for investigation. If consecutive_failures >= 3, note that the dev-cycle chain trigger condition is met.

---

Graceful Degradation

Scenario	Behavior
health_endpoints empty or missing	Print skip message, exit 0
curl not available	Fall back to wget; if both missing, record status_code: 0
All endpoints unreachable	Record status: "critical", write state, report — do NOT crash
service_health.json missing	Create with initial structure and continue
First run (no baseline for endpoint)	Record current values as baseline; skip response_degradation alert (no prior avg to compare)
HALT file present	Print reason, exit immediately before any checks