Prometheus anchored query

prometheus-anchored-query

Purpose

Make ad-hoc PromQL queries safe by default by wrapping the "Contention And O11y Pattern" discipline (anchor labels in the knowledge base FIRST, then run the query) as a reusable skill. Saves the raw payload to a provenance-bearing bundle so future queries can replay the same shape, and so the perf-lake-contract is honored without operator effort.

This skill exists because PromQL is easy to write wrong at cluster scale: a missing label selector turns a 100-series query into a 10,000-series query that overloads the Prometheus backend. The bundled server's mcp-composition.md Contention And O11y Pattern (step 3) makes the knowledge-base-first call mandatory. This skill enforces it.

Use it for ad-hoc questions ("what's the IB traffic on <node> right now?" / "is the dataloader being starved?" / "what's the median dcgm_xid rate across a partition?").

When to use

• Operator has a question that maps to a PromQL query but doesn't know the exact labels yet.
• Operator wants to save a Prometheus payload to disk with provenance (for evidence bundles, post-incident write-ups, share-with-team artifacts).
• Operator wants to confirm a metric exists + has the expected cardinality before writing a dashboard panel for it.

Do not use this skill for:

• Already-vetted queries - call mcp__prometheus_mcp__query_prometheus directly (or run them from whatever dashboarding UI they live in). The anchor pass adds nothing when the query shape is already known.
• Log queries - this skill emits PromQL only. The MCP server also fronts a log datasource via query_loki_logs, but anchored log queries are out of scope here.

Example prompts

• "What's the IB traffic on these 8 nodes for the last hour?"
• "Is dataloader starvation showing up in dcgm_gpu_utilization for the cohort I just ran?"
• "Save the dcgm_xid trajectory across the gb300 partition for the last 24h."
• "What's the median GPU temp on this rack right now?"
• "Anchored PromQL for nccl_perf_bandwidth across the cohort."
• /prometheus-anchored-query --question "IB tx bytes by node for <partition> last 1h" --time-range "now-1h..now"
• /prometheus-anchored-query --metric dcgm_gpu_temp --labels '{partition="gb300"}' --time-range "now-24h..now"

Inference-perf example

The inference-perf-bench workflow scrapes four canonical vLLM Prometheus metrics pre/post each benchmark run:

Metric	What it answers
vllm:prefix_cache_hit_rate	KV cache hit rate during the run (delta pre vs post)
vllm:gpu_cache_usage_perc	GPU cache utilization peak
vllm:num_requests_running	Concurrent requests peak
vllm:avg_generation_throughput_toks_per_s	Generation throughput

Operators replaying a perf-bench run via this skill should anchor against these four metric names first (knowledge-base call confirms cardinality + datasource UID), then run the narrow PromQL with {model_name="<served-name>", namespace="<ns>"} selectors. See inference-perf-bench Phase 4 / 7 for the in-pod scrape pattern this skill complements at the Prometheus MCP layer.

Prerequisites

1. Prometheus MCP reachable (PROMETHEUS_MCP_URL set).
2. Operator question or metric+labels - one of:
   • Free-form question (skill derives metric + labels via knowledge base).
   • Explicit --metric <name> --labels <json>.
3. Time range - --time-range "now-Nh..now" or absolute UTC range. Default now-1h..now.
4. PROFILE_AND_OPTIMIZE_REPO_ROOT for the payload bundle path (falls back to ${PWD}/prometheus-bundles/<run-id>/).

Interaction style

Iterative. The whole point of the skill is the "anchor first, then query" discipline. Pause after the knowledge-base call to surface the proposed PromQL before running it.

Workflow

Phase 0: resolve the question

If operator provided a free-form question, restate the question + the proposed metric / labels in one sentence. Get confirmation.

If operator provided explicit metric + labels, proceed.

Phase 1: knowledge-base anchor

mcp__prometheus_mcp__query_observability_knowledge_base with:
  query: "<one-line question or metric+context>"
  user_prompt: "<operator's original prompt>"

The knowledge-base response tells us:

• Whether the metric exists in the Prometheus MCP's catalog.
• The available labels for that metric.
• The expected cardinality (so we don't accidentally select 100k series).
• The canonical datasource UID for this metric on this cluster (the MCP can front several datasources - e.g. per-cluster Prometheus instances plus a log backend - so every query tool takes an explicit datasourceUid).
• Any noted caveats (sampling rate, retention, legacy renames).

If the knowledge base does NOT cover the metric: stop. Report what's missing. Per mcp-composition.md "Contention And O11y Pattern" step 3, running uncovered PromQL is unsafe.

Phase 2: validate labels exist

For each label the proposed query selects on:

mcp__prometheus_mcp__list_prometheus_label_values with:
  datasourceUid: "<from knowledge-base anchor in Phase 1>"
  labelName: "<label>"
  matches: ["<metric>{<other-selectors>}"]
  user_prompt: "<operator's original prompt>"

If a label or value doesn't exist, the query would be silently empty. Stop and surface.

Phase 3: emit the PromQL

Construct the narrow PromQL using only knowledge-base-confirmed labels and the operator's --time-range. Print it back:

Proposed PromQL: <full query>
Datasource UID:  <from knowledge base>
Time range:      <start> .. <end>
Expected series count: ~<from knowledge base cardinality estimate>

Ask: "Run this query?" Do not auto-advance.

Phase 4: run + save

After confirmation:

mcp__prometheus_mcp__query_prometheus with:
  datasourceUid: "<from-anchor>"
  expr: "<promql>"
  queryType: "range"
  startTime: "<start>"
  endTime: "<end>"
  stepSeconds: <resolution-seconds>
  user_prompt: "<operator's original prompt>"

(Use queryType: "instant" + only endTime for a single-point query. Drop startTime / stepSeconds in that case.)

Save the raw payload to:

${PROFILE_AND_OPTIMIZE_REPO_ROOT}/experiments/artifacts/prometheus-bundles/<run-id>/
  SOURCE.md                     # operator + question + cluster + UTC-ts
  query.promql                  # the exact PromQL run
  query.json                    # the exact request JSON
  response.json                 # the raw Prometheus response (saved verbatim)
  summary.md                    # human-readable summary: series count, range, top-5 anomalies
  knowledge-base-anchor.json    # the knowledge-base response (provenance for the label discipline)

Phase 5: report

Print:

• Series count + time-bucket count (sanity check: did we get the cardinality the knowledge base predicted?).
• Top 5 anomalies (highest / lowest values, recent step-deltas).
• Bundle path for the saved payload.

Full-context reporting (no bare numbers)

Per docs/METHODOLOGY.md "Full-context reporting": every number this skill emits (throughput, latency, TPOT/ITL, BW, %SoL, speedup, efficiency, goodput, acceptance rate, scaling efficiency, thermal/failure rate - whatever it reports) MUST carry its full measurement-context descriptor, and every comparison MUST be matched on it. A bare number is a defect - it cannot set a default, ship a config, or appear in a report.

• Identity: model (+HF path), hardware (exact ceiling token GB300/B200), quant, kv-cache dtype.
• Parallelism: TP, DP (replicas), PP, EP, parallel_strategy.
• Serving cfg: max-num-seqs, max-num-batched-tokens, gpu-memory-utilization, max-model-len, cudagraph_mode/enforce_eager, async_scheduling, prefix-caching.
• Workload: dataset, ISL/OSL (or mean in/out tokens), concurrency, num-prompts.
• Regime: warm vs cold. Latency vs throughput tier.
• Stack: image/vllm commit, bench backend, serving engine.
• Grounding: %SoL (+ ceiling key from configs/sol-ceilings.yaml - never inline a peak), sol_rigor (L1-L4), trials n (mean±std), same-node, baseline named. (If the metric is not roofline-bound - e.g. accuracy/acceptance - omit %SoL but keep the rest of the descriptor.)
• Per-number exact shape (no smoothing): when reporting more than one number, keep EACH with its own exact shape (ISL/OSL, concurrency, dataset, regime) - never normalize a set to one uniform descriptor that hides per-point variation (e.g. c=1 @ ISL1024/OSL256 + c=64 @ ISL4096/OSL512, NOT one shared "random").

Next lever / BREAKTHROUGH (Grind Mandate)

If this skill emits a measured result, its output MUST end by naming the next perf lever, its expected unlock (direction + rough magnitude), and the gate that proves/refutes it, per docs/METHODOLOGY.md "Always be grinding". A measured win is the new floor, not the finish -- so do everything we can to find the next BREAKTHROUGH: the highest-EV unlock toward Speed-of-Light (a new champion / kernel / router / quant / parallelism / spec-decode win, or an unblocked stack), not just the next micro-lever. Rank the candidate breakthrough levers by value x cost (the GRIND FRONTIER, perftunereport value_view), pursue the top, bank the rest with evidence. Record WHY a refuted lever loses, update the standing frontier in the active bundle's HANDOFF.md. Never conclude "exhausted/optimal/done" without an explicit next-lever frontier (an empty frontier AND a documented SoL wall only). Delete this section ONLY if the skill produces no measurements.

Safety

• Knowledge base first is mandatory. Phase 1 cannot be skipped - fail fast, no silent fallbacks. The whole point of the skill is to enforce that discipline.
• No high-cardinality queries. If the knowledge base reports the proposed query would return >10k series, the skill refuses to run it. Operator narrows the selector and retries.
• Raw payload preservation. Per perf-lake-contract.md, every saved query records the datasource UID, the exact PromQL, the time range, the response time, and the source label.
• Read-only. The skill never mutates state through the Prometheus MCP. Never creates dashboards / annotations / alerts (those would need create_* tools which are not in allowed-tools).

Known limitations

• The agent's MCP token may lack datasources:read. If mcp__prometheus_mcp__query_prometheus returns 403 datasources:read on <uid> for every Prometheus datasource UID, Phases 2-4 of this skill are blocked end-to-end. Workaround: the operator runs the exact PromQL produced in Phase 3 from an authenticated shell or the Prometheus web UI and pastes the response JSON into ${PROFILE_AND_OPTIMIZE_REPO_ROOT}/experiments/artifacts/prometheus-bundles/<run-id>/response.json. The skill's Phase 5 summary still works against the operator-pasted payload.

Source-of-truth references

• server/docs/mcp-composition.md - "Contention And O11y Pattern" (the workflow this skill generalizes).
• server/docs/perf-lake-contract.md - raw-payload provenance contract.
• docs/METHODOLOGY.md - measurement canon (full-context reporting, verdict rigor).