Generator

PromQL Query Generator

When to Use This Skill

Use this skill when you need to:

• Write a PromQL query for error rates, latency percentiles, or availability.
• Author recording rules or SLO burn-rate alerting rules.
• Aggregate metrics across labels using RED or USE monitoring patterns.
• Build queries for Prometheus counters, gauges, histograms, or summaries.

When NOT to use this skill:

• Validating or optimising an existing query — use promql-validator instead.
• Non-Prometheus query languages (LogQL, MetricsQL dialect specifics, SQL) — use the matching tool.
• Configuring Prometheus scrape targets or storage — that is server config, not query authoring.

Mindset

• Match the function to the metric type. rate() is for counters, histogram_quantile() for histograms; applying rate() to a gauge is meaningless.
• Error rate is a ratio. Express it as rated errors over rated totals across the same window, not as raw counts.
• Control cardinality on purpose. Choose by()/without() labels deliberately so a query cannot explode a dashboard.
• Alert on sustained burn. SLO alerts use multi-window burn rates and a for clause, not a single instantaneous threshold.

Interactive Query Planning Workflow

CRITICAL: Always engage the user in collaborative planning before generating any query. Never skip the planning phase.

Workflow (7 stages)

1. Understand the goal — Ask what the user wants to monitor (request rate, error rate, latency, resource usage, availability, SLO tracking) and the use case (dashboard, alert, recording rule, ad-hoc).
2. Identify metrics — Confirm metric names, types (counter/gauge/histogram/summary), and relevant labels. Suggest common naming patterns if uncertain.
3. Determine parameters — Confirm time range, label filters, aggregation, and thresholds. If the user already specified values (e.g., "5-minute window", "> 5% error rate"), acknowledge them as pre-filled defaults and allow quick confirmation rather than re-asking.
4. Present the query plan — Before writing any code, present a plain-English plan (goal, query structure, expected output, example interpretation) and ask for confirmation via AskUserQuestion with options: "Yes, generate this query" / "Modify [aspect]" / "Show alternatives".
5. Generate the query — Once confirmed, read the relevant reference file(s) before writing code, cite the applicable pattern, and apply the best practices below.
6. Validate — Automatically invoke devops-skills:promql-validator. Display structured results (syntax, best practices, explanation). Fix any issues and re-validate until all checks pass.
7. Deliver — Provide the final query, plain-English explanation, usage instructions (dashboard / alert / recording rule), customization notes, and related query suggestions.

Ask vs. Infer: If the user's request already clearly specifies goal, use case, and context, acknowledge those details instead of re-asking. Only ask for missing or ambiguous information.

---

Best Practices for Query Generation

Always consult the relevant reference file before writing code.

Scenario	Reference File
Histogram queries	references/metric_types.md (Histogram section)
Error/latency patterns	references/promql_patterns.md (RED section)
Resource monitoring	references/promql_patterns.md (USE section)
Optimization / anti-patterns	references/best_practices.md
Specific functions	references/promql_functions.md

Key Rules

1. Always add label filters — reduces cardinality and improves performance.
2. Match functions to metric types — rate()/increase() on counters; *_over_time() or direct use for gauges; histogram_quantile() for histograms.
3. Prefer by()/without() on all aggregations.
4. Prefer exact label matches over regex when the value is known.
5. Use recording rules for queries that are expensive or reused frequently (naming: level:metric:operations).
6. Format multi-line for complex queries.

Core Patterns

# Request rate (counter)
sum(rate(http_requests_total{job="api-server"}[5m])) by (endpoint)

# Error rate ratio
sum(rate(http_requests_total{job="api-server", status_code=~"5.."}[5m]))
/ sum(rate(http_requests_total{job="api-server"}[5m]))

# P95 latency (classic histogram)
histogram_quantile(0.95,
  sum by (le) (rate(http_request_duration_seconds_bucket{job="api-server"}[5m]))
)

# P95 latency (native histogram, Prometheus 3.x+)
histogram_quantile(0.95,
  sum by (job) (rate(http_request_duration_seconds[5m]))
)

# Availability
(count(up{job="api-server"} == 1) / count(up{job="api-server"})) * 100

# Burn rate (99.9% SLO, 1h window)
(
  sum(rate(http_requests_total{job="api", status_code=~"5.."}[1h]))
  / sum(rate(http_requests_total{job="api"}[1h]))
) / 0.001

# Multi-window burn-rate alert (page: 2% budget in 1h, burn rate 14.4)
(
  sum(rate(http_requests_total{job="api", status_code=~"5.."}[1h]))
  / sum(rate(http_requests_total{job="api"}[1h]))
) > 14.4 * 0.001
and
(
  sum(rate(http_requests_total{job="api", status_code=~"5.."}[5m]))
  / sum(rate(http_requests_total{job="api"}[5m]))
) > 14.4 * 0.001

For complete SLO patterns, Native Histogram functions (histogram_count, histogram_sum, histogram_fraction), subqueries, offset/@ modifiers, vector matching, and Kubernetes patterns — see the assets/ files.

---

Validation Checklist

After generating, invoke devops-skills:promql-validator and display results in this format:

## PromQL Validation Results

### Syntax Check
- Status: ✅ VALID / ⚠️ WARNING / ❌ ERROR
- Issues: [list any syntax errors]

### Best Practices Check
- Status: ✅ OPTIMIZED / ⚠️ CAN BE IMPROVED / ❌ HAS ISSUES
- Issues: [list problems found]
- Suggestions: [list optimizations]

### Query Explanation
- What it measures: [plain English]
- Output labels: [label list or "None (scalar)"]
- Expected result structure: [instant vector / scalar / etc.]

Fix all issues and re-validate until clean.

---

Alerting and Recording Rule Snippets

# Alerting rule with for clause
alert: HighErrorRate
expr: |
  (
    sum(rate(http_requests_total{status_code=~"5.."}[5m]))
    / sum(rate(http_requests_total[5m]))
  ) > 0.05
for: 10m

# Recording rule (naming: level:metric:operations)
- record: job:http_requests:rate5m
  expr: sum by (job) (rate(http_requests_total[5m]))

---

Anti-Patterns

NEVER use rate() on a gauge metric

• WHY: rate() computes per-second rate of increase and assumes monotonically increasing counters. Applied to a gauge, it produces nonsensical results because gauges can decrease.
• BAD: rate(node_memory_MemFree_bytes[5m]) — memory is a gauge
• GOOD: node_memory_MemFree_bytes (direct use) or delta(node_memory_MemFree_bytes[5m]) for change over time

NEVER query a histogram with avg() across quantile labels

• WHY: Summary metrics expose pre-aggregated {quantile="0.95"} labels that cannot be re-aggregated across instances. Using avg() on them produces statistically meaningless results.
• BAD: avg(http_request_duration_seconds{quantile="0.95"})
• GOOD: histogram_quantile(0.95, sum by (le) (rate(http_request_duration_seconds_bucket[5m]))) — use histogram type with histogram_quantile()

NEVER use a high-cardinality label in by() without filtering first

• WHY: Aggregating by user_id, request_id, or pod without label filters produces thousands of series, overwhelming dashboards and recording rules.
• BAD: sum by (user_id) (rate(http_requests_total[5m]))
• GOOD: sum by (job, status_code) (rate(http_requests_total{job="api"}[5m])) — filter and aggregate on stable, low-cardinality labels

NEVER use increase() for alerting thresholds

• WHY: increase() is extrapolated and can return non-integer values on sparse counters. For alerting, rate() produces stable per-second thresholds that are scrape-interval independent.
• BAD: increase(http_requests_total{status=~"5.."}[5m]) > 10
• GOOD: rate(http_requests_total{status=~"5.."}[5m]) > 0.033 (2 errors/minute = 0.033/second)

NEVER omit the for clause on alert rules

• WHY: Alerts without for fire immediately on a single evaluation, causing false positives from transient spikes. The for clause requires the condition to be true for a sustained period.
• BAD: alert: HighErrorRate with expr: error_rate > 0.05 and no for
• GOOD: Add for: 5m to require the condition to hold for 5 minutes before firing

Documentation Lookup

1. context7 MCP (preferred): resolve prometheus, then fetch docs with the relevant topic.
2. Fallback WebSearch: "Prometheus PromQL [function/operator] documentation [version] examples"

---

Error Handling Quick Reference

Symptom	Likely Cause	Fix
Empty results	Wrong label filters or metric not scraped	Check up{job="..."}, verify label values
Too many series	High cardinality	Add label filters, aggregate, use recording rules
Wrong values	Wrong function for metric type	rate() on counters; direct or *_over_time() on gauges
Slow queries	Large range vectors or missing filters	Narrow time range, add filters, use recording rules

---

References

Internal:

• PromQL Functions — all PromQL functions with examples; read for specific function syntax questions
• PromQL Patterns — RED/USE method patterns, alerting rules, recording rules; read for standard monitoring patterns
• Best Practices — anti-patterns, performance optimization, cardinality management; read when optimizing queries
• Metric Types — counter/gauge/histogram/summary guide; read to confirm correct function choice
• Common Queries — reusable request rate, error rate, latency, and availability query templates
• RED Method — complete RED method (Rate, Errors, Duration) implementation
• USE Method — complete USE method (Utilization, Saturation, Errors) implementation
• SLO Patterns — SLO, error budget, burn rate, and multi-window alerting patterns
• Alerting Rules — example alerting rules with thresholds and for clauses
• Recording Rules — example recording rules with level:metric:operations naming
• Kubernetes Patterns — kube-state-metrics, cAdvisor, and vector matching examples