Autogen promql tests

autogen-promql-tests

Coverage auditor and test-skeleton generator for Prometheus rule files. It does not know your actual metric shapes — it emits a promtool-compatible test file full of TODO placeholders that a human (or an agent that has read the real metrics) must fill in with realistic input series and expected values. Hermetic — no network access.

When to use this skill

• A rules file has alerts or recording rules with no corresponding promtool test rules coverage, and you want a starting skeleton rather than writing input_series/exp_alerts blocks from scratch.
• You want a quick coverage report: how many of N rules currently have tests, and which ones don't.
• This is a bootstrap tool, best run interactively the first time a rule is added — not a CI gate, since its generated output is intentionally incomplete (TODO markers) and would fail promtool test rules until a human fills them in.

Setup

go build -o bin/autogen-promql-tests ./cmd/autogen-promql-tests
# or: go install github.com/conallob/o11y-analysis-tools/cmd/autogen-promql-tests@latest

Usage

autogen-promql-tests [options]

Flag	Default	Effect
--rules	(required)	Path to the Prometheus rules YAML file to audit.
--tests	""	Path to an existing test file to check coverage against. If omitted, the tool auto-discovers <rules-basename>_test.yml next to the rules file.
--fix	false	Generate a test file covering the untested alerts/rules.
--verbose	false	Print discovery/loading detail.

Typical invocations

# Coverage report only (exit 1 if anything is untested)
autogen-promql-tests --rules=./alerts.yml

# Generate a test skeleton for whatever's untested
autogen-promql-tests --rules=./alerts.yml --fix

# Check against an explicit, non-default-named test file
autogen-promql-tests --rules=./rules.yml --tests=./custom_tests.yml

Without --tests, it looks for <rules>_test.yml (rules file's basename with .yml/.yaml stripped, _test.yml appended) in the same directory. With --fix, the same path is the default write target unless --tests is also given, in which case that path is used both to read and write.

What the generated file contains

For every rule not already covered (alert: or record: name absent from the tested set), it appends four test cases to the output YAML:

1. True positive — input_series tuned (via a TODO) to make the alert fire; asserts exp_alerts with the rule's actual labels: and a TODO placeholder for each annotations: template value.
2. False positive — same shape, but TODO'd to not trigger the condition; asserts exp_alerts: [].
3. Hysteresis check — only emitted if the rule has a for: duration: evaluates at roughly half the for: duration and asserts the alert has not fired yet, to test the hold-down timer itself.
4. Edge cases — an empty placeholder section reminding the human to add boundary values, missing-metric behavior, label-combination cases, and recovery behavior.

The input_series blocks always start from a single placeholder example_metric{job="test", instance="localhost:9090"} series with a TODO — the tool does not parse the expression's actual metric names into realistic series, it only echoes the first line of the expression as a comment reminder. An agent should replace these placeholders using the real metric names and label sets from the rule's expr:.

Reading the output

═══════════════════════════════════════════════════════════
Test Coverage Analysis
═══════════════════════════════════════════════════════════

Total rules/alerts: 8
Tested: 5
Untested: 3

Untested alerts/rules:
  • HighErrorRate
  • job:http_requests:rate5m
  • LowDiskSpace

Run with --fix to generate tests for untested alerts

Exit code 1 when untested rules exist and --fix wasn't passed, 0 otherwise (including after a successful --fix run).

Agent workflow

1. Run without --fix first to see the coverage gap and decide scope.
2. Run with --fix to generate the skeleton file.
3. Do not stop there. Open the generated _test.yml and replace every TODO with real series data derived from the rule's actual expr: — correct metric names, label sets that match what the alert's labels: expects, and values that genuinely cross (or stay under) the alerting threshold. The file will not pass promtool test rules until this is done.
4. Verify with promtool test rules <generated-file> once filled in.
5. Re-run autogen-promql-tests --rules=... --tests=<generated-file> to confirm the rule now shows as tested.