Verify forecasting docs

Skill lightning-rod-labs/lightningrod-python-sdk/skills/verify-forecasting-docs

Verify the forecasting docs (quickstart.md, sdk.md, openai.md) by executing every Python code snippet and checking that the surrounding prose accurately describes the actual behavior. Use when docs have changed, when the SDK has changed, or when a new API feature is added.From its SKILL.md

Install

npx -y skills add lightning-rod-labs/lightningrod-python-sdk --skill verify-forecasting-docs

Assembled from the repository path, not quoted from the project. Check it against their README if it does not work.

SKILL.md

2.9 KB, as published. Nobody here has run it

Verify Forecasting Docs

Two-phase verification of docs/forecasting/: run every code snippet against the live API, then check that the prose around each snippet matches observed behavior.

Setup

Ask the user for an API key if one hasn't been provided. The project .env may point to staging — bypass it so snippets run against production:

LIGHTNINGROD_BASE_URL="https://api.lightningrod.ai/v1" python ...

Activate the venv before running any Python:

source venv/bin/activate

Phase 1: Execute every code snippet

Read each doc file in turn. For every fenced Python block, run it verbatim (substituting the real API key for "your-api-key"). Capture stdout and any exception.

Cover all files under docs/forecasting/** in the order they are listed in SUMMARY.md.

For each snippet record:

file and approximate line range
exit status (pass / error / exception type + message)
stdout (first 300 chars is enough)

Phase 2: Semantic validation

For each snippet that passed, check the prose immediately before and after it against what you actually observed. Flag any mismatch as a finding.

Common things to check:

Prose claim	How to verify
"field X is populated when answer_type=Y"	Confirm the field is non-None in the output
"answer_type='auto' raises ValueError in predict()"	Run it and confirm the error message matches
Parameter descriptions in tables (type, default, description)	Cross-check against `client.py` and `prediction.py` source
Response field table in sdk.md	Confirm each field is present on the returned `PredictionResult`
Cost/token counts mentioned	Confirm `usage` fields are present and non-zero
Source objects have `url` and `title`	Confirm with a research call
`answer_type='auto'` note says it raises `ValueError`	Run it and confirm

Flag findings as:

⚠️ wrong — prose makes a claim that contradicts observed behavior
⚠️ stale — prose describes something that no longer exists (removed field, renamed param)
📝 missing — behavior exists but isn't documented

Report format

## Doc Verification: forecasting docs

**Verdict:** PASS | FAIL

### Snippet results
| # | File | Description | Result |
|---|------|-------------|--------|
| 1 | quickstart.md | OpenAI client minimal | ✅ pass |
| 2 | sdk.md | lr.predict() minimal | ✅ pass |
...

### Semantic findings
- ⚠️ <finding>
- 📝 <finding>
(empty = all prose checked out)

FAIL if any snippet raises an unexpected exception, or if any ⚠️ finding is found. 📝 findings don't fail but should be noted.