agentsclimarketplace

Datadog recheck recovery

Skill NorthIsUp/errandd/skills/datadog-recheck-recovery

Re-check whether a firing Datadog monitor / the metric that tripped it has recovered, after attempting a remediation. Use when an incident-response agent needs to confirm a fix worked before resolving a page, or to decide whether to escalate. Trigger phrases include "did the alert recover", "re-check the monitor", "is the metric back to normal", "confirm recovery", "check monitor state", "has it recovered", "verify the fix cleared the alert".From its SKILL.md

Install
npx -y skills add NorthIsUp/errandd --skill datadog-recheck-recovery

Assembled from the repository path, not quoted from the project. Check it against their README if it does not work.

One thing to look at

  • 1 stars1 stars. Stars are a popularity signal and not a quality one, but at this level it is likely that nobody has read this closely except its author, and you would be relying on your own review.

SKILL.md

4.6 KB, as published. Nobody here has run it

Datadog — re-check recovery

Confirm, from Datadog itself, whether a monitor that triggered has returned to OK (and/or the underlying metric is back within threshold). This is the read step an incident-response flow runs after attempting a fix and before deciding to resolve the page or escalate.

Inputs you need

  • monitor_id — numeric id of the monitor that fired. It is in the alert payload ($ALERT_ID / alert_id / the monitor URL .../monitors/<id>).
  • Datadog site + credentials from the environment (do not hard-code):
    • DD_SITE (e.g. us3.datadoghq.com; API base is https://api.$DD_SITE)
    • DD_API_KEY, DD_APP_KEY

If the datadog MCP server is configured with the monitors + metrics toolsets, prefer its tools (get_monitor, query_timeseries) over raw curl — same data, no key handling. Fall back to the REST calls below when the MCP isn't available.

Step 1 — read the monitor's current state (authoritative)

GET https://api.$DD_SITE/api/v1/monitor/{monitor_id} and read overall_state:

curl -sf -H "DD-API-KEY: $DD_API_KEY" -H "DD-APPLICATION-KEY: $DD_APP_KEY" \
  "https://api.$DD_SITE/api/v1/monitor/$MONITOR_ID" \
  | jq '{state: .overall_state, name: .name, query: .query}'
overall_stateMeaning
OKRecovered — the fix worked. Safe to resolve/dismiss the page.
Alert / WarnStill firing — do not resolve; keep working or escalate.
No DataInconclusive — the metric stopped reporting. Treat as not recovered (a service that stopped emitting is not healthy); investigate or escalate.

Grouped monitors: a multi-group monitor is only truly clear when every alerting group is OK. Check state.groups (or overall_state_modified) and confirm the specific group named in the alert is OK, not just the rollup.

Step 2 — corroborate with the metric (guards against latch)

overall_state can lag or latch (e.g. absent-when-healthy series). Confirm the actual signal recovered by querying the monitor's own query over the last few minutes:

FROM=$(($(date +%s) - 300)); TO=$(date +%s)
curl -sf -H "DD-API-KEY: $DD_API_KEY" -H "DD-APPLICATION-KEY: $DD_APP_KEY" \
  "https://api.$DD_SITE/api/v1/query?from=$FROM&to=$TO&query=<monitor metric query>" \
  | jq '.series[].pointlist[-3:]'

Take the monitor's query, strip the <aggregation>(<window>): prefix and the > <threshold> comparator to get the bare metric query, and confirm the recent points are back on the healthy side of the threshold.

Step 3 — poll, don't assume

Recovery is rarely instant after a fix (pods reschedule, metrics roll up). Re-check on a short cadence with a ceiling — e.g. every 30s up to ~5 min:

  • Recovered (state OK and metric healthy for 2 consecutive checks) → hand off to datadog-resolve-page to explicitly dismiss the page.
  • Still firing at the deadline → hand off to datadog-escalate-page; do not silently give up.

Step 4 — render the recovered graph (optional, on recovery)

When the verdict is recovered, render the tripping metric as an image spanning the incident so the page thread shows the spike and the return to normal in one graph. Same bare metric query as Step 2; window from just before the trigger to now:

START=$(($(date +%s) - 1800)); END=$(date +%s)   # trigger-ish → now (~30m)
curl -sf -G -H "DD-API-KEY: $DD_API_KEY" -H "DD-APPLICATION-KEY: $DD_APP_KEY" \
  "https://api.$DD_SITE/api/v1/graph/snapshot" \
  --data-urlencode "metric_query=<bare metric query>" \
  --data-urlencode "start=$START" --data-urlencode "end=$END" \
  | jq -r '.snapshot_url'

The returned snapshot_url is a PNG that takes a few seconds to become available. It's a bonus — never block or fail recovery on it; if the call errors, just report the numeric points from Step 2 instead.

Output

Report: monitor id + name, final overall_state, the last few metric points, the recovery verdict (recovered | still-firing | no-data), the snapshot_url of the recovered graph (when rendered), and the next action taken.

Keep looking

Skills are one crate of 608,666. Ordering is by how many stacks a row turns up in, so the top of any crate is what has actually been picked rather than what has the most stars.