agentsclimarketplace
Sign in

Tuning detections to reduce false positives

Skill meltedinhex/analyst-ai-pack/skills/tuning-detections-to-reduce-false-positives

Tunes noisy detections by analyzing historical alert data to quantify false-positive sources, recommend allow-list filters and threshold adjustments, and measure precision impact. Activates for requests to tune a detection, reduce false positives, or analyze alert noise and recommend filters.From its SKILL.md

Install
npx -y skills add meltedinhex/analyst-ai-pack --skill tuning-detections-to-reduce-false-positives

Assembled from the repository path, not quoted from the project. Check it against their README if it does not work.

One thing to look at

  • 21 stars21 stars. Stars are a popularity signal and not a quality one, but at this level it is likely that nobody has read this closely except its author, and you would be relying on your own review.

What its file declares

Copied from the file, not written here

The file declares its own license as Apache-2.0. That is the author’s claim about this one file, and it is not the same thing as the license GitHub reports for the repository, which is listed with the other numbers below.

SKILL.md

2.7 KB, 396 tokens by cl100k_base, as published. Nobody here has run it

Tuning Detections to Reduce False Positives

When to Use

  • A detection generates excessive false positives and you want a data-driven tuning plan: which fields/values drive the noise, what filters to add, and the precision impact.
  • You are balancing recall against alert fatigue.

Do not use tuning that suppresses true positives — filters must target benign noise, not the malicious pattern. Always estimate the impact on true positives before applying.

Prerequisites

  • Labeled alert history (CSV/JSON) with at least an outcome/label field (true_positive vs false_positive) and the alert's distinguishing fields.

Workflow

Step 1: Quantify false-positive sources

python scripts/analyst.py analyze alerts.csv --label verdict --field ParentImage

Ranks field values by how much false-positive volume they drive and computes current precision/false-positive rate.

Step 2: Recommend filters

Suggest allow-list filters for values that are overwhelmingly benign (high FP, no/low TP), with the projected precision gain.

Step 3: Estimate impact

Report how many true positives would be lost (should be zero/near-zero for safe filters).

Step 4: Apply and monitor

Add the safe filters to the rule's filter/exclusion and continue monitoring precision.

Validation

  • FP-driving values are ranked by their false-positive contribution.
  • Recommended filters target values with negligible true-positive loss.
  • Projected precision and TP-loss are reported.

Pitfalls

  • Filtering a value that also carries true positives, blinding the detection.
  • Over-fitting to one time window's noise.
  • Tuning by gut feel instead of measured FP/TP contribution.

References

What ships with it: 3 files

4.8 KB alongside SKILL.md, 1 of them executable

references/

scripts/

Keep looking

Skills are one crate of 325,949. Ordering is by how many stacks a row turns up in, so the top of any crate is what has actually been picked rather than what has the most stars.