Datasketch resource deduplication starter
Skill ma-compbio-lab/SkillFoundry/skills/meta-maintenance/datasketch-resource-deduplication-starter
A framework for discovering, compiling, and validating reusable skills for scientific agents.From the repository description
npx -y skills add ma-compbio-lab/SkillFoundry --skill datasketch-resource-deduplication-starterAssembled from the repository path, not quoted from the project. Check it against their README if it does not work.
SKILL.md
0.3 KB, 76 tokens by cl100k_base, as published. Nobody here has run it
datasketch Resource Deduplication Starter
Use this skill to scan a tiny resource registry sample for near-duplicate text records with MinHash LSH.
What This Skill Does
- reads a toy JSONL resource list
- tokenizes title and summary text
- builds MinHash sketches with
datasketch - returns candidate duplicate pairs above a configurable threshold
What ships with it: 9 files
6.7 KB alongside SKILL.md, 2 of them executable
assets/
- datasketch_dedup_summary.json381 B
- README.md105 B
examples/
- README.md363 B
- toy_resources.jsonl506 B
scripts/
- run_datasketch_resource_deduplication.pyruns2.5 KB
tests/
- README.md138 B
- test_run_datasketch_resource_deduplication.pyruns1.3 KB
- metadata.yaml1.3 KB
- refs.md190 B