agentsclimarketplace
Sign in

Source validation

Skill maxamillion/claude-oss-eval-plugin/skills/source-validation

Claude Code skills to evaluate open source projectsFrom the repository description

Install
npx -y skills add maxamillion/claude-oss-eval-plugin --skill source-validation

Assembled from the repository path, not quoted from the project. Check it against their README if it does not work.

One thing to look at

  • 1 stars1 stars. Stars are a popularity signal and not a quality one, but at this level it is likely that nobody has read this closely except its author, and you would be relying on your own review.

SKILL.md

8.2 KB, as published. Nobody here has run it

Source Validation Skill

Purpose

Ensure accuracy and recency of information sources during OSS evaluation. This skill provides protocols for verifying source credibility, data freshness, and multi-source validation.

Activation

This skill is critical during:

  • All phases requiring quantitative data (stars, downloads, versions)
  • Phase 4: Licensing (license information must be current)
  • Phase 5: Community Health (metrics must be current)
  • Phase 11: Operational Costs (pricing must be current)
  • Any phase using web search for research

Source Hierarchy

Tier 1: Primary Sources (Highest Authority)

## Primary Sources

| Source Type | Example | Reliability |
|-------------|---------|-------------|
| Official repository | github.com/<org>/<repo> | Highest |
| Official documentation | docs.<project>.io | Highest |
| Official package registry | npmjs.com, pypi.org | Highest |
| Project announcements | Official blog, GitHub releases | High |
| Maintainer statements | Verified maintainer comments | High |

Usage Rules:

  • Always prefer primary sources
  • Link directly to evidence
  • Note version/date of information

Tier 2: Secondary Sources (Verified Third-Party)

## Secondary Sources

| Source Type | Example | Reliability |
|-------------|---------|-------------|
| Reputable benchmarks | TechEmpower, js-framework-benchmark | High |
| Conference talks | Official recordings | High |
| Established tech blogs | Company engineering blogs | Medium-High |
| Security advisories | CVE database, Snyk | High |
| Stack Overflow (answers) | Highly-voted answers | Medium |

Usage Rules:

  • Verify with primary source when possible
  • Note publication date
  • Cross-reference with other sources

Tier 3: Community Sources (Unverified)

## Community Sources

| Source Type | Example | Reliability |
|-------------|---------|-------------|
| Blog posts | Medium, Dev.to | Variable |
| Reddit discussions | r/<framework> | Low-Medium |
| Twitter/Social | Developer tweets | Low |
| Forum posts | Unverified claims | Low |
| AI-generated content | ChatGPT, etc. | Do not use |

Usage Rules:

  • Use only for qualitative signals
  • Never use for quantitative data
  • Always seek primary verification
  • Note as "community sentiment" not "fact"

Data Freshness Requirements

Critical Data (Must Be Current)

## Data Requiring Real-Time Verification

| Data Type | Max Age | Verification Method |
|-----------|---------|---------------------|
| Version numbers | Real-time | WebSearch → repo/registry |
| Pricing | Real-time | WebSearch → pricing page |
| License | Real-time | WebSearch → LICENSE file |
| Stars/Forks | 24 hours | GitHub API or web |
| Download counts | 7 days | Registry API |
| Last commit | 24 hours | Repository check |
| Open issues | 7 days | Issue tracker check |

Standard Data (Moderate Freshness)

## Data with Moderate Freshness Needs

| Data Type | Max Age | Notes |
|-----------|---------|-------|
| Benchmarks | 6 months | Check for newer versions tested |
| Feature documentation | 1 month | Verify version matches |
| Tutorial content | 3 months | Check still applies |
| Community health | 1 month | Trends matter |

Stable Data (Long-Term Valid)

## Data with Long-Term Validity

| Data Type | Max Age | Notes |
|-----------|---------|-------|
| Architectural patterns | Years | Core design rarely changes |
| Historical milestones | Permanent | Facts don't change |
| Design philosophy | Years | Unless pivot announced |

Verification Protocol

Quantitative Data Verification

For any number (stars, downloads, version, price):

## Verification Record

**Claim**: <the data point>
**Source**: <where obtained>
**Verification Date**: <timestamp>
**Verification Method**: <how verified>
**Result**: <confirmed value>
**Status**: ✅ Verified / ⚠️ Unable to verify / ❌ Contradicted

Example:
**Claim**: "FastAPI has 70k+ GitHub stars"
**Source**: Initial AI knowledge
**Verification Date**: 2025-01-16
**Verification Method**: WebSearch → github.com/tiangolo/fastapi
**Result**: 78,234 stars
**Status**: ✅ Verified (updated to current)

Qualitative Claim Verification

For subjective claims (best practices, recommendations):

## Claim Verification

**Claim**: "<qualitative statement>"
**Original Source**: <where claim originated>

**Corroborating Sources**:
1. <source 1> - <how it supports>
2. <source 2> - <how it supports>

**Contradicting Sources**:
1. <source if any> - <contradiction>

**Verification Status**:
- ✅ Well-supported (3+ corroborating)
- ⚠️ Mixed evidence
- ❌ Contradicted
- ❓ Unverifiable

Multi-Source Validation

For critical decisions, require multiple sources:

## Multi-Source Requirement Matrix

| Decision Impact | Required Sources | Source Tier |
|-----------------|------------------|-------------|
| Recommendation change | 3+ sources | At least 2 Tier 1 |
| Feature claim | 2+ sources | At least 1 Tier 1 |
| Risk assessment | 2+ sources | Mix of Tier 1-2 |
| Community sentiment | 3+ sources | Any tier |
| Pricing/licensing | 1 source | Must be Tier 1 |

WebSearch Best Practices

Effective Search Patterns

## Search Query Templates

### Version/Release Info
- `<framework> latest version release`
- `<framework> changelog 2024 2025`
- `site:github.com/<org>/<repo>/releases`

### Pricing/Licensing
- `<product> pricing 2024 2025`
- `<project> license change announcement`
- `site:<product>.com/pricing`

### Community Health
- `<framework> maintenance status`
- `<framework> future roadmap`
- `"<framework>" "maintainers" OR "contributors" 2024`

### Security
- `<framework> CVE`
- `<framework> security vulnerability 2024`
- `site:snyk.io <framework>`

### Performance
- `<framework> benchmark 2024`
- `<framework> vs <competitor> performance`

Search Result Evaluation

## Result Quality Checklist

For each search result, assess:

- [ ] Publication date visible and recent
- [ ] Author/source credibility established
- [ ] Content matches current version
- [ ] No obvious bias (e.g., competitor hit piece)
- [ ] Specific evidence provided, not just claims

If < 3 checked, seek additional sources

Handling Conflicting Information

When sources disagree:

## Conflict Resolution Protocol

### Step 1: Identify Conflict
- Source A says: <claim>
- Source B says: <contradicting claim>

### Step 2: Assess Source Quality
| Criterion | Source A | Source B |
|-----------|----------|----------|
| Tier | X | Y |
| Date | <date> | <date> |
| Specificity | High/Med/Low | High/Med/Low |
| Evidence | <type> | <type> |

### Step 3: Resolution
- [ ] More recent source wins (if data could change)
- [ ] Higher tier source wins (if equal recency)
- [ ] More specific source wins (if equal tier)
- [ ] Document both if unresolvable

### Step 4: Documentation
**Resolved Claim**: <what we're using>
**Confidence**: High/Medium/Low
**Note**: <any caveats about conflicting info>

Citation Format

Standard citation format for evaluation documents:

## Citation Formats

### Inline Citation
"FastAPI supports async/await natively" [FastAPI Docs, 2025]

### Verification Citation
Stars: 78,234 [Verified: 2025-01-16, github.com/tiangolo/fastapi]

### Full Reference
[FastAPI Docs, 2025]: https://fastapi.tiangolo.com/async/
Retrieved: 2025-01-16
Version documented: 0.109.0

Red Flags

Watch for these source reliability issues:

## Source Red Flags

### Immediate Disqualification
- AI-generated content (including other AI assistants)
- Unverified social media claims
- Anonymous sources
- Sources with obvious commercial bias

### Requires Additional Verification
- Blog posts without dates
- Content older than 1 year
- Single-source claims for critical data
- Community forums without official confirmation

### Acceptable with Caveats
- Dated content with "as of <date>" note
- Community sentiment with "anecdotal" qualifier
- Older benchmarks with version context

Integration with Bias Prevention

This skill directly addresses:

  • Pitfall 1: Stale/Outdated Knowledge (verification protocol)
  • Pitfall 6: Marketing Language Adoption (source hierarchy)
  • Pitfall 9: Confirmation Bias (multi-source validation)

Keep looking

Skills are one crate of 628,971. Ordering is by how many stacks a row turns up in, so the top of any crate is what has actually been picked rather than what has the most stars.