Find duplication

Find Duplication

Overview

Workflow for surfacing duplicate and near-duplicate code ("clones") and producing a merge plan. Default mode is LLM-driven (anchor-grep + structured reading); an optional tool-driven mode uses canonical open-source detectors for token-level precision. Like cleanup-stale-code, it enforces a two-phase workflow: produce a prioritized inventory of clone families first, then decide a merge strategy per family. The key insight is that detection and refactoring are separate steps with different failure modes — merging prematurely turns a typo into a behavior change.

When to Use

• "find duplication / dedupe code / DRY violations / repeated code" — see Step F1
• A codebase has accumulated copy-pasted helpers, copy-pasted error-handling wrappers, or structurally similar functions across modules
• Code review surfaces "this is the same as <other-file>" comments

When NOT to Use

• Refactoring a single function
• Third-party / vendor / generated code (node_modules, dist, build artifacts)
• Type-only duplication (two identical type definitions that exist because of structural or duck typing — usually a feature, not a bug)
• Change is ≤3 lines (do it inline)

General Workflow

Phase 1: Detect       →  cluster inventory of clone families (do not edit files)
Phase 2: Decide       →  merge strategy per family (extract / parameterize / delete / keep)
Phase 3: Execute      →  apply strategy, verify with tests/lint/typecheck

Do not skip Phase 1 even if "I can already see the duplicates." The inventory forces a per-family merge-strategy decision, which is where the work actually happens.

---

Detection Modes

Step F1 — Pick the detection mode

Default: LLM mode. Use anchor-grep + targeted reads. Cheap, no setup, works on any language.

Opt-in: Tool mode. Use a canonical clone detector for token-level precision on large codebases. The tool gives you a list of candidate clones; the LLM still has to classify and decide the merge.

Language family	Tool	Install / invocation
JavaScript / TypeScript	jscpd	npx jscpd <src-dir> --reporters json
Python	copydetect	pip install copydetect && copydetect <src-dir> --json
Java / Kotlin / C#	pmd-cpd	pmd cpd --language java --dir <src-dir>
Go	dupl	go install github.com/mibk/dupl@latest && dupl <src-dir>
Cross-language (token-based)	lizard	lizard <src-dir>

Use tool mode when:

• The codebase has > 50k lines (LLM context budget)
• Token-level diff matters (e.g. block-level, not just file-level)
• The CI pipeline already runs the tool and the report is the entry point

Use LLM mode when:

• The codebase is small or medium
• The team hasn't installed any detector
• The suspected clones are structural (same shape, different names) — tools miss these

Step F2 — LLM-mode anchor search

Search with anchors, not full reads. Patterns that signal a clone (adapt to the project's language):

• Same doc comment / docstring across files: identical multi-line doc blocks in two or more files
• Same constant block: identical sequences of constant definitions across files
• Same error-handling wrapper: identical error-guard or exception-handling shapes
• Same boilerplate accessor: identical getter/setter chains
• Same utility function copy: common helpers (date formatting, string truncation, slug generation, etc.) defined in multiple files
• Same imports block: identical import groups across siblings — often a sign that files were forked
• Same test fixture: identical mock objects or test data definitions in multiple test files

Use whatever text-search tool your environment provides (ripgrep, grep, IDE search). Read ±5 lines around each match. Do not full-read every file — it burns context with diminishing returns and finds off-scope issues.

Step F3 — Build the clone-family inventory

Group findings into clone families — the original + all of its copies. For each family, produce a row:

Field	Meaning
family_id	A short hash or label (e.g. clone-7a3b or format-date-v1)
anchor	The signature / first line / docstring that ties the family together
members	List of file:line ranges
size	Lines per member (smallest → largest)
drift	None (identical), minor (1–2 token diffs), structural (same shape, different names)
merge_strategy	TBD (decide in Step F5, simultaneously with priority)
priority	P0–P3 (decide in Step F4, simultaneously with strategy)

Important: a clone family is not the same as a single duplicated file. A family with 5 copies of the same 30-line block is one family, not five findings.

---

Priority Tiers

Step F4 — Output a prioritized table (fill priority and merge_strategy together)

Assign priority and pick a merge strategy at the same time — they are interdependent decisions, not sequential ones.

Priority	Criterion	Examples
P0	Identical ≥ 10 lines, zero drift, ≥ 2 members	Same 30-line config-loader block in 3 files
P1	Identical < 10 lines OR 1–2 token differences	Same 5-line string-slugging function in 2 files; same 20-line block with one variable renamed
P2	Structural clones (same shape, different names) — extraction candidate	Two auth-wrapper functions with the same lifecycle and different storage backends
P3	Similar intent, different implementation — usually keep	Two different mergeSort implementations in different modules

A finding is P0 only if removing it is unambiguously correct. If you need to think, it's P1 or lower.

Heuristic for sizing: clones that are 5+ lines and have ≥ 2 members are almost always worth acting on. Smaller clones (1–3 lines) and 2-member families are usually noise unless the line is a hot path.

---

Merge Strategies

Step F5 — Decide the merge strategy per family

For each P0/P1/P2 family, pick one strategy. Do not invent intermediate strategies; if none of these fit, the family is a P3 keep.

Strategy	When	Steps
Extract helper	All copies do the same thing with the same inputs/outputs	Create a new module (or extend an existing one) with the shared logic. Replace all call sites. Delete the duplicates.
Parameterize	Copies do the same thing but with different constants/types	Identify the varying axes. Replace copies with calls to one parameterized function.
Delete dead copy	One copy is the canonical one, the others have zero callers OR callers can switch trivially	Keep the canonical. Migrate callers. Delete the duplicates.
Keep all	Copies exist for a real reason (different teams, different lifecycles, intentional API duplication)	Mark as P3. Add a one-line comment in each copy explaining the deliberate duplication.

Important: if the user said the goal is to delete duplicated code, treat P0/P1 as Extract helper / Delete dead copy candidates by default, not as Keep all. Confirm before keeping a family that the user wanted removed.

Step F6 — Execute per family

Within a priority batch, parallelize edits across files. For extractions, do this in order:

1. Create the new shared module (with the unified implementation).
2. Replace one call site at a time. Verify the project still builds after each.
3. Delete the duplicate after the last call site is migrated.

For parameterizations, the same shape — but you may need to thread a parameter through call sites. Verify the parameter defaults to the original behavior to keep tests passing.

For delete-dead-copy, a bulk deletion is usually safe if the analyzer confirmed zero callers.

Step F7 — Verify (per priority batch)

After each priority batch completes, run the project's standard checks (formatter / linter / type checker / tests). Do not wait until all batches are done — verifying per batch makes it easier to isolate which family caused a failure. All checks must pass before moving to the next batch. After the final batch, re-run the detection pass (tool or LLM) and confirm the priority table is now empty for the strategies you applied.

Step F8 — Look for second-order cleanup

After merging, the new shared module may itself have dead code revealed (e.g. a parameter that turned out to be unused after all copies collapsed to one). Run cleanup-stale-code Section A against the new module's comments and Section B against its tests as a follow-up — but only as a follow-up, not as part of this task's scope.

---

Common Mistakes

Mistake	Fix
Inventing extra strategies the user didn't ask for	Stick to the four strategies above; ask if unclear
Full-reading every file to "be thorough"	Anchor greps + ±5 line context
Merging detection and refactoring	Always: inventory first, then decide strategy, then execute
Calling two implementations with different shapes "duplication"	If the behavior differs, it's not duplication; it's a polymorphism candidate
Extracting a helper that's only called once	The "extract" just moved code; you need a real second caller
Adding a P4 tier for "subjective" items	If user said 3 tiers, give 3 tiers
Confirming "duplicates are gone" without re-running detection	Always re-run after Phase 3
Treating tool output as ground truth	Tools report candidate clones; the LLM still classifies and decides
"I'll also rewrite the call sites for clarity"	Out of scope; the user asked to dedupe, not to refactor call sites

Anti-Patterns — Reject These

• "Let me also unify the doc comment style across all these files" → out of scope unless user asked
• "I noticed <file>:<line> has a real performance bug" → surface as a separate finding; don't expand scope
• "I'll do a comprehensive DRY sweep to be safe" → user asked for X, do X
• "Adding a 'possibly-duplicate' tier for ambiguous cases" → if you can't decide, mark P3 keep and explain why
• "I'll extract a base class to unify these" → if no inheritance relationship exists, the base class is just a container for shared code; extract a shared helper or utility instead
• "Let me also consolidate the test fixtures while I'm at it" → second-order task; do it as a follow-up

Common Workflow Endings

After Phase 3, report:

• Clone families: total count, count per priority tier, count per strategy
• Lines removed: total line count delta per file (always non-positive after dedup)
• Verification: lint + typecheck + test status
• Second-order findings: surface any dead code or comment/test drift revealed by the merge (propose a follow-up cleanup-stale-code pass)

Then ask if the user wants to commit. Do not auto-commit — refactoring is review-heavy and the user often wants to adjust the diff before it lands.

See also

• cleanup-stale-code — for cleaning up the comments and tests around the code you've just deduplicated
• consolidate-imports — for auditing import statements in the files you've just merged; deduplication often reveals import-style drift
• The four canonical detectors (jscpd, copydetect, pmd-cpd, lizard) are the de-facto standard for token-level detection; the LLM mode here is the fallback when the project hasn't installed any