Messy archive curator

Messy Archive Curator

Turn a sprawling, machine-generated dump (scraped site, course export, doc archive — often handed over as a .zip) into a small, deliberately-structured set of Markdown files a human would actually want to read, optionally filtered down to what a specific audience actually needs, and handed back as a single zip. The core moves are: stop mirroring the source's file structure, throw away everything that isn't real content (including content that's real but irrelevant to the reader), and ship the result as one clean archive.

This was reverse-engineered from a real before/after pair: an MIT OCW course site export (~300 files: data.json + index.html pairs, hash-named PDFs/VTTs, CSS/JS/fonts, duplicate resource variants) was turned into 5 Markdown files (syllabus.md, calendar.md, readings.md, assignments.md, recommended_books.md) plus a slides/ folder with cleanly renumbered PDFs. Apply the same philosophy to any similarly messy input, adapting the specific output files to whatever content the archive actually contains, and adapting the filtering pass to whoever the curated archive is actually for.

Step 0 — Unpack the input

If the input is a .zip (or .tar, .tar.gz, etc.) rather than an already-extracted folder, extract it first into a scratch working directory before doing anything else. If there are multiple zips, or zips nested inside zips (a zip of zips, or a zip containing per-unit zips), extract all of them recursively until you're looking at a flat tree of real files — don't leave any archive unopened on the assumption it's "probably just assets." Note the original top-level archive name; it's a reasonable default for naming the final output zip in Step 8.

Step 1 — Inventory before touching anything

view the directory tree first (or find/ls -R for huge trees). Do NOT start writing output files yet. You're looking for:

• Real content clusters: pages describing structure/schedule (syllabus, overview, README, index), per-unit content (sessions/chapters/articles/tickets), reference lists (readings, bibliography, links, FAQ), and deliverables (assignments, tasks, specs).
• Structural noise to ignore entirely: site chrome (nav/footer/css/js/fonts/mathjax/analytics), duplicate format variants of the same resource (e.g. the same lecture exposed as a video page, a transcript page, and a notes page — these are redundant, pick the one with the actual substance), search/filter pages, sitemap/robots files, and admin/UUID-named junk.
• Asset files: PDFs, images, etc. — usually hash-prefixed or UUID-named, scattered across multiple folders, often duplicated (the same file linked from 2-3 different generated pages).

Build a quick mental (or literal, in a scratch file) map: source location → which output file this content belongs in. If the archive is large, grep for a few candidate keywords (e.g. "syllabus", "schedule", "readings", "assignment") to find the highest-signal pages first rather than reading everything linearly.

Step 2 — Extract real text AND real links — don't trust just one source

Generated archives frequently split content across two parallel representations:

• A stripped-text field (e.g. a JSON content field) — readable prose, but links and structure are gone, and HTML entities (’, &, +, non-breaking spaces) are often left undecoded.
• The raw HTML (index.html) — has the real <a href="..."> links, but buried in heavy markup/nav noise.

Always cross-reference both when a content cluster has hyperlinks (citations, "read more" links, resource links). Pull clean prose from the text field, but recover URLs by grepping the matching HTML for the anchor tags whose visible text matches. Never invent or guess a URL — if you can't find it in the source, render that item as plain text without a link rather than fabricating one.

Clean every piece of extracted text:

• Decode HTML entities (& → &, ’/‘ → straight ', + → +,  /\u00a0 → normal space).
• Normalize curly/smart quotes to straight quotes for titles unless the source consistently uses a different convention.
• Normalize dashes: en dash – for ranges ("pages 3–21", "July 16–20"), not hyphens.
• Collapse run-on text. Source text extraction often glues a heading directly onto the paragraph that follows it (no whitespace boundary) — actively look for and split these seams; don't reproduce a wall of unbroken text.

Step 3 — Filter for the actual reader, not just for structural noise

Structural noise (Step 1) is junk regardless of who's reading. This step is different: it's about real, well-formed content that is nonetheless useless to the specific person who will read the output, and should be cut just as ruthlessly.

The default audience to assume, unless told otherwise, is a student studying the material on their own, after the fact — not someone currently enrolled and attending. Under that lens, cut anything whose only purpose is logistics for attending a live offering rather than learning the material:

• People/contact logistics: instructor/TA office hours, office room numbers, email addresses and phone numbers for staff, "who to contact for X" sections.
• Place/time logistics: classroom/building locations, exam room assignments, lecture meeting times tied to a specific past term, recitation/section sign-up info.
• Administrative boilerplate: late-submission policy mechanics, regrade-request procedures, academic-integrity/honor-code statements, accessibility/accommodation statements, parking/building-access notes, syllabus-change disclaimers.
• Term-bound dates with no learning value once the course has ended: "due Friday at 5pm," specific past-semester exam dates — unless the relative order or pacing of units is itself useful information, in which case keep the structure (e.g. "Unit 6 precedes Unit 7") but drop the wall-clock specifics.

Do not cut content just because it's administrative-sounding if it actually carries learning value — e.g. a grading rubric that reveals what's emphasized, or a prerequisite list, can stay if it helps a self-studying reader calibrate. When in doubt, ask: if this person is studying alone, months or years after this was offered, does this sentence help them learn the subject, or does it only help them show up to a room on time? Cut the latter.

If the user names a different audience (e.g. "for new hires onboarding," "for a teammate picking this up"), re-derive this list for that audience instead of defaulting to the self-study case — the principle (cut what's irrelevant to this reader, keep what helps them) transfers; the specific list of things to cut does not.

Step 4 — Re-organize by topic, not by source structure

This is the most important judgment call. The source's file/folder layout reflects how it was generated, not how a reader wants to consume it. Decide the output file set based on the kinds of content actually present (after Step 3's filtering), e.g.:

If the archive contains...	Consider a file like...
Course/program overview, instructor, grading breakdown	syllabus.md
A schedule of dated units/sessions/milestones	calendar.md (table: unit #, topic, key dates)
Per-unit reading lists + discussion/study questions	readings.md (one section per unit)
Deliverables, homework, project specs	assignments.md
A curated list of external books/resources	recommended_books.md / resources.md
Slide decks, handouts, per-unit attachments	a flattened asset folder (e.g. slides/) referenced from the relevant content file

Don't force content into a file it doesn't belong in, and don't be afraid to deviate from this exact list — a software-docs archive might instead want architecture.md, setup.md, faq.md. The principle transfers even when the file names don't: few files, each with a single clear purpose, each named for what a reader is looking for. If Step 3 filtering removed an entire category (e.g. no logistics content survives), don't create a near-empty file for it — just don't produce that file.

Within a content file that has repeating per-unit structure (sessions, chapters, tickets), use one consistent template per unit, separated by ---, e.g.:

## Session N · Topic Title

**Study Questions**
1. ...
2. ...

**Readings**
- '[Title](url)' Author (Date) (pages X–Y, if applicable)
- 'Title with no recoverable link' Author (Date)

**Lecture Slides** · [Topic Title (PDF)](slides/N.pdf)

---

Adapt the bold-label sub-sections to whatever the unit actually contains — don't include an empty "Study Questions" header if that unit has none.

Step 5 — Rename and flatten assets, verifying ambiguous cases by content

Source archives usually name binary assets with hashes/UUIDs (ed8cf9250a04f1b9be6fb0d582c79db4_MIT15_S12F18_ses6.pdf) scattered across many folders, sometimes with the same file duplicated under multiple resource entries (video/transcript/notes variants of one lecture).

• Copy each unique asset once into a single flat output folder (e.g. slides/), renamed to something short and meaningful to the new structure — usually the unit number (6.pdf), with a letter suffix for sub-parts (6a.pdf for a second deck in the same session).
• When metadata/title alone doesn't disambiguate which asset is which (e.g. two PDFs both vaguely titled "Session 6"), don't guess from the filename — open/view the actual file content (or enough of it) to confirm which is the guest-lecture deck vs. the regular lecture deck, then name and caption it accordingly. Trustworthy disambiguation beats fast guessing.
• Drop assets that are pure noise: site icons/logos, font files, captions/transcripts that duplicate a more substantive page, thumbnail images.

Step 6 — Formatting conventions for the output Markdown

Keep a consistent, plain, scannable style across all output files:

• # Title at the top of each file, then a --- rule before the body.
• ## headers for major sections or repeating units; --- between repeating units (sessions/chapters) for visual separation; no --- needed between unrelated top-level sections that already have clear ## headers if the file is short.
• Bold labels (not headers) for sub-parts within a unit (**Study Questions**, **Readings**, **Lecture Slides**) — this keeps the heading hierarchy shallow and the table of contents clean.
• Tables for genuinely tabular data (calendar/schedule, grading breakdown) instead of bullet lists.
• Citation-style bullets for reading/reference lists: '[Title](url)' Author (Date) (page info) — title in straight quotes, linked if a URL was recoverable, author and date outside the quotes, optional page/section info in trailing parens.
• Cross-link between output files with relative Markdown links ([Assignments section](./assignments.md)) instead of repeating content.
• One blank line between elements; no trailing site-chrome, no leftover "Skip to content"/nav artifacts, no broken/relative links pointing back into the discarded source structure, and no orphaned logistics fragments left over from Step 3 (e.g. a dangling "Contact:" label with the email removed but the label still there).

Step 7 — Final pass

Before packaging the result:

• Skim every output file top to bottom as a reader would — if a sentence still reads like raw scraped text (run-on, weird entity remnants, mid-word breaks), fix it.
• Re-check against Step 3's filter: skim once specifically hunting for missed logistics (a stray office hour, a building name, a leftover phone number) rather than relying on having caught it all the first pass.
• Spot-check a sample of links actually resolve to what the anchor text claims.
• Confirm the file count is small and each file's purpose is obvious from its name — if you've produced something close to a 1:1 mirror of the source's folder count, you haven't actually restructured it, go back to Step 4.
• Discard (don't copy into output) anything that was pure source scaffolding: original JSON, original HTML, CSS/JS/fonts, duplicate resource variants, sitemap/robots, etc. The output should be a fraction of the source's file count.

Step 8 — Package the output as a single zip

The deliverable is one zip file, not a loose folder of files for the user to gather themselves:

• Put every output file (the Markdown files plus any asset folders like slides/) inside one top-level folder named for the source content (e.g. mit-15_s12-f18/), then zip that folder — so unzipping produces one tidy top-level directory, not a scatter of files at the zip root.
• Don't include any scratch/intermediate files (the unpacked source tree from Step 0, notes-to-self, etc.) inside the final zip — only the curated output.
• Name the zip after the source archive or course/topic, not something generic like output.zip.
• Present that single zip as the result. If the user asked for filtering (Step 3) on top of restructuring, briefly note in your reply what categories of content were removed (e.g. "removed office hours, room numbers, and contact info — kept all readings, assignments, and study questions") so they can sanity-check the cut without having to diff the archives themselves.

Reference

See references/worked-example.md for the full before/after of the MIT OCW case this skill was derived from, useful as a calibration example when in doubt about formatting density, how aggressively to restructure, or what counts as audience-irrelevant logistics to cut.