Pdf to epub

pdf-to-epub

Convert a text-native PDF book into a faithful EPUB. Built and validated against a real 350-page Vietnamese book (Chuyện Với Thanh, Nguyễn Thành Nam) with chapters, dialogue, ~100 footnotes, and 20 embedded photos; table detection was later validated against a 30-page Vietnamese reference book (Tử Vi Đẩu Số Tân Biên, Vân Đằng Thái Thứ Lang) that is almost entirely bordered lookup tables.

Why this isn't VLM-first (unlike document-processor)

document-processor exists because business documents (slide decks, dashboards) have charts/tables that a plain text extraction silently drops -- the chart is the data, so it must go through a VLM. A book is different: an image is just an image, and the words are the content the reader cares about. Two things were tested and rejected for text extraction:

• VLM page transcription (Qwen3-VL-32B on real pages): fast and cheap, but it silently changed real words -- "làm nền" → "lảm nhảm" (different word, changed meaning) and dropped a diacritic turning "quần bò sờn" (worn jeans) into a different word. Fine for cheap RAG ingestion where a human never reads the raw text; not acceptable for something Boss will actually read.
• PyMuPDF's own text layer: on the font used in the validation book, it silently drops spaces around certain Vietnamese diacritics ("ởNghệTĩnh", "ChữQuốc"). Poppler's pdftotext did not have this bug on the same file.

So: poppler (pdftotext -bbox) is the source of truth for characters -- it decodes the PDF's actual embedded text, not a re-guess -- and structure (headings, paragraphs, footnotes) is inferred purely from font-height + x-position coordinates, never from an LLM.

Quick start

# 1. Find and extract any real images (see the Pattern-fill gotcha below --
#    do not trust "0 images" from pdfimages/get_images() without visually
#    checking a rendered page first).
python3 scripts/extract_images.py book.pdf --out-dir out/images --map out/page_to_image.json

# 2. Extract clean structured Markdown.
python3 scripts/extract_text.py book.pdf --out out/book.md \
    --images out/page_to_image.json \
    --heading-regex-h1 '^(PART|Part)\s+\d+' \
    --heading-regex-h2 '^Chapter\s+\d+:'

# 3. Add YAML frontmatter (title/author/language) to the top of out/book.md,
#    then hand off to the make-epub skill:
python3 ~/.claude/skills/make-epub/scripts/make_epub.py out/book.md --split-level 2

# 4. Optional Kindle delivery via the publish-to-kindle skill (ask first
#    unless Boss already asked for Kindle delivery in this conversation).

--heading-regex-h1/--heading-regex-h2 are optional hints (plain Python regex, matched at line start). Font-height alone finds a heading; the regex narrows which level it is and guards against a large-font line that isn't actually a chapter title. If you don't know the book's chapter vocabulary yet, run without them first -- height-only detection falls back to tagging every oversized line as H2, which is enough to eyeball the structure before tuning.

The method, end to end

1. Render + scan for images first (extract_images.py). See the Pattern-fill gotcha below -- this step is not optional busywork, a real book's photos were completely invisible to standard tools until this was built.
2. Detect body-text height by sampling line heights across the document and taking the mode. Everything else scales off this number:
   • >= body_height * 1.12 → heading tier
   • body_height * 0.72 .. body_height * 0.92 → footnote body-text tier
   • < footnote_height * 0.72 and a bare 1-3 digit number → superscript marker (either an inline footnote reference, or the leading number of a footnote definition)
3. Paragraph boundaries: a line is the end of a paragraph if its right edge falls short of the document's right margin by more than ~15pt (justified text fills the margin except on a paragraph's last line -- "ragged right" is the tell). Dialogue lines (- ...) always start a fresh paragraph regardless of raggedness.
4. Page-number footers: any line whose entire text is exactly the current page number, at any y-position (the very first page of a book may place it mid-page under the title, not at the bottom).
5. Headings that wrap across two lines: after finding a heading-tier line, keep consuming subsequent lines that are also heading-tier and don't themselves start a fresh heading -- otherwise a two-line chapter title gets silently truncated to its first line.
6. Footnotes: see below, including a real ordering bug this surfaced.
7. Tables: see Gotcha 3 below -- detected separately via pdfplumber's ruling-line reader, not via the font-height heuristics above.
8. Running headers/footers: any line of text (not just a bare page number) that repeats verbatim near the top or bottom of many pages -- typically author name / book title printed on every page -- is detected and stripped so it doesn't bleed into body text or get glued onto a table/paragraph.
9. Feed the resulting Markdown to the existing make-epub skill (chapter splitting on H1/H2, Vietnamese/Hán glyph font subsetting, YAML frontmatter metadata) -- don't reimplement EPUB packaging here.

Gotcha 1: images hidden behind a Pattern fill

pdfimages -list and PyMuPDF's page.get_images() both enumerate a page's Resources/XObject dictionary directly. At least one real PDF (from a "print/export to PDF" pipeline) embedded every photo as a Pattern fill instead:

page content stream:  ... /Pattern cs /P1 scn f ...
Resources:            << /Pattern << /P1 17 0 R >> >>
object 17 (Pattern):  PatternType 1, Resources << /XObject << /X1 16 0 R >> >>
object 17 stream:     q 237 0 0 250 0 0 cm /X1 Do Q      <- draws the real image

Both listing tools return zero images for a page like this, even though a real photo is plainly visible on render. This was only caught because Boss looked at the actual rendered page and pushed back on "0 images found" -- three separate times, for three separate pages, before the pattern was understood. Lesson: never report "no images" on a PDF that plausibly has photos in it without rendering and visually checking at least one representative page yourself (you have vision -- use it, it's free and more reliable than the extraction tools). extract_images.py checks both direct XObjects and pattern-nested ones.

Once found, decode via PyMuPDF's xref_stream(xref) (it transparently handles DCTDecode/FlateDecode) and re-save as JPEG, not PNG -- saving a decoded photo as lossless PNG can be 5-10x larger for a photograph with no visual benefit, and this concretely broke Kindle delivery once (a 22.5MB PNG-heavy EPUB failed to send over SMTP with a timeout; re-encoding the same photos as JPEG brought it to 3.6MB and it sent clean).

Gotcha 2: footnotes

Real PDF footnotes show up as: an inline superscript digit next to a word, and a smaller-font block near the bottom of the page starting with the same digit. Naive line-based extraction glues that footnote block straight into the surrounding body paragraphs as if it were more prose -- exactly the "text mixed together, confusing" bug Boss flagged from reading the actual PDF (page 23: a footnote about King Khải Định's court was appearing as if it were part of the narrator's own commentary).

Fix: detect the marker/footnote font-height tier (above) and emit real Markdown footnote syntax instead:

Body text with a reference[^6].

[^6]: The footnote's own text.

make-epub's make_epub.py already enables python-markdown's footnotes extension, so this renders as a real clickable footnote in the EPUB, not more inline prose.

A page can have more than one footnote (one real page had three: 11, 12, 13). Don't assume one footnote block per page. Distinguish a genuine new footnote-definition start from an ordinary number that happens to appear inside footnote text (e.g. "...vào đầu thế kỷ 20...") by checking both conditions together: the leading token's font-height must be in the marker tier (smaller than the footnote body-text tier), and it must sit at the page's left margin. An inline number inside a footnote's own running text is regular footnote-body height, not marker height -- it will never trigger a false split.

Ordering bug to watch for: the paragraph that references a footnote very often continues onto the next page, while the footnote definition sits mid-page, physically between two halves of that same paragraph. If you buffer paragraph lines and only flush the buffer at a paragraph boundary, the footnote definition (which gets emitted immediately) will land in the output before the still-buffered paragraph that cites it -- reordering content relative to the source. Fix: flush any in-progress paragraph buffer the moment a footnote definition begins, before emitting the footnote text. The referencing paragraph ends up split into two Markdown paragraphs around the footnote (a minor formatting artifact) rather than being silently reordered (a real content bug).

Gotcha 3: bordered tables get flattened into word soup

pdftotext's text layer has no concept of a table -- a table row is just a short line that never reaches the page's right margin. The paragraph "ragged right ends a paragraph" heuristic above then does the opposite of what you want: because every table row looks ragged, it never treats consecutive rows as one block, and because none of them reach the right margin, they don't naturally break either -- rows get glued into a single run-on paragraph with all row/column structure destroyed (verified: a 5-column x 8-row Can/Chi lookup table came out as one sentence of the values in reading order, no way to tell which value belonged to which row or column).

Fix: pdfplumber reads the PDF's actual vector ruling lines directly (page.find_tables()) and reconstructs the true grid, including merged/spanned cells. Detected tables are excluded from the normal pdftotext line-flow (by word bounding-box) so their words aren't also emitted as prose, and rendered as real GFM Markdown tables instead:

| Kim Mệnh | Mộc Mệnh | Thủy Mệnh |
| --- | --- | --- |
| Giáp Tý | Mậu Thìn | Bính Tý |

make-epub's make_epub.py already enables python-markdown's tables extension.

Coordinate systems match without conversion: both poppler's -bbox output and pdfplumber use a top-left origin with y increasing downward, in PDF points, over the same page box.

Known limitation: Markdown tables have no merged-cell syntax, so a spanned cell (e.g. a header cell spanning 5 columns, or a value cell spanning 2 rows) is rendered as one filled cell plus blank cells for the rest of the span -- correct position and value, just without the visual merge. Multi-line cell content is joined with <br> rather than losing the line breaks. If pdfplumber isn't installed, table detection is silently skipped and the old flattened-prose behavior returns -- extract_text.py prints a warning to stderr in that case.

Gotcha 4: running headers/footers leak into body text

Many books print the author name and/or book title on every single page, often as one line split left/right (e.g. author on the left, title on the right) that pdftotext still reports as one grouped line since both halves sit at the same y-coordinate. Left undetected, this line reappears in the Markdown on every page, at unpredictable points relative to the surrounding content -- and because it sits right at a page boundary, it's especially prone to landing in the middle of a table or paragraph that spans two pages, corrupting it.

Fix: sample lines near the top and bottom margin of every Nth page; any exact text that repeats on at least ~30% of sampled pages (and at least 2 pages) is treated as a running header/footer and stripped everywhere it appears in that zone, on every page -- not just the sampled ones. This is separate from (and runs alongside) the existing bare-page-number stripping, which matches only pure digit lines.

Known limitations (acceptable trade-offs, not fixed)

• Italics/bold/colored emphasis from the original print is not preserved -- extraction is plain text only. Confirmed by literally looking at the rendered page: some phrases were genuinely italicized (book/subject titles) or colored (a highlighted phrase) in the source, and that styling is dropped. Worth fixing if a book leans heavily on emphasis for meaning; skipped so far because getting the words right took priority and diminishing returns set in fast.
• Footnote markers inside body text that use non-superscript conventions (some documents use plain parenthetical numbers instead of true superscript) won't be caught by the marker-height heuristic -- check the font-height distribution on a new book before assuming the same three-tier split applies.

Dependencies

• poppler-utils (pdftotext, pdfinfo, pdfimages) -- apt install poppler-utils if missing.
• Python: pymupdf (pip install pymupdf), pillow, pdfplumber (pip install pdfplumber) -- required for table detection (Gotcha 3); extraction still runs without it but silently falls back to flattening tables into prose.
• The make-epub skill for the final Markdown -> EPUB step.
• Optionally the publish-to-kindle skill for delivery.