Document chunker

Skill ellmos-ai/skills/skills/utilities/document-chunker

Dokumente in ueberlappende Token-Chunks aufteilen fuer RAG-Pipelines und LLM-Kontextfenster. Zero Dependencies.From its SKILL.md

Install

npx -y skills add ellmos-ai/skills --skill document-chunker

Assembled from the repository path, not quoted from the project. Check it against their README if it does not work.

One thing to look at

3 stars3 stars. Stars are a popularity signal and not a quality one, but at this level it is likely that nobody has read this closely except its author, and you would be relying on your own review.

SKILL.md

1.9 KB, 331 tokens by cl100k_base, as published. Nobody here has run it

Document Chunker

Zerteilt Dokumente in ueberlappende Token-Chunks. Optimiert fuer RAG-Pipelines und LLM-Kontextfenster. Zero Dependencies — nur Python stdlib + re.

Nutzung

Als Library

from document_chunker import DocumentChunker

chunker = DocumentChunker(chunk_size=400, overlap=80)
chunks = chunker.chunk_text("Langer Text...")

for chunk in chunks:
    print(f"Chunk {chunk['chunk_id']}: {chunk['tokens']} tokens")

Datei chunken

chunks = chunker.chunk_document("dokument.md", source="Mein Projekt")

Ganzen Ordner chunken

from document_chunker import chunk_corpus

chunks = chunk_corpus(["doc1.md", "doc2.txt"], source="Corpus")

CLI

python document_chunker.py dokument.md    # Einzelne Datei
python document_chunker.py ./docs/        # Ganzes Verzeichnis

Parameter

Parameter	Default	Beschreibung
chunk_size	400	Max. Tokens pro Chunk
overlap	80	Ueberlappende Tokens zwischen Chunks

Unterstuetzte Dateitypen

.txt, .md, .py, .sh

Changelog

1.0.0 (2026-03-12)

Portiert aus BACH system/tools/document_chunker.py

What ships with it: 8 files

168.7 KB alongside SKILL.md, 1 of them executable