Tokenizer design
[COLM'26] SkillLearnBench is the first benchmark for evaluating continual learning methods that automatically generate agent skills.
npx -y skills add cxcscmu/SkillLearnBench --skill tokenizer-designAssembled from the repository path, not quoted from the project. Check it against their README if it does not work.
SKILL.md
1.1 KB, 221 tokens by cl100k_base, as published. Nobody here has run it
name: tokenizer-design description: Design principles for Scala-based tokenization systems. Use this skill when designing or implementing tokenization logic in Scala, particularly when porting from Python-based designs. Covers immutable structures, type-safe hierarchies, and monad/functor patterns for data transformation.
Scala Tokenization Principles
- Immutability First: Represent all data structures (Tokens, Batches) as immutable case classes.
- Type Safety: Use Scala's type system (sealed traits, ADTs) instead of Python's dynamic
Unionand duck typing. - Functional Composition: Use
map,flatMap, andforcomprehensions for transformations instead of procedural loops. - Error Handling: Use
Option,Either, orTryfor fallible operations instead of raising exceptions or returningNone. - Type Classes: Replace Python's
ProtocolandGenericconstraints with Scala type classes (traits with context bounds). - Builder Pattern: Utilize Scala's constructor overloading or implicit parameter lists for flexible configuration instead of mutable builders.
What ships with it
Read from the repository
Just SKILL.md. No reference files, no scripts.