Token optimizer

Token Optimizer

Reduce tokens in prompts, docs, and prose while preserving meaning.

When to Use

• Prompt >1500 tokens or contains redundant phrasing
• Docs/markdown need compression for LLM context
• Writing prose for humans (docs, commits, PRs, error messages)
• Converting JSON/YAML/XML data for LLM input (TOON)

Quick Routing

Input Type	Go To
User prompt / chat message	Section 1: Prompt Compression
Docs / markdown / README	Section 2: Doc Formatting
Commits / PRs / error messages	Section 3: Prose Clarity
JSON / YAML / XML data	Section 4: TOON Format

Compression Levels

Level	Reduction	Use When
Light	20-30%	Keep readability, human-facing prose
Medium	40-50%	Default for LLM context
Heavy	60-70%	Max compression, symbols over words

---

1. Prompt Compression

Process

1. Identify core intent
2. Extract essential context only
3. Remove filler phrases
4. Apply abbreviations & symbols
5. Output compressed version w/ token savings %

Remove Phrases

Remove	Transform To
"Please help me with"	(delete)
"I need you to"	(delete)
"Could you please"	(delete)
"I would like to"	(delete)
"I think", "Maybe", "Perhaps"	(delete)
"This might be a dumb question"	(delete)
"For your reference"	(delete)
"As I mentioned before"	(delete)

Transform Patterns

Verbose	Compressed
"I want to create a fn that takes X and returns Y"	fn(X) -> Y
"The error message says..."	Error: ...
"In the file located at..."	File: ...
"I'm trying to..."	Goal: ...
"Here is my code..."	Code:
"The problem is that..."	Issue: ...

Example

Before (847 tokens):

Hello! I hope you're doing well. I was wondering if you could help me.
I'm trying to build a React app and I need a custom hook that fetches
user data from /api/users. It returns JSON. I'd like loading, error
states, and caching. I think useEffect and useState but not sure...

After (156 tokens):

Goal: React hook for user data fetching
- Endpoint: /api/users -> JSON user obj
- Handle: loading, error states
- Cache response
Stack: React (useEffect, useState)

---

2. Doc Formatting

Remove Filler Words

Remove when possible: "basically", "essentially", "actually", "really"

Verbose	Compressed
"in order to"	"to"
"due to the fact that"	"because"
"at this point in time"	"now"
"in the event that"	"if"
"has the ability to"	"can"
"it is important to note that"	(remove)

Compress Lists

Before:

The following features are included:
- User authentication with JWT tokens
- Role-based access control for authorization
- Password hashing using bcrypt algorithm

After:

Features:
- JWT auth
- RBAC authz
- bcrypt passwords

Structured Data Over Prose

Before:

The API accepts three parameters. The first is "userId" which is a
required string. The second is "limit", optional number, defaults to 10.

After:

Params:
- userId: str (req) - user ID
- limit: num (opt, def=10)

Heading Compression

Before: # Comprehensive Guide to User Authentication and Authorization After: # Auth Guide

Table Compression

Shorten column headers, use abbreviations in cells:

Param	Type	Req	Def	Desc
userId	str	Y	-	user ID
limit	num	N	10	max results

Code Block Compression

Remove comments that restate the code. Collapse trivial logic:

Before:

// Check if email is valid
const isEmailValid = emailRegex.test(email);
// Check if password is at least 8 characters
const isPasswordValid = password.length >= 8;
// Return true if both are valid
return isEmailValid && isPasswordValid;

After:

return emailRegex.test(email) && password.length >= 8;

URL & Path Compression

Replace full home paths: /Users/username/Documents/Projects/MyApp/config/settings.json -> ~/Projects/MyApp/config/settings.json

Error Message Compression

Keep error codes and stack traces exact. Compress surrounding prose: Err: Invalid/expired token. Re-authenticate at /auth/login

---

3. Prose Clarity (Strunk's Rules)

Apply when writing prose for humans (docs, commits, PRs, errors).

Grammar Essentials

1. Form possessive singular by adding 's
2. Use comma after each term in series except last
3. Don't join independent clauses by comma (use semicolon or period)
4. Participial phrase at beginning refers to grammatical subject

Composition Essentials

1. One paragraph per topic
2. Begin paragraph with topic sentence
3. Express coordinate ideas in similar form
4. Keep related words together
5. Place emphatic words at end of sentence

Limited Context Strategy

When context is tight: write draft, dispatch subagent with draft + these rules, have subagent copyedit and return revision.

Core Principles

Omit needless words:

Wordy	Concise
the question as to whether	whether
there is no doubt but that	no doubt
he is a man who	he
the reason why is that	because

Use active voice:

Passive	Active
The file was deleted by the user	The user deleted the file
Errors are logged by the system	The system logs errors

Positive form over negative:

Negative	Positive
did not remember	forgot
not important	trifling
not honest	dishonest

Specific over vague:

Vague	Specific
A period of unfavorable weather set in	It rained every day for a week
The data was processed	The server parsed 10,000 records

Technical Writing

# Commit: Bad
Made some changes to fix the bug that was causing issues

# Commit: Good
Fix null pointer in user authentication

# Error: Bad
An error occurred while processing your request

# Error: Good
Database connection failed: timeout after 30s

# Doc: Bad
This function is used for the purpose of validating user input

# Doc: Good
Validates user input

Editing Checklist

• Unnecessary words (especially "that", "very", "really", "just")
• Passive voice -> convert to active
• Negative statements -> make positive
• Vague language -> make specific
• Long sentences -> break up or simplify

---

4. TOON Format (Data Serialization)

TOON replaces JSON/YAML/XML data serialization when sending to LLMs. ~40% fewer tokens.

Convert to TOON: JSON, YAML, XML (data objects/arrays only) Keep as-is: Markdown, plain text, code files, CSV

Precedence note: TOON converts the data format structure (braces, quotes, colons). The "Never Compress" rules protect values inside data (API keys, URLs, version numbers). Both apply: convert structure to TOON, but preserve exact values.

JSON:

{"users":[{"id":1,"name":"John","role":"admin"},{"id":2,"name":"Jane","role":"user"}]}

TOON:

users[2]{id,name,role}:
  1,John,admin
  2,Jane,user

See: .claude/skills/document-skills/toon/SKILL.md

---

Never Compress

See references/never_compress.md for full list:

• Auth tokens, API keys, credentials, secrets
• Error stack traces (keep full)
• Code blocks, inline code, regex
• URLs, UUIDs, version numbers
• SQL queries, shell commands
• JSON keys, config values, date formats
• Legal text, mathematical formulas

---

Abbreviations

fn=function  ret=return  str=string  num=number
bool=boolean arr=array   obj=object  param=parameter
config=configuration     env=environment
auth=authentication      authz=authorization
db=database  repo=repository  dir=directory
req=required opt=optional def=default
max=maximum  min=minimum  ex=example
impl=implementation      docs=documentation
app=application          info=information

Symbols

-> = returns/produces    & = and
|  = or                  w/ = with
w/o = without            ~ = approximately
=> = therefore           bc = because
Y  = yes                 N  = no
-  = none/null

Output Format

## Compressed

[Content]

---
Original: X tokens | Compressed: Y tokens | Saved: Z%

Scripts

# Compress text
python scripts/compress.py input.md > compressed.md
python scripts/compress.py input.md --stats
python scripts/compress.py --level 2 < input.md

# Count tokens
python scripts/count_tokens.py document.md

# Compress prompt
python scripts/compress_prompt.py "your prompt text"
python scripts/compress_prompt.py --file prompt.txt --level heavy

Integration

Called by: brainstorm (Phase 5 docs), any skill producing documentation Pairs with: document-skills/toon (data serialization)