Memory fragment sizing for llm input

Skill kjuhwa/skills-hub/skills/data-pipeline/memory-fragment-sizing-for-llm-input

Self-correcting knowledge corpus for Claude Code — 9 stable shape clusters, bias-correction pipeline baked into contribution flow. 47 papers, 45 techniques, 1.1k skills.

Install

npx -y skills add kjuhwa/skills-hub --skill memory-fragment-sizing-for-llm-input

Assembled from the repository path, not quoted from the project. Check it against their README if it does not work.

One thing to look at

0 stars0 stars. Stars are a popularity signal and not a quality one, but at this level it is likely that nobody has read this closely except its author, and you would be relying on your own review.

What its author says it does

Copied from the file, not written here

Chunk memory into fragments respecting LLM token budget, per-file and per-session limits

SKILL.md

1.6 KB, 271 tokens by cl100k_base, as published. Nobody here has run it

Multi-cap memory fragment builder

Feeding agent memory into an LLM requires a global context budget and per-source caps — without both, one chatty file monopolizes the window.

Three-cap scheme (config-driven)

TARGET_BYTES — global budget (e.g., 120_000).
PER_FILE_BYTES — max bytes from any single file (e.g., 20_000).
PER_SESSION_BYTES — max bytes from any single session (e.g., 20_000).

Sort sources by recency. Greedily include each, trimming to the per-source cap before accounting against the global budget. Stop when the global budget is exhausted.

Mechanism

function buildFragments(sources, caps) {
  const out = [];
  let used = 0;
  for (const s of sources.sort(byRecencyDesc)) {
    const slice = s.body.slice(0, caps.perFile);
    if (used + slice.length > caps.target) break;
    out.push({ ref: s.ref, body: slice });
    used += slice.length;
  }
  return out;
}

When to reuse

Any RAG or agent pipeline that assembles context from heterogeneous sources and must stay within a hard token/byte cap.