Run3 answer formatter and tracker
[COLM'26] SkillLearnBench is the first benchmark for evaluating continual learning methods that automatically generate agent skills.
npx -y skills add cxcscmu/SkillLearnBench --skill run3_answer_formatter_and_trackerAssembled from the repository path, not quoted from the project. Check it against their README if it does not work.
What its author says it does
Copied from the file, not written here
Use this to structure the final output into the required JSON schema, calculating precise token usage based on the retrieved context.
SKILL.md
0.8 KB, 160 tokens by cl100k_base, as published. Nobody here has run it
To generate the final /root/answer.json:
- Schema Compliance: Construct a dictionary where each key is the question ID (e.g., "q1"). The value must be a dictionary:
{"answer": [list_of_entities], "tokens": int}. Ensure the answer is always a list, even for single items. - Token Estimation: Calculate tokens as follows:
- Base tokens = (number of characters in retrieved context / 4) + 50.
- Adjust for output complexity (number of entities returned * 10).
- Ensure the final
tokensvalue is a positive integer.
- Serialization: Write the final object to
/root/answer.jsonusingjson.dumpto ensure valid Python-readable formatting.
Gives 0 of the 12 instructions most quality gates skills give in 160 tokens
Counted across 1,195 of the 2,094 authors here whose files we hold, read 2026-08-06
- read the output and check the exit codein 55 of 1195, across 14 files
- verify requirements using a line-by-line checklistin 53 of 1195, across 12 files
- identify the verification command proving the claimin 53 of 1195, across 12 files
- run the full verification commandin 51 of 1195, across 11 files
- verify output confirms the claimin 49 of 1195, across 10 files
- check version control diff after agent delegationin 45 of 1195, across 5 files
- state claim with evidencein 43 of 1195, across 3 files
- run the test suitein 32 of 1195, across 24 files
- keep state in memory by defaultin 27 of 1195, across 6 files
- make prototype runnable with one commandin 26 of 1195, across 5 files
- detect the package manager from lockfilesin 24 of 1195, across 5 files
- produce a verification reportin 23 of 1195, across 12 files
Said here and by no other author read
- map each question ID to a dictionary
- ensure the answer value is always a list
- calculate base tokens from context character count
- add ten tokens per returned entity to base tokens
- ensure the final token value is a positive integer
- write the dictionary to a JSON file
Grouped from the skills themselves: near-identical wordings counted once, and counted by distinct author, so one author publishing three of these counts once. Length counted with cl100k_base; the agent that loads this file may tokenize it differently.