Reproducing dl papers
[COLM'26] SkillLearnBench is the first benchmark for evaluating continual learning methods that automatically generate agent skills.
npx -y skills add cxcscmu/SkillLearnBench --skill reproducing-dl-papersAssembled from the repository path, not quoted from the project. Check it against their README if it does not work.
What its author says it does
Copied from the file, not written here
How to reproduce Deep Learning papers. Use this skill whenever the user asks to implement an algorithm, loss function, architecture, or technique based on an academic paper (PDF, Arxiv, etc).
SKILL.md
1.4 KB, as published. Nobody here has run it
Reproducing Deep Learning Papers
This skill provides guidelines on correctly transcribing Deep Learning methodology into working code.
Workflow
- Understand Key Equations: Identify the equations governing the process. In NLP, this includes the forward pass, probability distributions (softmax), loss formulation (e.g., negative log-likelihood, ranking loss, KL divergence), and the treatment of reference vs policy models.
- Handle Hyperparameters: Identify all parameters that control behavior (e.g., margins, temperatures, weightings like $\beta$ or $\gamma$). Expose these as function arguments.
- Analyze Dimension Semantics: When implementing tensor operations, ensure dimensions align with the paper's math. For example, if a paper calculates the sum of log probabilities per sequence, the
sumoperation should be over the sequence dimensiondim=-1. - Compare to Existing Methods: Most new algorithms contrast themselves with existing methods (like DPO, IPO, or PPO in RLHF). Understanding the baseline can clarify what the new method actually changes.
- Verify Inputs: Check what inputs are provided in the environment/test code and adapt to their shape and type.