Simpo loss

Skill cxcscmu/SkillLearnBench/skills/b1-one-shot-claude-sonnet-4-6/nlp-paper-reproduction/simpo-loss

Implements the SimPO (Simple Preference Optimization) loss function from the paper "SimPO: Simple Preference Optimization with a Reference-Free Reward". Use this when implementing or understanding SimPO training objectives.From its SKILL.md

Install

npx -y skills add cxcscmu/SkillLearnBench --skill simpo-loss

Assembled from the repository path, not quoted from the project. Check it against their README if it does not work.

SKILL.md

2.5 KB, 634 tokens by cl100k_base, as published. Nobody here has run it

SimPO Loss Implementation

Overview

SimPO (Simple Preference Optimization) is a reference-free preference optimization method that:

Uses length-normalized log probability as the implicit reward (aligns with generation metric)
Introduces a target reward margin γ to separate winning/losing responses
Eliminates the need for a reference model (unlike DPO)

Loss Formula (from paper Eq. 6)

LSimPO(πθ) = −E(x,yw,yl)∼D [ log σ( β/|yw| * log πθ(yw|x) - β/|yl| * log πθ(yl|x) - γ ) ]

Where:

β: scaling factor (default: 2.0)
γ: target reward margin (default: gamma_beta_ratio * β = 0.25 * 2.0 = 0.5)
|yw|, |yl|: lengths of winning/losing responses
policy_chosen_logps, policy_rejected_logps: AVERAGE log probabilities per token

Key Notes

policy_chosen_logps = average log prob per token = (1/|yw|) * log πθ(yw|x)
So β * policy_chosen_logps = β/|yw| * log πθ(yw|x) (already normalized)
gamma_beta_ratio = γ/β, so γ = gamma_beta_ratio * β

Implementation

def simpo_loss(
    self,
    policy_chosen_logps: torch.FloatTensor,
    policy_rejected_logps: torch.FloatTensor,
) -> Tuple[torch.FloatTensor, torch.FloatTensor, torch.FloatTensor]:
    # Length-normalized log prob difference
    pi_logratios = policy_chosen_logps - policy_rejected_logps
    # Subtract gamma/beta ratio (equivalent to subtracting gamma from beta*logratios)
    gamma_logratios = self.gamma_beta_ratio
    logits = pi_logratios - gamma_logratios

    if self.loss_type == "sigmoid":
        losses = (
            -F.logsigmoid(self.beta * logits) * (1 - self.label_smoothing)
            - F.logsigmoid(-self.beta * logits) * self.label_smoothing
        )
    elif self.loss_type == "hinge":
        losses = torch.relu(1 - self.beta * logits)
    else:
        raise ValueError(f"Unknown loss type: {self.loss_type}")

    # Rewards (scaled average log probs)
    chosen_rewards = self.beta * policy_chosen_logps.detach()
    rejected_rewards = self.beta * policy_rejected_logps.detach()

    return losses, chosen_rewards, rejected_rewards

Config Parameters (SimPOConfig)

beta: float = 2.0
gamma_beta_ratio: float = 0.25 (γ/β ratio)
label_smoothing: float = 0
loss_type: "sigmoid" | "hinge" = "sigmoid"

What ships with it

Read from the repository

Just SKILL.md. No reference files, no scripts.