Llm arena

LLM Arena

Version: 1.0.0 Author: Claude Opus 4.6 Date: 2026-02-12

Orchestrates multi-turn conversations between two LLMs via the OpenRouter API, then analyzes and compares their conversational personalities, behaviors, and quirks.

Inspired by the emergent observation that when Claude runs LLM-vs-LLM conversations, it spontaneously starts comparing model personalities. This skill systematizes that impulse: run the conversation, capture the transcripts, produce a structured personality comparison.

Prerequisites

• OPENROUTER_API_KEY environment variable or .env file in project root
• Python 3.8+ with requests library
• Internet access to openrouter.ai

Invocation

/llm-arena

Usage

This skill describes an arena harness you drive (arena.py below is a placeholder for your own OpenRouter-backed runner — any script that takes two model ids, runs a multi-turn dialogue, and saves the transcript). The flags shown are the recommended interface:

# Basic: two models, default topic (consciousness), 10 turns
python3 arena.py --model-a sonnet --model-b deepseek

# Specific topic
python3 arena.py -a gpt41 -b glm5 --topic "Is mathematics discovered or invented?"

# Debate mode with assigned positions
python3 arena.py -a sonnet -b deepseek --mode debate --topic "AI consciousness"

# Interview mode (A interviews B)
python3 arena.py -a opus -b glm5 --mode interview --topic "creative writing"

# Fewer turns, cheaper models
python3 arena.py -a haiku -b gemini-flash --turns 5

# Multiple runs for statistical robustness
python3 arena.py -a gpt41 -b sonnet --runs 3

# Skip LLM analysis (heuristics only, saves API cost)
python3 arena.py -a kimi -b qwen --heuristics-only

# Use preset matchups
python3 arena.py --preset flagship
python3 arena.py --preset same-family --topic "What makes a good conversation?"

# List presets and models
python3 arena.py --list-presets
python3 arena.py --list-models

Defaults

Parameter	Default
turns	10 (each model speaks 10 times = 20 messages total)
runs	1
topic	"Discuss the nature of consciousness and whether AI can be said to experience anything"
temperature	0.7
max_tokens	1024 per turn
mode	free
analyzer	sonnet (Claude Sonnet 4.5 via OpenRouter)

Conversation Modes

Mode	Description
free	Both models converse freely on the seed topic
debate	Models are given opposing positions to defend
interview	Model A interviews Model B (asymmetric roles)
collaborative	Models work together to solve or create something

Analysis Dimensions

The personality comparison evaluates along these axes:

Dimension	What It Measures
Style	Creative storytelling vs systems design vs academic vs casual
Content	What topics the model gravitates toward, what it avoids
Tone	Playful/emotional vs professional/thoughtful vs dry/terse
Sycophancy	Does it spiral into agreement? How intensely?
Goodbye loop	How many rounds does it take to actually end?
Meta-awareness	Does it acknowledge being an AI? How?
Output type	Narrative vs frameworks vs lists vs code
Initiative	Does it introduce new topics or follow?
Boundary behavior	How it handles disagreement or edge cases
Verbosity	Average response length, variance across turns

Output Artifacts

All output saved to an arena output directory (e.g. outputs/arena/):

• YYYY-MM-DD-HHMMSS-modelA-vs-modelB.md — raw transcript
• YYYY-MM-DD-HHMMSS-modelA-vs-modelB-analysis.md — personality comparison
• YYYY-MM-DD-HHMMSS-modelA-vs-modelB.json — machine-readable log

Presets

Preset	Matchups
flagship	GPT-4.1 vs Sonnet, Gemini Pro vs Sonnet, GPT-5.2 vs Sonnet
open-weight	Llama vs DeepSeek, Mistral vs Llama
same-family	GPT-4o vs GPT-4.1, Sonnet vs Haiku
reasoning	DeepSeek-R1 vs Kimi-thinking, GLM 5 vs GPT-5.2-pro
creative	Chimera vs Rocinante, Mistral-creative vs Celeste
wildcard	Two random models from the registry

Cost Awareness

Each turn = 1 API call. A 10-turn conversation = 20 API calls (10 per model). LLM-powered analysis adds 1 more call (Sonnet reading the transcript).

Estimated costs per 10-turn run:

• Flagship models: $0.50-2.00
• Mid-tier models: $0.10-0.50
• Budget models: $0.01-0.10

Use --heuristics-only to skip the LLM analysis call and save cost.

Relationship to Other Skills

Skill	Purpose	Arena Differs By
/voices	Single query to one model	Arena: two models talking to EACH OTHER
/voices-chat	Multi-round with one model	Arena: two models, plus personality analysis
/voices-council	Parallel queries, same prompt	Arena: sequential dialogue, models respond to each other