agentsclimarketplace
Sign in

Benchmark model runtime

Skill gaelic-ghost/socket/skills/benchmark-model-runtime

Benchmark model runtimes across latency, throughput, memory, energy, load time, size, and stability. Use when comparing Core AI, Core ML, MLX, ExecuTorch, PyTorch, quantization, devices, or packaging.From its SKILL.md

Install
npx -y skills add gaelic-ghost/socket --skill benchmark-model-runtime

Assembled from the repository path, not quoted from the project. Check it against their README if it does not work.

One thing to look at

  • 6 stars6 stars. Stars are a popularity signal and not a quality one, but at this level it is likely that nobody has read this closely except its author, and you would be relying on your own review.

SKILL.md

2.0 KB, 338 tokens by cl100k_base, as published. Nobody here has run it

Benchmark Model Runtime

Define A Fair Workload

Pin the exact model artifact, tokenizer/template, runtime and version, device and OS, precision, cache policy, batch size, prompt-length buckets, generated-token target, sampling settings, and measurement tool. Compare numerical or behavioral parity before performance.

Workflow

  1. Verify each artifact produces acceptable outputs on the same small parity set.
  2. Separate cold load, warm load, prompt processing, time to first token, decode throughput, and end-to-end latency.
  3. Measure peak and steady memory; include model, cache, runtime, and process overhead consistently.
  4. Stabilize device power, charging, background load, and thermal state. Record deviations instead of silently rerunning only slow samples.
  5. Warm up separately, then run enough measured repetitions to report median and tail percentiles.
  6. Sweep representative prompt lengths, output lengths, and batch/concurrency levels.
  7. Record failures, fallback execution, recompilation, memory pressure, and thermal throttling.
  8. Measure energy with an appropriate system tool when the decision depends on battery or sustained deployment.
  9. Retain raw samples and summarize them with units, sample counts, and uncertainty.

Apple Runtime Checks

  • Confirm delegated operator coverage and fallback behavior rather than assuming the named backend ran the whole graph.
  • Distinguish compilation/conversion time from load and inference time.
  • For stateful generation, include key-value cache initialization, update, and memory growth.
  • Treat ExecuTorch MLX results as revision-specific while the upstream backend remains experimental.

References

Use references/runtime-benchmarking.md for metric definitions and reporting requirements.

What ships with it: 2 files

1.3 KB alongside SKILL.md

agents/

references/

Gives 0 of the 12 instructions most performance cost skills give in 338 tokens

Counted across 797 of the 1,117 authors here whose files we hold, read 2026-09-06

  • Check for product marketing context firstin 46 of 797, across 20 files
  • Measure before optimizingin 31 of 797, across 25 files
  • Profile first to identify the actual bottleneckin 23 of 797, across 22 files
  • Verify your robots.txt allows AI crawlersin 21 of 797, across 12 files
  • Import directly and avoid barrel filesin 19 of 797, across 15 files
  • Spawn all runs in the same turnin 18 of 797, across 11 files
  • Write a draft of the skillin 17 of 797, across 10 files
  • Understand the user's intentin 17 of 797, across 10 files
  • Use React.cache for per-request deduplicationin 16 of 797, across 11 files
  • Profile before optimizingin 16 of 797, across 14 files
  • Include specific numbers with sourcesin 15 of 797, across 8 files
  • Add lazy loading to below-fold imagesin 15 of 797, across 10 files

Said here and by no other author read

  • Pin the exact model artifact and settings
  • Compare numerical parity before performance
  • Verify each artifact produces acceptable outputs
  • Separate load time and inference time
  • Measure peak and steady memory
  • Stabilize device thermal and power state

Grouped from the skills themselves: near-identical wordings counted once, and counted by distinct author, so one author publishing three of these counts once. Length counted with cl100k_base; the agent that loads this file may tokenize it differently.

Keep looking

Skills are one crate of 325,949. Ordering is by how many stacks a row turns up in, so the top of any crate is what has actually been picked rather than what has the most stars.