agentsclimarketplace

Kernel benchmark

Skill fmh66/kernel-opt-agent/skills/kernel-benchmark

Standalone kernel benchmarking skill for cuda-cpp, cutlass, cute-dsl, and triton implementations. Use when the user wants to compare a custom CUDA/CUTLASS .cu kernel or CuTe DSL/Triton .py kernel against selectable PyTorch eager, torch.compile, or FlashInfer baselines, validate correctness, measure execution time with KernelBench-style CUDA event timing, or generate benchmark.md for kernel optimization results.From its SKILL.md

Install
npx -y skills add fmh66/kernel-opt-agent --skill kernel-benchmark

Assembled from the repository path, not quoted from the project. Check it against their README if it does not work.

2 things to look at

  • no licenseNo license file was found in the repository. Code published without one is not open source by default, so using it at work is a question for whoever answers licensing questions where you are.
  • 19 stars19 stars. Stars are a popularity signal and not a quality one, but at this level it is likely that nobody has read this closely except its author, and you would be relying on your own review.

SKILL.md

6.3 KB, as published. Nobody here has run it

kernel-benchmark

Use this skill to benchmark a custom GPU kernel against a PyTorch reference implementation.

What It Does

scripts/benchmark.py compares a solution against selectable baselines:

VariantTiming method
Custom CUDA-C++/CUTLASS/CuTe DSL/Triton solutioncuda_event by default; optional host_time
PyTorch eager referencesame timing method; default baseline
torch.compile referencesame timing method; opt in with --baselines=torch-compile
FlashInfer baselinesame timing method; opt in with --baselines=flashinfer

The benchmark validates correctness against the PyTorch eager reference before timing. Selected baselines are also checked against the eager reference. The report is written to <output-dir>/benchmark.md.

Default timing uses KernelBench-style CUDA events with explicit warmup/trial counts and cold-cache L2 thrashing. Use --timing-method=host_time only when diagnosing Python/runtime overhead.

Required Interfaces

Reference And Baselines

The reference file must define:

def reference(**kwargs):
    ...

The function receives the same tensor/scalar kwargs as the solution. It may update output tensors in place or return a tensor, tuple/list of tensors, or dict keyed by output name. Optional scalar and dimension parameters are passed through kwargs.

When --baselines=flashinfer is selected, the runner calls the same function with baseline="flashinfer" in kwargs. When no baseline kwarg is provided, reference(**kwargs) is the PyTorch eager reference.

CUDA-C++ / CUTLASS Solution

CUDA-C++ and CUTLASS solutions are .cu files with a compiled shared library beside them. The script loads the shared library; it does not compile it.

The .cu file must expose:

extern "C" void solve(...);

Compile first:

nvcc -shared -std=c++17 -arch=sm_90 -O3 -Xcompiler -fPIC -o kernel.so kernel.cu

CUTLASS code must be wrapped by solve(...); compile with the required CUTLASS include/library flags.

CuTe DSL / Triton Solution

CuTe DSL and Triton solutions are .py files and must define:

def setup(**kwargs):
    return {"inputs": {...}, "outputs": ["out"]}

def run_kernel(**kwargs):
    ...

setup()["inputs"] must contain every tensor/scalar passed to the kernel and reference. setup()["outputs"] names the tensors compared for correctness.

Supported Implementations

ImplementationInputBenchmark ABINotes
cuda-cpp.cu + compiled .soextern "C" void solve(...)Default for non-.py files
cutlass.cu + compiled .soextern "C" void solve(...)CUTLASS code must be wrapped by solve(...)
cute-dsl.pysetup(**kwargs) + run_kernel(**kwargs)Pass --implementation=cute-dsl explicitly
triton.pysetup(**kwargs) + run_kernel(**kwargs)Default for .py files

--backend is kept as a compatibility alias for --implementation. Prefer --implementation in new commands.

Usage

CUDA-C++:

python scripts/benchmark.py kernel.cu \
  --implementation=cuda-cpp \
  --baselines=pytorch-eager \
  --ref=ref.py \
  --output-dir=bench_out \
  --M=1024 --N=1024

CUTLASS:

python scripts/benchmark.py cutlass_kernel.cu \
  --implementation=cutlass \
  --baselines=pytorch-eager,torch-compile \
  --ref=ref.py \
  --output-dir=bench_out \
  --M=1024 --N=1024

Triton:

python scripts/benchmark.py kernel.py \
  --implementation=triton \
  --baselines=pytorch-eager,flashinfer \
  --ref=ref.py \
  --output-dir=bench_out \
  --M=1024 --N=1024

CuTe DSL:

python scripts/benchmark.py cute_kernel.py \
  --implementation=cute-dsl \
  --ref=ref.py \
  --output-dir=bench_out \
  --M=1024 --N=1024

--implementation=auto selects Triton for .py files and CUDA-C++ otherwise.

Options

OptionDefaultMeaning
solution_filerequired.cu or .py solution file
--ref=<path>requiredPython reference defining reference(**kwargs)
--output-dir=<dir>requiredDirectory for benchmark.md
--implementation=<auto/cuda-cpp/cute-dsl/cutlass/triton>autoSolution implementation; --backend is an alias
--baselines=<list>pytorch-eagerComma-separated baselines from pytorch-eager, torch-compile, flashinfer, or all/none
--timing-method=<cuda_event/host_time>cuda_eventTiming backend
--num-warmup=<n>5Warmup calls for fixed-trial timing
--num-trials=<n>100Measured trials for fixed-trial timing
--discard-first=<n>1First measured trials to discard for fixed-trial timing
--prewarm-calls=<n>1Untimed calls before correctness/timing to trigger lazy init/JIT
--ptr-size=<n>0Override CUDA pointer buffer element count
--arch=<sm_XX>autoGPU architecture, for messages and CUDA compile hint
--gpu=<id>0CUDA device index
--atol, --rtolref.py module-level if set, else 1e-4 / 1e-3Correctness tolerances. CLI flag takes highest precedence, then ref.py module-level atol/rtol, then internal defaults. Set atol=1e-2, rtol=1e-2 in ref.py for bfloat16 FlashInfer baselines.
--seed=<n>42Random seed
--NAME=VALUEas neededInteger dimensions/scalars used by the kernel signature or Triton setup

Keep dimensions, seed, pointer sizing, GPU, and tolerances fixed when comparing kernel versions.

Output

FilePurpose
<output-dir>/benchmark.mdCorrectness status, GPU/arch/dimensions, timing config, mean/median/p20/p80/min/max/std/sample-count timing, and speedups versus selected baselines

If correctness fails, inspect the output tensors before using the timing result for optimization decisions.

For GPU-kernel latency, use the default cuda_event. For end-to-end Python/runtime overhead diagnostics, use host_time.

Keep looking

Skills are one crate of 611,178. Ordering is by how many stacks a row turns up in, so the top of any crate is what has actually been picked rather than what has the most stars.