Benchmark

Overview

Scalex has a multi-layered profiling and benchmarking system. Pick the right layer for the situation:

Layer	Tool	When to use	Works in native?
1. --timings	Built-in flag	Quick phase breakdown, first look at any perf question	Yes
2. hyperfine	bench.sh	Reproducible before/after comparison with statistics	Yes
3. async-profiler	profiling/profile.sh	Deep CPU/alloc/lock flame graphs to find hotspots	JVM only
4. JFR	profiling/scalex.jfc	GC pressure, file I/O patterns, thread utilization	JVM only
5. Microbenchmarks	src/bench.scala	Isolate per-function cost with warmup + statistics	JVM only
6. Memory profiling	bench.sh memory	Heap usage, GC pressure, peak memory across scenarios	JVM only

Decision guide

"Where is time spent?" → Start with --timings (Layer 1)

"Is this change faster?" → Use hyperfine before/after (Layer 2), optionally with bench-compare.sh

"Why is parsing slow?" → async-profiler CPU flame graph (Layer 3)

"Why are allocations high?" → async-profiler alloc or JFR ObjectAllocationSample (Layer 3/4)

"Is there GC pressure?" → JFR (Layer 4)

"How fast is extractSymbols on one file?" → Microbenchmark (Layer 5)

"How much memory does indexing use?" → Memory profiling (Layer 6)

"Is there a memory leak or GC regression?" → Memory profiling before/after (Layer 6)

---

Layer 1: --timings flag

The fastest way to see where time goes. Works in both JVM and native image. Prints to stderr.

# Cold index phase breakdown
rm -rf benchmark/scala3/.scalex
./scalex index benchmark/scala3 --timings

# Warm index
./scalex index benchmark/scala3 --timings

# Query with bloom/text-search breakdown
./scalex refs benchmark/scala3 Compiler --timings

# JVM mode
scala-cli run src/ -- index benchmark/scala3 --timings

Phases reported

Index phases: git-ls-files, cache-load, oid-compare, parse, index-build, cache-save

Query phases (refs/imports/coverage): bloom-screen, text-search

Reading the output

Timings:
  git-ls-files          12.3 ms  ( 1%)
  cache-load            45.2 ms  ( 5%)
  oid-compare            3.1 ms  ( 0%)
  parse                782.0 ms  (80%)
  index-build           89.4 ms  ( 9%)
  cache-save            42.1 ms  ( 4%)
  total                974.1 ms

• parse > 70%: Scalameta parsing dominates — look at parallelism, parser options, or reducing parsed file count
• cache-load > 20%: Index deserialization — check bloom skip, file size, buffering
• cache-save > 10%: Index serialization — check if save is running unnecessarily (when parsedCount == 0)
• bloom-screen > text-search: Bloom filter is slow — check expected element count, FPP
• text-search >> bloom-screen: Text scanning dominates — check candidate count (bloom too permissive?)

---

Layer 2: hyperfine benchmarks (bench.sh)

Reproducible, statistical benchmarks using hyperfine against the Scala 3 compiler repo (~17.7K files).

Prerequisites

• hyperfine installed (brew install hyperfine)
• Native scalex binary built (./build-native.sh)
• The scala3 repo is cloned automatically on first run

Running

# Full suite (cold + warm + query + diverse + timings)
.Codex/skills/benchmark/scripts/bench.sh

# Individual modes
.Codex/skills/benchmark/scripts/bench.sh cold
.Codex/skills/benchmark/scripts/bench.sh warm
.Codex/skills/benchmark/scripts/bench.sh query
.Codex/skills/benchmark/scripts/bench.sh diverse    # miss, heavy refs, fuzzy, grep, hierarchy
.Codex/skills/benchmark/scripts/bench.sh timings    # --timings output for cold/warm/refs
.Codex/skills/benchmark/scripts/bench.sh memory     # heap usage, GC pressure (JVM only)

# Custom runs/binary
BENCH_RUNS=10 SCALEX_BIN=./target/scalex .Codex/skills/benchmark/scripts/bench.sh

Before/after comparison

# 1. Benchmark current state
BENCH_EXPORT=benchmark/results/before.json .Codex/skills/benchmark/scripts/bench.sh

# 2. Make changes, rebuild
./build-native.sh

# 3. Benchmark new state
BENCH_EXPORT=benchmark/results/after.json .Codex/skills/benchmark/scripts/bench.sh

# 4. Compare (flags >5% regressions)
.Codex/skills/benchmark/scripts/bench-compare.sh benchmark/results/before.json benchmark/results/after.json

bench-compare.sh exits non-zero if any benchmark regressed >5%.

Typical ranges (Apple M3 Max)

Metric	Range	Bottleneck
Cold index	3-5s	Scalameta parsing (CPU-bound, parallel)
Warm index	0.8-1.0s	OID compare + index load
Query (any)	1.2-1.5s	Index deserialization from disk
refs (heavy symbol)	2-4s	Text search across candidate files

---

Layer 3: async-profiler flame graphs

Reveals call-stack-level CPU hotspots. No code changes needed — JVM agent only.

Prerequisites

brew install async-profiler
# Or set AP_HOME to your installation

Running

# CPU flame graph of cold index
./profiling/profile.sh benchmark/scala3

# Wall-clock (includes I/O wait — useful for parallelStream bottlenecks)
./profiling/profile.sh benchmark/scala3 wall

# Allocation hotspots (where objects are created)
./profiling/profile.sh benchmark/scala3 alloc

# Lock contention (parallelStream synchronization)
./profiling/profile.sh benchmark/scala3 lock

Output: profiling/profile-<event>.html — open in browser for interactive flame graph.

What to look for

• CPU: Wide bars in Scalameta parsing → specific parser methods. Wide bars in parallelStream infrastructure → overhead from small task granularity.
• Alloc: Hot allocation sites → potential for object reuse or structural changes.
• Lock: Contention in ConcurrentLinkedQueue.add() → batch results instead of per-item add.
• Wall: I/O wait in Files.readAllLines or Files.readString → potential for memory-mapped I/O.

---

Layer 4: JFR (Java Flight Recorder)

Built into JDK 21. Near-zero overhead. Best for GC, file I/O, and thread analysis.

Running

# Record with custom config
scala-cli run src/ \
  --java-opt "-XX:StartFlightRecording=filename=profiling/scalex.jfr,settings=profiling/scalex.jfc,duration=60s" \
  -- index benchmark/scala3

# Quick summary
jfr summary profiling/scalex.jfr

# Specific events
jfr print --events jdk.ObjectAllocationSample profiling/scalex.jfr | head -100
jfr print --events jdk.GarbageCollection profiling/scalex.jfr
jfr print --events jdk.FileRead profiling/scalex.jfr | head -50
jfr print --events jdk.ThreadPark profiling/scalex.jfr | head -50

# GUI analysis
open profiling/scalex.jfr  # Opens in JDK Mission Control

Custom config

profiling/scalex.jfc is tuned for scalex — it enables allocation sampling, GC events, file I/O (>1ms threshold), and thread parking/monitor events.

---

Layer 5: Microbenchmarks (src/bench.scala)

Isolate per-function costs with warmup and statistical measurement.

Running

# Specific benchmark
scala-cli run src/bench.scala src/*.scala -- extract-single benchmark/scala3
scala-cli run src/bench.scala src/*.scala -- bloom-build benchmark/scala3
scala-cli run src/bench.scala src/*.scala -- persistence-load benchmark/scala3
scala-cli run src/bench.scala src/*.scala -- search benchmark/scala3
scala-cli run src/bench.scala src/*.scala -- refs benchmark/scala3

# All benchmarks
scala-cli run src/bench.scala src/*.scala -- all benchmark/scala3

# Custom warmup/iterations
scala-cli run src/bench.scala src/*.scala -- extract-single benchmark/scala3 --warmup 3 --iterations 10

Available benchmarks

Benchmark	What it measures
extract-single	extractSymbols on the largest file
extract-batch	extractSymbols on 100 files (sequential AND parallel)
bloom-build	buildBloomFilterFromSource on a large source
persistence-load	IndexPersistence.load with and without bloom deserialization
search	WorkspaceIndex.search("Compiler") warm
refs	findReferences("Phase") warm
index-cold	Full cold index including map building

Reports: mean, median, p99, stddev, min, max per benchmark.

---

Layer 6: Memory profiling (bench.sh memory)

Measures heap usage, GC pressure, and peak memory across three scenarios: cold index (full parse), warm index (cache load), and refs query. Uses JVM GC logging via -Xlog:gc* — requires scala-cli (JVM mode), not native binary.

Running

# Full memory profile (cold + warm + refs)
.Codex/skills/benchmark/scripts/bench.sh memory

No prerequisites beyond scala-cli. Does not require hyperfine or native binary.

Output

Reports per-scenario: phase timings (from --timings), then memory stats:

--- Cold index (full parse, ~17.7k files) ---
  git-ls-files            60.7 ms  ( 1%)
  parse                 5576.5 ms  (94%)
  cache-save             228.7 ms  ( 4%)
  total                 5952.2 ms

  Peak pre-GC heap:        790 MB
  Heap at exit (used):     676 MB
  Heap at exit (committed): 1184 MB
  GC pauses:               34
  Total GC pause time:     185.3 ms

Ends with a summary table:

=== Memory Summary ===

Scenario             Peak Heap      Exit Used    Exit Commit     GC #
--------             ---------      ---------    -----------     ----
Cold index              790 MB       676.2 MB      1184.0 MB       34
Warm index              256 MB       271.8 MB       584.0 MB        2
refs Phase               53 MB        29.7 MB        56.0 MB        0

Reading the output

• Peak pre-GC heap: Highest live heap before any GC — the true high-water mark. This is the number that determines minimum -Xmx for constrained environments.
• Heap at exit (used): Retained heap at process exit — the steady-state footprint of the loaded index + results.
• Heap at exit (committed): OS-committed memory — what the JVM actually reserved. Higher than "used" because G1 keeps headroom.
• GC pauses: Number of young-gen collections. High count during cold index is normal (Scalameta ASTs are short-lived).
• Total GC pause time: Sum of all GC pauses. Should be small relative to wall time (<5%).

What to watch for

• Cold peak > 1 GB: parallelStream is creating too many concurrent ASTs. Consider batching files in chunks.
• Warm used > 400 MB: Index deserialization is holding too much data. Check if lazy maps are working.
• Refs used > 100 MB: Text search is accumulating results. Check if bloom filtering is effective.
• GC time > 10% of wall time: GC pressure is impacting performance. Look at allocation hotspots (async-profiler alloc, Layer 3).

Baseline ranges (scala3 corpus, 17.7k files, ~38k symbols)

Scenario	Peak Heap	Exit Used	GC Pauses
Cold index	700-900 MB	600-700 MB	30-70
Warm index	200-300 MB	250-300 MB	1-3
refs query	30-60 MB	25-35 MB	0-1

Ad-hoc memory profiling

For one-off measurements without the script:

# Run any scalex command with GC logging to a temp file
scala-cli run src/ \
  --java-opt "-Xlog:gc*=info:file=/tmp/scalex-gc.log" \
  -- overview benchmark/scala3

# Check peak heap
grep -o '[0-9]*M->' /tmp/scalex-gc.log | sed 's/M->//' | sort -n | tail -1
# Check exit heap
grep "garbage-first" /tmp/scalex-gc.log | tail -1

---

Native image profiling

async-profiler and JFR don't work with GraalVM native images. Options:

# macOS Instruments (Time Profiler)
xcrun xctrace record --template "Time Profiler" --launch -- ./scalex index scala3

# --timings always works
./scalex index scala3 --timings

---

Performance budget (from AGENTS.md)

When evaluating changes:

• Index times: Accept <5% regression
• Index size: 0% growth for non-index features, <10% if schema changes
• Query latency: No regression
• Feature gate: "Is this better than grep, or does it introduce a worst case?"

The bench-compare.sh script automates the regression check against these thresholds.

Typical workflow for a performance change

1. --timings to identify which phase to optimize
2. BENCH_EXPORT=before.json bench.sh to capture baseline
3. If deeper analysis needed: async-profiler flame graph or JFR
4. Make the change
5. --timings to verify phase improvement
6. BENCH_EXPORT=after.json bench.sh to capture new numbers
7. bench-compare.sh before.json after.json to check for regressions
8. Microbenchmarks if isolating a specific function