Performance regression testing

Performance Regression Testing

Instructions

Perf regressions ship silently. Unit tests pass, UI tests pass, and the app is 400 ms slower to open on a Pixel 4a. Catching them requires baselines, noise-aware thresholds, and dedicated benchmarks — not ad-hoc manual measurements.

1. Dimensions Worth Tracking

• Cold startup time — time to first frame and to first interactive.
• Warm / hot startup time.
• Frame time / jank during scroll on a known list.
• Time to interaction on core flows (home, search, checkout).
• Binary size (APK/AAB split by ABI; IPA thin + universal).
• Memory footprint on a critical screen.
• Battery / CPU on a 10-minute scripted session (ongoing monitoring, not per-PR).

2. Android — Macrobenchmark

Use androidx.benchmark:benchmark-macro-junit4 for startup and frame-time metrics on a real device.

@RunWith(AndroidJUnit4::class)
class StartupBenchmark {
    @get:Rule val rule = MacrobenchmarkRule()

    @Test fun coldStart() = rule.measureRepeated(
        packageName = "com.example.app",
        metrics = listOf(StartupTimingMetric()),
        iterations = 10,
        startupMode = StartupMode.COLD,
    ) {
        pressHome()
        startActivityAndWait()
    }
}

For jank:

metrics = listOf(FrameTimingMetric())

Run on at least one low-RAM device and one high-end device; compare like-for-like.

3. iOS — XCTest Performance & os_signpost

func test_checkoutStartup() {
    measure(metrics: [XCTApplicationLaunchMetric(), XCTCPUMetric()]) {
        XCUIApplication().launch()
    }
}

For custom spans, instrument with OSSignposter:

let signposter = OSSignposter(subsystem: "checkout", category: .pointsOfInterest)
let id = signposter.makeSignpostID()
let state = signposter.beginInterval("load", id: id)
// ... work ...
signposter.endInterval("load", state)

Consume traces in Instruments or via MetricKit for production telemetry.

4. Flutter

Use flutter drive --profile --trace-startup for startup; parse start_up_info.json:

flutter drive --profile --trace-startup --target=test_driver/app.dart

For frame metrics, wrap the test with binding.watchPerformance and assert on FrameTiming build/raster durations. Fail the test if 95th-percentile frame time exceeds your budget (e.g., 16 ms @ 60 Hz, 8 ms @ 120 Hz).

5. React Native

• Startup: react-native-performance + PerformanceObserver to mark nativeLaunchEnd, runJsBundleEnd, appReady.
• Frame time: Flipper Perf Monitor, or on-device FPS Monitor gated by a dev flag.
• Bundle size: fail CI when the bundle grows beyond budget.

6. Baselines and Noise

A single measurement is meaningless. For each metric:

1. Run N = 10–30 iterations on the same device class.
2. Take the median or 90th percentile, not the mean.
3. Compare against a baseline stored in the repo (one per device class), refreshed on a schedule — not on every PR.
4. Fail only when the delta exceeds a noise threshold (e.g., +5 % and + 20 ms absolute, whichever is larger).

Single-run regressions should be logged, not blocking.

7. Binary Size

• Android: Gradle task to print APK/AAB size; fail PR if delta > budget. Use apkanalyzer / Size Analyzer to attribute growth.
• iOS: parse .ipa size after Archive; use App Store Connect size report per release.
• Flutter: --analyze-size to get per-library breakdown.
• RN: track bundle size + native dep sizes per platform.

Per-PR budgets are one of the cheapest, highest-signal perf tests you can add.

8. Where to Run

• On-device for perf-sensitive metrics (startup, frames). Simulators and emulators are unreliable proxies for CPU/GPU perf.
• Same device class and battery / thermal state across runs. Device farms vary; prefer a reserved physical device pool for perf.
• Keep perf jobs separate from functional CI; mixed flakes hide real regressions.

9. Triage Workflow

When a perf alert fires:

1. Re-run N times to confirm it is not noise.
2. Bisect via git bisect between last known good and first bad commit.
3. Attribute with a profiler (Perfetto / Instruments / DevTools).
4. Land a fix or explicitly update the baseline with a note and approver.

10. Checklist

• At least one startup and one frame-time benchmark per platform.
• Baselines stored in the repo per device class; refreshed on a schedule, not silently overwritten.
• Thresholds are noise-aware (percentage + absolute).
• Binary size is tracked with a per-PR budget.
• Perf jobs run on real devices, pinned hardware where possible.
• Regressions trigger a documented triage workflow (re-run → bisect → profile → fix/baseline).
• Long-running metrics (battery, memory leaks) live in a nightly job, not per-PR.