Visual regression

Visual Regression

Instructions

Visual regression is how a design system stays a system. Every component × variant × theme must snapshot on every change, and diffs must block merges.

1. Coverage Contract

For every component, snapshot the full matrix:

• All variants (size × intent).
• Both themes (light + dark), plus any brand themes in the default bundle.
• Key states (default, focused, disabled, loading).
• Two content densities (short + long text, RTL if supported).
• Two dynamic type levels (default + 200%).

At minimum: variants × 2 themes × 4 states ≈ 40 snapshots per non-trivial component.

2. Per-Platform Tooling

Platform	Tool
Compose	Roborazzi (JVM, fast) or Paparazzi
SwiftUI / UIKit	swift-snapshot-testing (pointfreeco)
Flutter	flutter_test goldens + golden_toolkit / alchemist
Storybook (RN/web)	Chromatic, Loki, or Percy

3. Compose — Roborazzi

// build.gradle.kts
plugins { id("io.github.takahirom.roborazzi") }

dependencies {
    testImplementation("io.github.takahirom.roborazzi:roborazzi:1.+")
    testImplementation("io.github.takahirom.roborazzi:roborazzi-compose:1.+")
}

// ButtonSnapshotTest.kt
@RunWith(RobolectricTestRunner::class)
class ButtonSnapshotTest {
    @get:Rule val composeRule = createComposeRule()

    @Test fun `primary medium light`() {
        composeRule.setContent { AppTheme(darkTheme = false) {
            Button(onClick = {}, label = "Save")
        }}
        composeRule.onRoot().captureRoboImage("out/button_primary_md_light.png")
    }
}

Run ./gradlew recordRoborazziDebug to regenerate; verifyRoborazziDebug fails on diffs.

4. SwiftUI — swift-snapshot-testing

import SnapshotTesting
import XCTest
@testable import DesignSystem

final class DSButtonTests: XCTestCase {
    func test_primary_medium_light() {
        let view = DSButton("Save", intent: .primary, size: .medium) {}
            .frame(width: 320, height: 48)
            .environment(\.colorScheme, .light)
        assertSnapshot(of: view, as: .image(layout: .fixed(width: 320, height: 48)))
    }
}

Baselines live beside the test. CI fails if a snapshot differs beyond the perceptual threshold.

5. Flutter — Golden Tests

testWidgets('DsButton primary medium light', (tester) async {
  await tester.pumpWidgetBuilder(
    DsButton(label: 'Save', onPressed: () {}, intent: DsButtonIntent.primary),
    wrapper: materialAppWrapper(theme: AppTheme.light()),
  );
  await screenMatchesGolden(tester, 'button_primary_md_light');
});

Use alchemist or golden_toolkit to run goldens reliably on CI (consistent font + DPI). Keep a single CI-only image baseline; developer-machine mismatches are not investigated.

6. Storybook — Chromatic / Loki / Percy

One story per variant; the tool diffs per story per browser.

export default { title: 'Primitives/Button', component: Button };

export const PrimaryMedium = () => <Button intent="primary" size="md">Save</Button>;
export const DangerLarge   = () => <Button intent="danger"  size="lg">Delete</Button>;

Chromatic: npx chromatic --exit-zero-on-changes on PRs; blocking merges on main.

7. Thresholds and Flakiness

• Allow a small pixel-diff tolerance (e.g., 0.1%) for anti-aliasing drift across OS versions.
• Lock font metrics: bundle test fonts, disable system font loading in tests.
• Fix device configuration: one device / screen size / density per test class. Never run visual tests on a matrix of emulators.
• Freeze animations: set animation scale to 0 in tests.

8. Workflow

1. Developer makes a change.
2. CI runs visual tests; diffs are uploaded as PR artifacts.
3. Diffs are reviewed in the PR (Chromatic inline, or CI HTML report for native).
4. If intentional, baseline is updated in the same PR; if not, the code is fixed.

9. When to Skip

• Purely logical components (utilities, hooks) — unit tests suffice.
• Animations that evolve frame-by-frame — record fixed-time snapshots (t=0, t=mid, t=end) rather than video.

10. Anti-Patterns

• Snapshot baselines regenerated outside CI — drift between machines.
• Running visual tests only on main — regressions discovered after merge.
• Massive single-component snapshots (screenshot of a whole screen) — hide the diff that matters.
• Ignoring flake by raising the tolerance — fix the root cause (fonts, device, animations).
• No dark-theme snapshots — the dark theme rots silently.

Checklist

• Every component snapshots its full variant × theme × state matrix.
• Platform tools are chosen (Roborazzi/Paparazzi, swift-snapshot-testing, flutter goldens, Chromatic/Loki/Percy) and wired into CI.
• Baselines are generated only in CI; developer machines verify, never record.
• Fonts, DPI, animations, and device configuration are pinned in test setup.
• Visual diffs block merge; changelogs reference the PR diff when intentional.
• Dark theme and any shipped brand themes are covered.