Parallel test execution

Parallel Test Execution

Instructions

Parallel execution cuts wall-clock time — when done right. Done wrong, it multiplies flake, inflates cost, and makes failures harder to debug. The win comes from sharding the right work onto the right runner and from isolating state so parallel tests cannot collide.

1. Two Kinds of Parallelism

1. Intra-JVM / in-process parallelism — many tests run on threads of the same runner.
2. Inter-runner / inter-device sharding — tests are split across multiple machines or devices.

Use both, but for different layers:

• Unit + integration → intra-process parallelism (fast, cheap).
• UI + E2E → inter-device sharding (one device per shard, expensive, use sparingly).

2. Intra-Process Parallelism

Gradle (Android/Kotlin):

tasks.withType(Test).configureEach {
    maxParallelForks = Runtime.runtime.availableProcessors().intdiv(2) ?: 1
    forkEvery = 100
    systemProperty 'junit.jupiter.execution.parallel.enabled', 'true'
    systemProperty 'junit.jupiter.execution.parallel.mode.default', 'concurrent'
}

Swift:

xcodebuild test ... -parallel-testing-enabled YES \
  -parallel-testing-worker-count 4 \
  -maximum-concurrent-test-simulator-destinations 2

Flutter / Dart:

dart test --concurrency=4
flutter test --concurrency=4

Jest / RN:

jest --maxWorkers=50%

Each runner needs isolated temp dirs, DB files, and ports. Use UUID-based file names and PortProvider-like helpers. Never hard-code port 8080.

3. Inter-Runner Sharding (CI Matrix)

Split by test class or by weight. Prefer weight-based splits so shards finish at similar times.

GitHub Actions sketch:

strategy:
  fail-fast: false
  matrix:
    shard: [1, 2, 3, 4]
steps:
  - run: ./gradlew testDebugUnitTest \
           -Pshard.index=${{ matrix.shard }} \
           -Pshard.total=4

On the Gradle side, use a test filter plugin (e.g., gradle-test-distribution, Develocity, or a hand-rolled sharder keyed by class hash) to pick the subset.

4. Device Sharding

• Firebase Test Lab: --num-shards auto-splits by Smart Sharding (based on past durations). Supply --shard-index if driving sharding yourself.
• AWS Device Farm: define multiple device pools or split test specs.
• BrowserStack Espresso / XCUITest: shards field in the build JSON.
• Maestro Cloud: auto-shards across the provided flows directory.

Shard by flow duration, not flow count. A 10-flow shard with one 5-minute flow finishes later than a 30-flow shard of 10-second flows.

5. Isolating State for Parallel Runs

• Databases: per-test in-memory DB (no file handle sharing).
• HTTP mock servers: let the server bind port 0 and read back the assigned port.
• Seeded users on device farms: embed $RUN_ID and $SHARD_INDEX in user names/emails.
• Disk: per-test temp dir (TemporaryFolder, FileManager.default.temporaryDirectory.appendingPathComponent(UUID().uuidString)).
• Global singletons: wrap in a provider you can reset per-test.

6. Reporting Merged Results

• Produce JUnit XML per shard.
• Merge at the end with a tool (junit-merge, xcresulttool merge, custom script) so the PR sees a single summary.
• Upload all per-shard artifacts — logs, screenshots, videos — under a shard-prefixed path.

7. Cost Tradeoffs

• Each additional CI runner has a startup cost (checkout, dependency install). Very short shards are dominated by overhead.
• Target 3–6 minutes of test work per shard for non-device jobs; 10–15 minutes for device-farm shards.
• Device farms charge per device-minute; 4 shards on 4 devices costs 4× a single device, not ¼.

8. Debugging a Parallel Failure

• Reproduce the exact shard locally with the same --shard-index / filter.
• If the failure is shard-specific but not test-specific, the culprit is shared state.
• Run the failing test alone and inside the original shard to confirm it is a parallel-interaction bug.

9. Anti-Patterns

• Parallelizing E2E on shared backend data (cross-test races).
• Running 16 Android emulators on one runner (thermal throttling; flake explodes).
• Sharding by filename alphabetical order (durations drift over time).
• Retrying a whole shard to paper over one flaky test.

10. Checklist

• Unit + integration run with intra-process parallelism sized to the runner.
• UI/E2E shard across runners/devices, not inside one runner.
• Sharding is weight-based, not count-based.
• DB, HTTP ports, disk, and users are isolated per shard.
• JUnit XML is produced per shard and merged for the PR summary.
• Each shard has a target work duration (3–6 min off-device; 10–15 min on-device).
• Parallel-only failures are reproducible locally with the same shard index.