Ios testing

iOS Testing Skill

You are an expert in Apple platform testing. Follow these rules strictly.

Decision flowchart (READ FIRST)

   ┌─────────────────────────────────┐
   │  User wants tests written or run │
   └────────────────┬────────────────┘
                    │
                    ▼
        ┌────────────────────────┐
        │  What are we testing?  │
        └─┬──────────┬──────────┬┘
          │          │          │
   logic/async    UI/screens   perf / energy / hitches
          │          │          │
          ▼          ▼          ▼
   ┌─────────┐ ┌──────────┐ ┌────────────────────┐
   │  Swift  │ │ XCUITest │ │  XCTest +          │
   │ Testing │ │  (part   │ │  XCTMetric         │
   │         │ │  of      │ │  subclasses.       │
   │ default │ │  XCTest) │ │  Physical device + │
   │ for new │ │          │ │  Release scheme    │
   │ tests   │ │          │ │  required for      │
   │         │ │          │ │  accurate numbers. │
   └────┬────┘ └─────┬────┘ └──────────┬─────────┘
        │            │                 │
        └──────┬─────┴─────────────────┘
               │
               ▼
   ┌──────────────────────────────────────────────────┐
   │   Test compiles + builds (xcodebuild build /      │
   │   build-for-testing / swift build / etc.)         │
   └──────────────────┬───────────────────────────────┘
                      │
                      ▼
   ┌──────────────────────────────────────────────────┐
   │   How do we run it?                              │
   ├──────────────────────────────────────────────────┤
   │  logic / unit       → iostesting test run        │
   │  app-hosted XCTest +→ IOSTESTING_BACKEND=fb       │
   │   Swift Testing       iostesting test run         │
   │  UI test bundle     → xcodebuild test (FB path    │
   │                       has iOS 26 quirks → 2.1)    │
   │  performance        → xcodebuild test on real     │
   │                       device, Release scheme,     │
   │                       perf-only scheme           │
   └──────────────────────────────────────────────────┘

The two iron rules: (1) do not invent assertion syntax — when in doubt about #expect(throws:), try #require, confirmation, or @Test(arguments:), read swift-testing.md before writing. (2) the test must run and pass before the change is done — writing is half the job; closing the loop with iostesting test run (or xcodebuild test for UI/perf) is the other half.

Framework Selection

Choose the correct framework based on what you are testing:

What you are testing	Framework	Import
Unit tests, integration tests, logic, async code	Swift Testing	import Testing
Performance benchmarks (CPU, memory, clock, storage)	XCTest	import XCTest
Energy/power measurement	XCTest	import XCTest
Animation hitches, scroll performance, render metrics	XCTest + XCUITest	import XCTest
UI automation, element interaction, accessibility audits	XCUITest	import XCTest
Light/dark mode appearance testing	XCUITest (launch args)	import XCTest
Contrast and visibility testing	XCUITest (accessibility audit)	import XCTest
Snapshot/visual regression testing	XCTest (+ swift-snapshot-testing)	import XCTest
Exit/crash tests (Swift 6.2+)	Swift Testing	import Testing
Analyzing .trace files from Instruments	xctrace CLI	N/A (command-line tool)
Generating App Store screenshots / marketing tiles	XCUITest (matrix + framing)	import XCTest

Critical Rules

1. Default to Swift Testing for all new unit and integration tests. Never use XCTestCase for pure logic tests.
2. Never mix assertion styles in a single test function. Do not use XCTAssertEqual alongside #expect, or vice versa.
3. Both frameworks can coexist in the same test target. Use import Testing for Swift Testing tests and import XCTest for XCTest/XCUITest tests — but keep them in separate files.
4. Use Swift Testing equivalents, not XCTest legacy:
   • #expect(a == b) not XCTAssertEqual(a, b)
   • #expect(condition) not XCTAssertTrue(condition)
   • try #require(optional) not XCTUnwrap(optional)
   • #expect(throws: ErrorType.self) { ... } not XCTAssertThrowsError
   • confirmation { } not XCTestExpectation + wait(for:)
   • @Test(.disabled("reason")) not XCTSkipIf / XCTSkipUnless
5. Use init() for setup in Swift Testing, not setUp() / setUpWithError().
6. Performance tests require XCTest. Swift Testing has no measure {} equivalent.
7. UI tests require XCUITest (which is part of XCTest). Swift Testing cannot drive UI.
8. Each test gets a fresh instance. Swift Testing creates a new suite instance per test — do not rely on shared mutable state between tests.
9. Do not invent assertion syntax. If you can't recall the exact shape of #expect(throws:), try #require, confirmation, or @Test(arguments:), read swift-testing.md before writing it. Hallucinated APIs compile-fail and waste a build cycle. The cost of looking is one read; the cost of guessing is one cycle.

Common Mistakes

Mistake	Why it's wrong	Use instead
#expect(throws: SomeError())	takes an error type or value-matching closure, not an instance	#expect(throws: SomeError.self) or #expect(throws: SomeError.tooShort) { ... }
try #require placed inside #expect(...)	#require is its own assertion — it doesn't nest	call let x = try #require(optional) on its own line, then #expect(x.foo == ...)
confirmation { confirm in confirm.fulfill() }	confused with XCTestExpectation	the parameter is called like a function: confirm()
@Test(arguments: collectionA, collectionB) for paired inputs	produces a cartesian product, not pairs	@Test(arguments: zip(collectionA, collectionB)) for paired
setUp() / setUpWithError() inside a Swift Testing struct	wrong framework — those are XCTest methods	implement init() (synchronous) or init() async throws (async) on the suite
XCTUnwrap(optional) inside a @Test function	mixed-framework call	try #require(optional)
XCTestExpectation + wait(for:timeout:) inside @Test	XCTest pattern in Swift Testing file	await confirmation("description", expectedCount: 1) { confirm in ...; confirm() }
measure { ... } inside @Test	Swift Testing has no perf primitive	move perf tests to an XCTestCase file and use measure(metrics:) with specific XCTMetric subclasses
app.staticTexts["Welcome"] matching localized text	breaks under any non-English locale	use accessibilityIdentifier and query with [.byId("welcome_screen_title")] (or your project's helper)
Running performance tests on simulator	sim CPU/memory metrics are unreliable, sim only supports Duration	physical device, Release config, perf-only test scheme
Only auditing accessibility on the first screen	misses regressions on flows	call app.performAccessibilityAudit(for: [.contrast, .dynamicType]) on every screen
Hardcoded UIColor.black / Color.white	breaks dark mode + dynamic type contrast	semantic: UIColor.label, .systemBackground, SwiftUI .primary

Best Practices

General

• F.I.R.S.T. principles: Fast (leverage default parallelism), Isolated (fresh instance per test), Repeatable (control all inputs), Self-Validating (use assertions, never print), Timely (write tests alongside production code).
• Do not weaken encapsulation for testing. If something is private, do not make it public just to test it. Test through the public API.
• Do not randomize test data. Tests must verify that fixed input produces fixed output. Use deterministic values.
• Mark XCTest methods as throws instead of using do/catch or try!. Force unwraps crash the entire suite and produce incomplete reports.
• Never compare floating-point values for exact equality. Use a tolerance: #expect(abs(result - 0.3) < 0.0001).

Swift Testing Specific

• Prefer #expect over #require for most assertions. Use #require only when continuing after failure is meaningless (e.g., unwrapping an optional needed by subsequent assertions).
• Do not overuse .serialized. It masks concurrency bugs and slows tests. Only use it temporarily for legacy non-thread-safe code.
• Do not add @MainActor to tests unnecessarily. It forces serial execution and defeats parallelism.
• Apply .timeLimit() to async tests to prevent CI hangs.
• Use zip() for paired test arguments, not separate collections (which create a cartesian product).
• Conform parameterized types to CustomTestStringConvertible for readable failure messages.

XCTest/XCUITest Specific

• Always use specific XCTMetric subclasses in measure(metrics:), not bare measure {}.
• Run performance and animation tests on physical devices. Simulators do not produce accurate hitch data and only support the Duration metric.
• Use a separate test scheme for performance tests with Release config, debugger disabled, screenshots off, code coverage off, and all diagnostics disabled.
• Always use accessibilityIdentifier for element queries, never localized display text.
• Set continueAfterFailure = false in UI test setUpWithError().

Reliability, Concurrency & the CI host

Tests that pass on an idle laptop but flake on a loaded/shared/newer-OS runner. Full patterns + code in test-reliability.md — read it before "fixing" a flaky or CI-only crash. Highest-value rules:

• Never assert a wall-clock UPPER bound (#expect(elapsed < X)). It tests machine speed, not correctness, and flakes under load. Keep meaningful lower bounds; use .timeLimit() to catch real hangs.
• Poll to a condition, don't sleep a fixed amount to wait for async work — a fixed wait under-waits a starved background task under load.
• When timing IS the contract, measure it relatively (compare two runs on the same machine — the difference is load-independent), never with an absolute bound.
• @MainActor types must not be decoded/used off the main thread. On the iOS 26+/Swift 6 runtime an off-main call into MainActor-isolated code is a hard EXC_BREAKPOINT/SIGTRAP (older runtimes silently continued → "passes locally, crashes in CI"). Codable DTOs/models should be nonisolated.
• Bridge continuations safely. A withCheckedContinuation over a callback API leaks and hangs forever if no callback fires — add a defensive timeout + a once-guarded resume. Mocks must invoke a reply/error handler like the real thing.
• Keep the unit-test host inert. Guard app-startup network/listeners/gates and crash-reporting SDKs (Sentry) on XCTestConfigurationFilePath — otherwise their background work trips the iOS watchdog and the crash lands on a random test.
• On CI: -parallel-testing-enabled NO on constrained runners; treat pure runner-flakiness crashes with -retry-tests-on-failure, not named-suite skips (whack-a-mole). Read the .xcresult/.ips before diagnosing.

Anti-Patterns to Avoid

• Writing multiple near-identical test functions instead of using @Test(arguments:).
• Using XCTestCase subclass with setUp()/tearDown() for pure logic tests.
• Mixing import Testing and import XCTest assertions in the same test function.
• Hardcoding timeouts without waitForExistence or expectations for async UI.
• Testing on simulator and expecting accurate performance/hitch metrics.
• Using .serialized or @MainActor as a default — these should be rare exceptions.
• Only testing in light mode — always test both light and dark appearances.
• Using hardcoded colors (UIColor.black, Color.white) instead of semantic colors.
• Running accessibility audits on only the first screen — audit every screen in the flow.
• Asserting a wall-clock upper bound (#expect(elapsed < 1.0)) — tests machine speed, not correctness; flakes under CI load. (See test-reliability.md §1.)
• A fixed Task.sleep to wait for async/background work instead of polling to a condition — under-waits a starved task under load. (§2)
• @MainActor on a Codable DTO/model that gets decoded off-main — a latent production SIGTRAP on iOS 26+ that only surfaces on newer CI runtimes. (§5)
• A withCheckedContinuation bridging a callback with no defensive timeout — leaks and hangs forever if the callback never fires. (§6)
• A test mock that records a call but invokes neither its reply nor error handler — hangs the awaiting caller. (§6)
• Booting the full app service stack (network, listeners, crash SDKs) in the unit-test host — background work trips the watchdog and crashes a random test. Guard on XCTestConfigurationFilePath. (§7)
• "Fixing" a flaky CI-only crash by skipping the named victim suite — the crash just moves to the next test. Use -retry-tests-on-failure and read the real crash log. (§8–9)
• Trusting .xctestplan skippedTests / -skip-testing: to quarantine a nested Swift Testing @Suite — it silently no-ops. Gate at the source with .enabled(if:)/.disabled. (§8)

Running Tests

Writing the test is half the job. The test must run and pass before the change is considered done.

This repo ships an iostesting CLI (in cli/) that drives simulators, physical devices, and .xctest bundles. When the user asks you to actually run the tests you just wrote, use iostesting rather than constructing raw xcodebuild test or xcrun simctl invocations.

Build with whatever build tool the project already uses, then run the resulting .xctest bundle:

# build the test bundle via the project's build tool (xcodebuild, swift build, etc.)
# then drive execution with iostesting:

iostesting sim list --state Booted                                # find or boot a sim
iostesting sim boot "iPhone 17 Pro"                               # idempotent
iostesting test list path/to/MyTests.xctest                       # confirm tests compiled in
iostesting test run path/to/MyTests.xctest --json                 # NDJSON events
iostesting test run path/to/MyTests.xctest --filter MySuite/testThing

When in doubt, ask the binary. Every command supports --examples:

iostesting test run --examples       # curated snippets — never guess flag shapes
iostesting sim location --examples
iostesting stop --examples

Two backends — pick per test type:

• Default IOSTESTING_BACKEND=simctl — shells to xcrun simctl spawn xctest. Logic tests only. Works on any Mac with Xcode installed.
• IOSTESTING_BACKEND=fb — direct linkage against XCTestBootstrap. Runs app-hosted XCTest + Swift Testing bundles end-to-end on iOS 26 simulators. Emits per-case NDJSON events (caseStarted/casePassed/caseFailed with durations + attachments). Set this env var when you need to run real test bundles via iostesting.

2.0.0 limitations:

• XCUITest via the FB backend has known iOS 26 quirks (DTX handshake stall / .xctestrun write fail). Use xcodebuild test for UI tests until 2.1.
• UI verbs beyond tap and swipe (find/wait/assert/type/screen+AX-tree) need FBAccessibilityElement bridging — queued for 2.1.
• Device runtime control + log streaming — device list/install/launch work via xcrun devicectl; richer device ops need FBDeviceControl wired through FBBridge.
• Performance tests still require physical hardware for accurate metrics. iostesting cannot make simulator perf numbers meaningful — that constraint is from XCTest itself.

Decision rule per test type:

• Unit / logic Swift Testing or XCTest, no host app → iostesting test run (default simctl backend is enough). Run after writing.
• App-hosted XCTest + Swift Testing bundles → IOSTESTING_BACKEND=fb iostesting test run. Auto-detects the host app from the bundle's .app/PlugIns/ parent.
• UI tests (XCUITest) → still xcodebuild test in 2.0.0. iostesting's XCUITest path lands cleanly in 2.1.
• Performance tests → physical device, Release scheme, xcodebuild test directly. Do not run on simulator.

Config makes flags implicit. Once iostesting config set --sim "iPhone 17 Pro" --bundle-id com.example.MyApp has been run, --sim and positional bundle ids become optional on every command. Use iostesting config show to see what's bound. For CI, set IOSTESTING_SIM and IOSTESTING_BUNDLE_ID env vars instead of writing a config file.

App registry + short IDs. iostesting launch records each launched app and prints a 6-character short ID. iostesting stop <short-id> and iostesting logs --bundle-id <short-id> resolve sim + bundle automatically — no need to retype them:

$ iostesting launch com.example.MyApp
Launched com.example.MyApp on iPhone 17 Pro (pid 12345) [id ab12kw]

$ iostesting logs --bundle-id ab12kw       # sim + bundle resolved from registry
$ iostesting stop ab12kw

iostesting apps list shows the registry; iostesting apps prune clears terminated records.

Physical devices. iostesting device list shows paired devices; device install and device launch work via xcrun devicectl. Device log streaming and richer runtime control land in a follow-up release via FBDeviceControl.

Other useful sim ops you don't have to remember the simctl flags for:

iostesting sim appearance light          # or dark — for screenshot tests
iostesting sim media-add ./fixtures/*.jpg
iostesting sim location --set "37.7749,-122.4194"
iostesting sim open-url myapp://reset-password
iostesting screenshot -o ./shot.png

SpringBoard alerts XCUITest can't reach (iOS 26)

XCUITest's addUIInterruptionMonitor is scoped to the app under test. It catches the initial CLLocationManager prompt ("When in Use / Once / Don't Allow") but reliably misses SpringBoard-owned modals that fire mid-test — most notably iOS 26's location-upgrade prompt:

Allow "X" to also use your location even when you are not using the app? [Keep Only While Using] [Change to Always Allow]

The test stalls while the runner tries to interact with elements behind the modal, eventually hitting --termination-timeout with "host application process stalled".

Two layered fixes, both wired into iostesting test run:

1. Pre-grant the privacy state before the runner launches. simctl privacy grant location-always writes Authorization=4, AuthorizationUpgradeAvailable=false into locationd's clients.plist. The upgrade prompt then never fires because iOS sees the auth is already at Always:

   IOSTESTING_BACKEND=fb iostesting test run ./MyAppUITests.xctest \
     --privacy-grant all-location
   

   The grant MUST happen after the app is installed (locationd needs the bundle path). iostesting test run does the install + grant in the right order.

2. Background watcher with OCR-driven dismissal. When a prompt is going to fire regardless (notifications, tracking, second-time grants), spawn a watcher that screenshots the sim every 2s, OCRs the screen with Vision, and taps any matching button:

   IOSTESTING_BACKEND=fb iostesting test run ./MyAppUITests.xctest \
     --auto-dismiss-alerts
   

   Belt-and-suspenders combination:

   IOSTESTING_BACKEND=fb iostesting test run ./MyAppUITests.xctest \
     --privacy-grant all-location \
     --auto-dismiss-alerts \
     --termination-timeout 300
   

Defaults cover the common labels: Keep Only While Using, Allow While Using App, Allow Once, Don't Allow, OK, Allow. Override with --auto-dismiss-label.

Standalone alert tools (when not running a full test):

IOSTESTING_BACKEND=fb iostesting alerts dismiss            # one-shot, exit 2 on no match
IOSTESTING_BACKEND=fb iostesting alerts watch --duration 120 --json
IOSTESTING_BACKEND=fb iostesting privacy grant --all-location com.example.MyApp

When to reach for which:

• App requires "Always" location for a background task → --privacy-grant all-location.
• App requires a one-time When-In-Use grant and you still want to test the prompt path → --auto-dismiss-alerts (the test still sees the prompt fire; the watcher handles it if XCUITest's monitor fails).
• App fires a notifications/tracking prompt mid-flow → --auto-dismiss-alerts (no --privacy-grant for that service exists; simctl doesn't expose notification-permission grants).

Optional enforcement hook. The repo ships hooks/iostesting-guard.sh, a Claude Code PreToolUse hook that blocks xcrun simctl boot/install/launch/... and xcodebuild test and suggests the iostesting equivalent. See hooks/README.md for install instructions. Without the hook the skill is advice; with it, it's a contract.

Analyzing .trace Files

You can read and analyze Xcode Instruments .trace files directly. When a user provides a .trace file:

1. Export the TOC with xctrace export --input /path/to/file.trace --toc to discover available data tables.
2. Export specific tables to a file using --xpath and --output /tmp/trace_data.xml (piping large XML loses ref context).
3. Parse with Python (xml.etree.ElementTree) to resolve the id/ref deduplication system and extract meaningful data.
4. Suggest code improvements based on the findings.

Critical Gotchas

• Use time-profile schema (symbolicated frames), NOT time-sample (raw kperf-bt hex addresses). This is the #1 time-waster.
• --filter-time does not exist — export full data and filter programmatically.
• Ref resolution is mandatory — threads, states, and frames use id="N"/ref="N" deduplication. Build lookup dicts.
• For hangs: export potential-hangs first for time ranges, then time-profile filtered to main thread + Running state.

See the trace analysis reference for the complete guide including hang analysis workflow with Python script, .trace file structure, all schemas, and XML format details.

Generating App Store Screenshots

This skill can also generate App Store screenshots — not just test an app, but drive it through every marketing state and produce framed store tiles. The same XCUITest muscle (launch args, accessibility identifiers, XCUIScreenshot) powers an end-to-end pipeline: capture deterministic states → apply a 9:41 status bar → extract from the .xcresult → composite into 1320×2868 store tiles with a baked headline.

Reach for this when the user asks to "generate App Store screenshots", "capture screenshots for the store", "make screenshot tiles", or "frame screenshots for submission".

Key moves (full playbook in the reference):

1. Matrix harness — one XCUITest iterates Config × ScreenKind, relaunching a fresh app per capture and attaching each PNG. Prove it with a smoke test first.
2. Deterministic seeding — seed the data layer before the scene renders; full-history seeds must wipe ALL prior rows or re-runs accumulate duplicates and silently misclassify.
3. 9:41 status bar — xcrun simctl status_bar <udid> override on the booted sim, then test-without-building on that exact id=<udid> so no reboot wipes it.
4. Extract + frame — xcresulttool export attachments maps UUIDs back to slug/screen.png; PIL composites them onto brand-framed store canvases. Archive the submission build with a stable Xcode (beta Xcode → ITMS-90534 on upload).

See the App Store screenshot pipeline for the complete runnable playbook including the TEST_RUNNER_ env-forwarding gotcha, the fileSystemSynchronizedGroup landmine, the reusable PIL framing snippet, and ASC upload specifics.

Reference Files

For detailed patterns and code examples, see:

• Swift Testing patterns and examples
• XCTest performance, power, and energy testing
• XCUITest UI automation and animation testing
• Test reliability: concurrency, timing, and the CI host — flaky-timing tests, iOS 26+ actor-isolation crashes, continuation hangs, test-host hygiene, CI runner flakiness, crash-log diagnosis
• Instruments .trace file analysis
• App Store screenshot pipeline — capture matrix, 9:41 status bar, deterministic seeding, .xcresult extraction, PIL framing into store tiles, ASC upload
• For running tests via the iostesting CLI: see ../../cli/README.md in this repo.

When Generating Tests

1. Ask yourself: "Is this testing logic/behavior, or performance/UI?" Pick the framework accordingly.
2. For Swift Testing: use free functions or structs, not classes. Use @Test and @Suite.
3. For parameterized tests: always prefer @Test(arguments:) over writing multiple similar test functions.
4. For async code: Swift Testing supports async natively. Use confirmation() for callback-based APIs.
5. For performance: use measure(metrics:) with specific XCTMetric subclasses, not bare measure {}.
6. For UI tests: always use accessibility identifiers, never match on localized strings.
7. For error testing: use #expect(throws:) with the specific error value or type.
8. For known bugs: use withKnownIssue("description") { } instead of disabling the test.
9. For appearance testing: test both light and dark mode using launch arguments + overrideUserInterfaceStyle.
10. For contrast/visibility: run performAccessibilityAudit(for: .contrast) in both appearance modes.
11. Use semantic/system colors (UIColor.label, .systemBackground, SwiftUI .primary) — never hardcode colors.