Device farms

Skill almasumdev/awesome-mobile-testing-agent-skills/.github/skills/e2e/device-farms

Agent skills for unit, widget, UI, and end-to-end testing of mobile apps across platforms.

Install

npx -y skills add almasumdev/awesome-mobile-testing-agent-skills --skill device-farms

Assembled from the repository path, not quoted from the project. Check it against their README if it does not work.

2 things to look at

no licenseNo license file was found in the repository. Code published without one is not open source by default, so using it at work is a question for whoever answers licensing questions where you are.
1 stars1 stars. Stars are a popularity signal and not a quality one, but at this level it is likely that nobody has read this closely except its author, and you would be relying on your own review.

What its author says it does

Copied from the file, not written here

Expert guidance on running mobile tests on Firebase Test Lab, AWS Device Farm, BrowserStack App Automate, and Sauce Labs. Use when asked to set up device-farm coverage, design a device matrix, or compare vendors.

SKILL.md

5.3 KB, as published. Nobody here has run it

Device Farms

Instructions

Device farms provide breadth of OS and hardware coverage no local lab can match. The goal is not to run every test on every device — that is expensive and rarely informative — but to cover the shapes of risk (min OS, latest OS, one low-end device, one form-factor outlier).

1. Vendor Snapshot

Vendor	Platforms	Strengths	Gotchas
Firebase Test Lab	Android (physical + virtual); iOS (XCTest via `xctestrun`)	Deep GCP integration, Robo crawler, generous free tier for OSS	iOS support is narrower; shell access limited
AWS Device Farm	Android + iOS	Appium, Espresso, XCUITest, `flutter drive`; per-minute pricing	UI is dated; artifact retention short by default
BrowserStack App Automate	Android + iOS (real devices)	Wide device matrix, fast session start, good iOS coverage	Interactive/manual sessions can compete for device pools
Sauce Labs Real Device Cloud	Android + iOS	Enterprise reporting, device reservations, Appium/XCUITest/Espresso	Pricing tier complexity
Maestro Cloud	Android + iOS	Native Maestro integration, auto-sharding	Only runs Maestro flows, not Espresso/XCUITest

Pick one primary and one fallback. Multi-vendor abstraction layers rarely pay off — tests lose features across the lowest common denominator.

2. Device Matrix Policy

A good matrix is small and intentional:

Android: min SDK + latest stable + 1 low-RAM device (e.g., Pixel 4a) + 1 tablet / foldable if relevant.
iOS: min iOS supported + latest iOS + one iPhone SE (small screen) + one iPad if the app supports it.

Run smoke on a single device per platform per PR. Run the full matrix nightly or on release branches.

3. Firebase Test Lab (Android)

From CI:

gcloud firebase test android run \
  --type instrumentation \
  --app app/build/outputs/apk/debug/app-debug.apk \
  --test app/build/outputs/apk/androidTest/debug/app-debug-androidTest.apk \
  --device model=redfin,version=33,orientation=portrait \
  --device model=oriole,version=34,orientation=portrait \
  --timeout 20m \
  --use-orchestrator \
  --num-flaky-test-attempts 1

Use --results-dir to pin artifact paths and --test-targets "class com.example.SmokeSuite" to shard.

For iOS XCTest on FTL, build a .xctestrun and .xcarchive, zip, and upload via gcloud firebase test ios run.

4. AWS Device Farm

aws devicefarm schedule-run \
  --project-arn "$PROJECT_ARN" \
  --app-arn "$APK_ARN" \
  --device-pool-arn "$POOL_ARN" \
  --name "pr-$CI_SHA" \
  --test type=INSTRUMENTATION,testPackageArn="$TEST_APK_ARN"

Artifacts (video, logcat, test specs) are downloadable per run. Keep runs under 150 min; longer runs are aborted on some tiers.

5. BrowserStack App Automate — Example (Espresso)

curl -u "$BS_USER:$BS_KEY" \
  -X POST "https://api-cloud.browserstack.com/app-automate/espresso/v2/build" \
  -d '{
    "app":        "bs://APP_HASH",
    "testSuite":  "bs://TESTSUITE_HASH",
    "devices":    ["Google Pixel 8-14.0","Samsung Galaxy S24-14.0"],
    "deviceLogs": true, "networkLogs": true, "video": true
  }' -H "Content-Type: application/json"

BrowserStack's REST API is thin and CI-friendly. For Maestro, use their official GitHub Action.

6. Sauce Labs — RDC

Sauce uses a single saucectl config for multiple frameworks:

apiVersion: v1alpha
kind: espresso
sauce:
  region: us-west-1
  concurrency: 3
espresso:
  app: ./app-debug.apk
  testApp: ./app-debug-androidTest.apk
suites:
  - name: smoke
    devices:
      - name: "Google Pixel 8"
        platformVersions: ["14"]

7. Artifacts and Debugging

Every farm run should produce, at minimum:

Video of the session.
Logcat (Android) / syslog (iOS).
Screenshots for each failing assertion.
Test report in JUnit XML.
Where supported, network HAR.

Fail the CI job loudly if artifacts are missing — "no repro" without artifacts is a process bug.

8. Cost Control

Cap per-run duration (--timeout, farm-specific).
Shard big suites; short jobs run in parallel for less wall-clock.
Use virtual devices on FTL for the bulk of runs; reserve physical devices for hardware-specific checks.
Nightly full-matrix runs are cheaper per finding than full-matrix on every PR.

9. Flake on Farms

Device farms amplify any non-determinism in your tests. If a test flakes locally at 1 %, it will flake at 3–5 % on a shared pool. Fix locally first (see flaky-test-reduction) — never paper over farm flake with high retry counts.

10. Checklist

Primary farm selected; secondary is documented as a fallback only.
Device matrix covers min OS, latest OS, one low-end device, and relevant form factors — and nothing else.
Smoke runs one device/platform per PR; full matrix runs nightly or on release branches.
Every run publishes JUnit XML, video, logs, and screenshots.
Per-run timeout caps are set.
Sharding is configured for suites > 10 min wall-clock.
Flake is fixed locally before being tolerated on the farm.