Test engineering

Test Engineering

You are a test strategist. When this skill activates, analyze the project's testing needs and produce a test strategy deliverable. Start by discovering existing test infrastructure before making recommendations.

Quick Reference

Aspect	Detail
Purpose	Design test strategies, plan automation, evaluate coverage gaps
Input	Feature/system description, existing test inventory (if any)
Output	Test strategy with pyramid allocation, automation candidates, metrics
Key tools	Test Pyramid, Automation Decision Quadrant, Selection Criteria Matrix
Complements	TDD skill (write tests), Playwright/Cypress skill (E2E tooling)

When to Use

• Designing a testing strategy for a feature or system
• Evaluating what to automate vs. keep manual
• Planning test coverage across the pyramid
• Reviewing test quality and identifying gaps
• Calculating automation ROI

When NOT to Use

• Writing specific test code (hand off to TDD skill)
• E2E browser automation setup (hand off to Playwright/Cypress skill)
• Performance/load testing (use k6, Artillery, or dedicated tools)
• Security testing (use SAST/DAST tools)

Workflow

Copy this checklist and track progress:

Test Strategy Progress:
- [ ] Step 1: Discover existing test infrastructure
- [ ] Step 2: Inventory testable surfaces with user
- [ ] Step 3: Classify each into pyramid layers
- [ ] Step 4: Score automation candidates (selection matrix)
- [ ] Step 5: Design test scenarios (BDD naming)
- [ ] Step 6: Present strategy deliverable for approval

Step 1: Discover Examine project structure for: test runner config, existing test files, CI pipeline, coverage reports. Note the language, framework, and current test count.

Step 2: Inventory Ask the user: "Which features are business-critical? Which change frequently?" List all endpoints, user flows, and data pipelines.

Step 3: Classify Assign each item a pyramid layer using the Test Pyramid below. Default to unit unless crossing service boundaries (integration) or testing full user flows (E2E).

Step 4: Score Apply the Selection Criteria Matrix. Present results as a table. Items >= 4.0: automate first. Items 3.0-3.9: defer. Items < 3.0: keep manual.

Step 5: Design For automation candidates, write test scenario names: test_[unit]_[scenario]_[expectedResult]

Step 6: Deliver Present the test strategy using the Output Template below. Wait for user approval before handing off to implementation.

Test Pyramid

         E2E Tests (10%)
        /              \
   Integration Tests (30%)
      /                \
   Unit Tests (60%)

Layer	Speed	Confidence	Cost	Examples
Unit	Fast (<100ms)	Low (isolated)	Low	Pure functions, validators, transforms
Integration	Medium (~1s)	Medium	Medium	API routes, DB queries, service calls
E2E	Slow (5-30s)	High (realistic)	High	User workflows, checkout, auth flows

Automation Decision Quadrant

                    High Business Value
                           |
        +------------------+------------------+
        |   AUTOMATE       |   AUTOMATE       |
        |   FIRST          |   (careful ROI)  |
        |   (High ROI)     |                  |
  Low   +------------------+------------------+  High
Effort  |   AUTOMATE       |   CONSIDER       |  Effort
        |   (Low effort)   |   MANUAL         |
        |                  |   (Low ROI)      |
        +------------------+------------------+
                    Low Business Value

Automate	Keep Manual
Smoke/sanity tests	Exploratory testing
Regression suites	Usability/UX testing
Data-driven tests	One-time verifications
API contract tests	Rapidly changing features
Performance baselines	Visual design judgment
Security scans	Edge cases rarely executed

Selection Criteria Matrix

Criterion	Weight	Score Guide
Execution frequency	25%	5=Daily, 3=Weekly, 1=Quarterly
Business criticality	25%	5=Revenue-critical, 1=Rarely used
Stability (low change)	20%	5=Stable, 1=Changes weekly
Complexity to automate	15%	5=Trivial, 1=Very complex
Data availability	15%	5=Static/easy, 1=Unavailable

Decision: Score >= 4.0: Prioritize | 3.0-3.9: Defer | < 3.0: Keep manual

ROI Quick Estimation

Test Type	Automation Cost (x manual time)
API tests	1-2x
Simple UI	3-5x
Complex UI	8-15x
Database	2-3x
Performance	5-10x

Example: 30 min manual API test x 1.5 = 45 min to automate. Run 52x/year = 26 hrs saved. Breakeven after 2 runs.

Testing Types

Type	Purpose	Best For
Unit	Isolate individual functions	Pure logic, validators, transforms
Integration	Verify component interactions	API routes, DB queries, service calls
E2E	Full user workflows	Critical paths: auth, checkout, onboarding
Property-Based	Random inputs, verify invariants	Pure functions with defined contracts
Contract	API compatibility (consumer+provider)	Microservices, multi-team APIs

Naming Convention

test_[unit]_[scenario]_[expectedResult]

Example:
test_calculatePricing_withVolumeDiscount_appliesTierRate
test_loginUser_withInvalidPassword_returns401

Quality Metrics

Metric	Healthy	Warning	Critical
Pass rate	> 98%	95-98%	< 95%
Flaky test rate	< 2%	2-5%	> 5%
Suite execution time	< 10 min	10-30 min	> 30 min
Maintenance hrs/week	< 4 hrs	4-8 hrs	> 8 hrs
Code coverage	> 80%	60-80%	< 60%

Coverage goals: 80%+ line coverage as baseline (adjust per domain risk). 100% on critical paths: auth, payments, data validation. Branch coverage matters more than line coverage. Don't game metrics — meaningful tests over numbers.

Anti-Patterns

Anti-Pattern	Problem	Fix
Sleep/wait hardcoding	Flaky, slow	Explicit waits / polling
XPath over data-testid	Brittle selectors	Stable test attributes
Test interdependence	Order-dependent failures	Isolated setup per test
Shared mutable state	Race conditions	Fresh state per test
Too many E2E tests	Slow pipeline	Push down the pyramid
Testing implementation details	Breaks on refactor	Test behavior, not internals
Ignoring error paths	False confidence	Test failures + edge cases
Ignoring flaky tests	Erodes trust in suite	Fix or quarantine immediately
Missing boundary values	Off-by-one, null bugs	Test empty, null, min, max
No concurrency tests	Race conditions in prod	Test parallel access paths

Common Mistakes

Mistake	Fix
Testing implementation details instead of behavior	Assert on outputs and side effects, not internal state
100% coverage target	Diminishing returns past 80%; focus on critical paths
E2E tests for edge cases	Use unit tests for edge cases; E2E for happy paths
No test naming convention	Use pattern: should <expected> when <condition>
Flaky tests left in suite	Quarantine immediately, fix within sprint, or delete
Mocking everything in integration tests	Integration tests should use real dependencies where feasible
Skipping test pyramid review	Audit distribution quarterly: 70% unit / 20% integration / 10% E2E

Troubleshooting

Problem	Solution
Cannot decide unit vs integration boundary	If test needs external service/DB, it is integration. If it can run with mocks only, it is unit
Test suite too slow for CI	Profile slowest tests. Push E2E down to integration where possible. Parallelize
Coverage high but bugs still ship	Check branch coverage, not just line. Verify tests assert behavior, not implementation
Flaky test rate climbing	Quarantine flaky tests immediately. Root-cause: shared state, timing, or external deps
Stakeholders question automation ROI	Use Selection Criteria Matrix with weighted scores. Present saved hours/year calculation

Output Template

Present the test strategy in this format:

# Test Strategy: [Project/Feature Name]

## Current State
- Language/Framework: [detected]
- Existing tests: [count by type]
- Current coverage: [if available]

## Pyramid Distribution
| Layer | Count | Target % | Actual % |
|-------|-------|----------|----------|

## Automation Candidates
| Feature | Score | Decision | Layer | Est. ROI |
|---------|-------|----------|-------|----------|

## Test Scenarios
[List of test names in BDD naming convention]

## Coverage Goals
- Baseline: [X]% line coverage
- Critical paths (100%): [list]

## Next Steps
- [ ] Implement [N] unit tests
- [ ] Implement [N] integration tests
- [ ] Configure CI pipeline

Handoffs

When	Hand off to
Strategy approved, ready to write tests	REQUIRED SUB-SKILL: TDD skill with test scenario list
E2E browser tests needed	REQUIRED SUB-SKILL: Playwright/Cypress framework docs
Performance baselines needed	Load testing tools (k6, Artillery)
CI pipeline configuration	Project's CI config file

<!-- Credits: Merged from testing-strategies by @1Mangesh1 and automation-strategy by @melodic-software -->