Solution testing

是什么

这是一份方案级端到端测试规范，覆盖 Playwright（浏览器自动化框架）E2E（端到端）测试、部署后冒烟测试、浏览器自动化稳定性、BDD（行为驱动开发）特性文件编写，让团队从用户视角验证核心链路是真的能跑通。

怎么用

1. 梳理产品核心用户旅程，按本文档的优先级矩阵选出 5-10 条主路径写 Playwright E2E 测试。
2. 每次部署后立刻跑冒烟测试套件，10 分钟内确认登录、下单、支付等关键流程没坏。
3. 浏览器测试容易抖动时，按文档的稳定性清单（显式等待、独立账号、清理数据）逐项排查 flaky 用例。
4. 业务方提需求时用 Gherkin（BDD 描述语言）写 Given-When-Then 三段式描述，让测试用例和产品文档对齐。
5. 测试报告里同时附上失败截图和 trace，方便定位前端、后端、网络三类问题，缩短定位耗时。

架构图

flowchart LR
    A[用户旅程清单] --> B[Playwright 脚本]
    B --> C[本地预演]
    C --> D[CI 冒烟跑]
    D --> E{通过?}
    E -->|否| F[截图与 trace]

Solution Testing

End-to-end and acceptance testing techniques for verifying that a feature works correctly across the full stack — browser, API, and data layer — from the user's perspective.

When to Activate

• Writing browser automation tests for user journeys
• Verifying a full feature works end-to-end (UI through DB)
• Setting up Playwright or Cypress for a project
• Writing BDD feature files with Gherkin syntax
• Designing smoke tests for post-deployment verification
• Debugging flaky E2E tests
• Deciding how many E2E tests to write for a feature

E2E vs Integration: The Boundary

E2E tests cover things integration tests cannot:

• Real browser rendering and JavaScript execution (layout, event handling, hydration)
• Full stack traversal: UI → API → DB → UI response cycle
• Multi-step user journeys across pages, sessions, and auth boundaries

Cost of Each Test Level

Type	Speed	Flakiness Risk	Maintenance Cost
Unit	ms	Very low	Low
Integration	seconds	Low	Medium
E2E	10s–minutes	High	High

The Honeycomb Model

Prefer more service-level integration tests over E2E tests. E2E tests are expensive to write, slow to run, and prone to flakiness. Use them sparingly.

• Write E2E tests only for critical user journeys: login, checkout, core business workflows
• Do not write E2E tests for every edge case — cover those with unit and integration tests
• Aim for: many unit tests → more integration tests → few targeted E2E tests

Playwright Setup and Patterns

Project Setup

npm init playwright@latest
# or add to an existing project:
npm install -D @playwright/test
npx playwright install

Config (playwright.config.ts):

import { defineConfig, devices } from '@playwright/test';

export default defineConfig({
  testDir: './e2e',
  fullyParallel: true,
  retries: process.env.CI ? 2 : 0,
  reporter: [['html'], ['list']],
  use: {
    baseURL: process.env.BASE_URL ?? 'http://localhost:3000',
    screenshot: 'only-on-failure',
    video: 'retain-on-failure',
    trace: 'on-first-retry',
  },
  projects: [
    { name: 'chromium', use: { ...devices['Desktop Chrome'] } },
  ],
  webServer: {
    command: 'npm run start',
    url: 'http://localhost:3000',
    reuseExistingServer: !process.env.CI,
  },
});

Page Object Model (POM)

Each page or major component has a class that encapsulates its selectors and actions. Test files use POM methods — never raw locators.

// pages/login.page.ts
import { Page, Locator } from '@playwright/test';

export class LoginPage {
  private readonly emailInput: Locator;
  private readonly passwordInput: Locator;
  private readonly submitButton: Locator;

  constructor(private page: Page) {
    this.emailInput = page.getByLabel('Email');
    this.passwordInput = page.getByLabel('Password');
    this.submitButton = page.getByRole('button', { name: 'Sign in' });
  }

  async goto() {
    await this.page.goto('/login');
  }

  async login(email: string, password: string) {
    await this.emailInput.fill(email);
    await this.passwordInput.fill(password);
    await this.submitButton.click();
  }
}

// e2e/auth.spec.ts
import { test, expect } from '@playwright/test';
import { LoginPage } from '../pages/login.page';

test('user can log in with valid credentials', async ({ page }) => {
  const loginPage = new LoginPage(page);
  await loginPage.goto();
  await loginPage.login('[email protected]', 'password123');
  await expect(page).toHaveURL('/dashboard');
});

Locator Strategy (Priority Order)

Locator	Example	Why preferred / when to use
getByRole	getByRole('button', { name: 'Submit' })	Accessibility-based, most stable, mirrors how users perceive UI
getByLabel	getByLabel('Email address')	Form inputs — semantically tied to label text
getByText	getByText('Welcome back')	Unique visible text content
getByTestId	getByTestId('submit-btn')	When no semantic selector works; use data-testid attribute
CSS selector	locator('.btn-primary')	Last resort — fragile, breaks on markup changes, avoid

// BAD
page.locator('#root > div > form > button:nth-child(2)')
// Fragile CSS path — breaks on any DOM restructure

// GOOD
page.getByRole('button', { name: 'Submit' })
// Semantic, resilient, matches accessibility tree

Waiting Strategy

Never use hardcoded sleeps. Always wait for an observable UI state.

// BAD
await page.click('#submit');
await page.waitForTimeout(2000);  // never do this — hides real timing issues

// GOOD
await page.click('#submit');
await expect(page.getByText('Payment confirmed')).toBeVisible();
// or wait for navigation:
await page.waitForURL('/confirmation');

API E2E Tests

Test complete API workflows over the network — not just service-level unit behavior. This verifies the full auth lifecycle, serialization, and routing.

Key patterns:

• Obtain an auth token, use it in subsequent requests, refresh before expiry
• Use Playwright's request fixture for co-located API and browser tests
• Assert on response status, body shape, and downstream side effects

test('create and retrieve payment', async ({ request }) => {
  // authenticate
  const authRes = await request.post('/api/auth/token', {
    data: { email: '[email protected]', password: 'password' }
  });
  const { access_token } = await authRes.json();

  // create resource
  const createRes = await request.post('/api/payments', {
    headers: { Authorization: `Bearer ${access_token}` },
    data: { amount: 100, currency: 'USD' }
  });
  expect(createRes.ok()).toBeTruthy();
  const { id } = await createRes.json();

  // retrieve and verify
  const getRes = await request.get(`/api/payments/${id}`, {
    headers: { Authorization: `Bearer ${access_token}` }
  });
  const payment = await getRes.json();
  expect(payment.amount).toBe(100);
});

BDD with Gherkin

When to Use BDD

Use BDD when:

• A product owner, QA, and developer need shared, readable test documentation
• Business rules are complex and non-engineers need to verify coverage

Do not use BDD when:

• The team is small and tickets already capture intent clearly
• The overhead of step definitions outweighs the communication benefit

Feature File Structure

Feature: User Authentication
  As a registered user
  I want to log in with my credentials
  So that I can access my account

  Background:
    Given a user exists with email "[email protected]"

  Scenario: Successful login
    When I submit valid credentials for "[email protected]"
    Then I should be redirected to the dashboard
    And I should see a welcome message

  Scenario: Failed login - wrong password
    When I submit the wrong password for "[email protected]"
    Then I should see "Invalid credentials"
    And I should remain on the login page

  Scenario Outline: Login with various invalid inputs
    When I submit email "<email>" and password "<password>"
    Then I should see error "<error>"

    Examples:
      | email            | password | error                    |
      | invalid-email    | pass123  | Invalid email format     |
      |                  | pass123  | Email is required        |
      | [email protected] |          | Password is required     |

BDD Tooling

Language	Tool
Node.js	@cucumber/cucumber
Python	behave
Go	godog
Java	Cucumber-JVM

Use tags to filter test runs: @smoke, @regression, @wip.

# Run only smoke-tagged scenarios
npx cucumber-js --tags @smoke

# Skip work-in-progress scenarios
npx cucumber-js --tags "not @wip"

Smoke Tests

Smoke tests answer one question: "Is the deployed system alive?" They are not comprehensive — they verify only the critical path. If a smoke test fails, the deployment must be rolled back or halted immediately.

Run smoke tests automatically after every deployment to staging and production.

Criteria for inclusion: if this breaks, the system is unusable for most users.

test.describe('Smoke', () => {
  test('health endpoint returns 200', async ({ request }) => {
    const res = await request.get('/health');
    expect(res.status()).toBe(200);
  });

  test('home page loads', async ({ page }) => {
    await page.goto('/');
    await expect(page.getByRole('heading', { level: 1 })).toBeVisible();
  });

  test('user can log in', async ({ page }) => {
    const loginPage = new LoginPage(page);
    await loginPage.goto();
    await loginPage.login(process.env.SMOKE_USER!, process.env.SMOKE_PASSWORD!);
    await expect(page).toHaveURL('/dashboard');
  });
});

Run with:

npx playwright test --grep @smoke

Tag smoke tests with @smoke in Playwright using test.describe metadata or a custom tag fixture so they can be selected independently from the full suite.

Flakiness Prevention

Root Causes and Fixes

Cause	Fix
Hardcoded waitForTimeout	Replace with observable state assertions (toBeVisible, etc.)
Shared test data across parallel tests	Use unique IDs per test run (e.g., Date.now() suffix)
Tests depend on execution order	Each test must set up its own state in beforeEach
Timezone or locale sensitivity	Fix locale in test environment config
Race conditions in UI during animation	Use toBeVisible() / toBeEnabled() — not isVisible()
Network variability in CI	Increase timeouts in CI config, not with waitForTimeout

Quarantine Pattern

When a test is flaky and cannot be fixed immediately, quarantine it rather than deleting it. Deletion loses coverage history; quarantine preserves intent and tracks remediation.

test.fixme('payment flow — FLAKY: race condition in payment widget', async ({ page }) => {
  // tracked in: https://github.com/org/repo/issues/123
  // do not delete — re-enable once widget stabilised
});

test.fixme skips the test and marks it as expected to fail. Remove the .fixme once the underlying issue is resolved.

Test Data Management

Never use production accounts or shared test users in E2E tests. Shared state causes interference between parallel runs and makes failures non-deterministic.

API-Driven Setup and Teardown

let testUser: { id: string; email: string };

test.beforeEach(async ({ request }) => {
  // create an isolated test user for this test run
  const res = await request.post('/api/test/users', {
    data: { email: `test-${Date.now()}@example.com` }
  });
  testUser = await res.json();
});

test.afterEach(async ({ request }) => {
  // clean up — do not leave test data in the database
  await request.delete(`/api/test/users/${testUser.id}`);
});

Rules for Test Data

• Test data helper endpoints (/api/test/*) must only be available in test and staging environments
• Gate them with a NODE_ENV check in the server — never expose in production
• Prefer creating data via API over direct DB mutations for portability
• Do not rely on seed data that may change — generate data at test time

// server-side guard (Express example)
if (process.env.NODE_ENV !== 'test' && process.env.NODE_ENV !== 'staging') {
  throw new Error('Test helpers only available in test/staging environments');
}

CI Integration

Recommended CI Configuration

# .github/workflows/e2e.yml
name: E2E Tests
on: [push, pull_request]

jobs:
  e2e:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-node@v4
        with:
          node-version: '20'
      - run: npm ci
      - run: npx playwright install --with-deps
      - run: npx playwright test
        env:
          CI: true
          BASE_URL: http://localhost:3000
      - uses: actions/upload-artifact@v4
        if: failure()
        with:
          name: playwright-report
          path: playwright-report/

Key CI settings:

• Set CI=true so Playwright applies retries: 2 from config
• Upload playwright-report/ as artifact on failure for post-mortem debugging
• Run smoke tests as a separate faster job on staging deploy; run full suite on PRs

Red Flags

• Locators by CSS class or generated attribute — class names change during refactoring; use getByRole, getByLabel, or data-testid attributes that survive UI changes
• waitForTimeout as an explicit sleep — arbitrary sleeps make tests slow and flaky; always wait on observable state (waitForSelector, expect(locator).toBeVisible())
• One long E2E test that covers the entire user flow — a 200-step test is slow, provides poor failure diagnosis, and fails for unrelated reasons; split into focused user-journey tests
• E2E tests run against a shared staging environment — tests that create or delete shared state break other developers' work; use isolated per-run environments or UUID-suffixed test data
• Hardcoded test user credentials — parallel CI runs create conflicts; generate unique test users per run or use an isolated test account per CI job
• No smoke test post-deployment — a full E2E suite takes too long to run immediately after deploy; define a 2-minute smoke test of critical paths that runs on every deployment
• Quarantining flaky tests indefinitely — flaky tests erode trust in the suite and mask real failures; quarantine with test.fixme and a tracking issue, fix within the same sprint

Checklist

• E2E tests cover only critical user journeys (login, core workflows, checkout)
• Page Object Model used — no raw locators in test files
• Locators use getByRole / getByLabel — no fragile CSS selectors
• No waitForTimeout — all waits are based on observable state
• Tests are fully isolated — no shared mutable state between tests
• Smoke tests defined and run automatically after every deployment
• Flaky tests are quarantined with test.fixme and a tracking issue, not deleted
• Test data created via API in beforeEach and cleaned up in afterEach
• CI retries E2E tests 2x before failing (retries: 2 in CI config)
• Screenshots and video captured on failure for debugging (playwright.config.ts)
• Test data endpoints are gated and unavailable in production
• BDD feature files reviewed by a non-engineer to confirm readability