Testing strategy

Accessibility Testing Strategy

"Automated testing with axe-core identifies 57% of digital accessibility issues by volume." — Deque, Automated Testing Study (2023)

"Automated scans reliably flag only 13% of WCAG 2.1 AA success criteria." — Accessible.org

Automated testing is necessary but insufficient. A complete accessibility testing strategy uses three tiers, each catching different categories of issues.

---

1. Three Testing Tiers

Tier 1: Automated Testing (~57% of issues by volume)

Tools: axe-core, Lighthouse, WAVE, HTML validators When: Every build / CI pipeline

Catches well:

• Color contrast ratios below thresholds (WCAG 1.4.3)
• Missing alt attributes, form labels, page lang (Deque, 2023)
• Invalid ARIA: duplicate IDs, broken references, invalid roles
• Structural issues: missing bypass blocks, empty/skipped headings
• Missing page title, missing language declaration

Key stat: Seven WCAG criteria account for over 80% of all issues found, with contrast alone at ~30%. (Deque, 2023)

Tier 2: Keyboard Testing

Tools: None required beyond a keyboard When: Every feature / PR

Catches issues automation misses:

• Focus visibility and indicator quality (WCAG 2.4.7)
• Tab order matching visual/reading order (WCAG 2.4.3)
• Keyboard traps (WCAG 2.1.2)
• Interactive element operability (WCAG 2.1.1)
• Skip links and modal focus management

See: ${CLAUDE_SKILL_DIR}/references/keyboard-testing-checklist.md

Tier 3: Screen Reader Testing

Tools: NVDA + Chrome (Windows), VoiceOver + Safari (macOS/iOS), TalkBack + Chrome (Android) When: New components, significant UI changes, pre-release

Catches issues that require hearing what AT communicates:

• Reading order and meaningful sequence (WCAG 1.3.2)
• Live region announcements (WCAG 4.1.3)
• Name, role, value correctness (WCAG 4.1.2)
• Info and relationships communication (WCAG 1.3.1)

See: ${CLAUDE_SKILL_DIR}/references/screen-reader-commands.md

---

2. What Automated Testing Catches vs Misses

Coverage Breakdown (WCAG 2.1 AA, 55 criteria)

Detection Level	Criteria Count	Percentage
Mostly accurate automated detection	7 of 55	13%
Partially detectable (flags, needs human review)	25 of 55	45%
Cannot be detected by automation	23 of 55	42%

Source: Accessible.org

What Automation Cannot Catch

Category	Why Automation Fails
Alt text quality	Detects missing alt but not whether text is meaningful
Reading order	DOM vs visual order requires human judgment
Keyboard operability	Dynamic interaction sequences can't be fully automated
Focus management	Context-dependent behavior in SPAs, modals
Meaningful labels	"Click here" is technically present but not descriptive
Error message quality	Detects presence but not helpfulness
Cognitive load	Information organization, plain language
Live region behavior	Timing and appropriateness of announcements

Source: Deque, Manual vs Automated Testing

---

3. Keyboard Testing Summary

Disconnect the mouse. Start from the browser address bar.

Check	What to Verify
Tab order	Tab moves through all interactive elements in logical order
Focus visibility	Every focused element has a visible indicator (3:1 contrast min)
Operability	Links: Enter. Buttons: Enter/Space. Checkboxes: Space. Radios: Arrows
No traps	Focus never gets stuck; Tab/Shift+Tab/Escape always works
Modals	Focus enters modal, cycles within it, Escape closes, focus returns
Skip links	First focusable element, moves focus past navigation

Standard keys:

Key	Expected Action
Tab / Shift+Tab	Move forward/backward through interactive elements
Enter	Activate link or button
Space	Activate button, toggle checkbox, open select
Arrow keys	Navigate within widget (radio group, menu, tabs, listbox)
Escape	Close modal, menu, or dialog
Home / End	Jump to first/last item in list or menu

Sources: WebAIM Keyboard Accessibility, UW Keyboard Checklist, W3C Easy Checks

Full checklist: ${CLAUDE_SKILL_DIR}/references/keyboard-testing-checklist.md

---

4. Screen Reader Testing Basics

Which SR + Browser Pairings to Test

Test in priority order based on market share (WebAIM Survey #10, 2024):

Priority	Combination	Coverage	Cost
1	NVDA + Chrome (Windows)	21.3% of users	Free
2	VoiceOver + Safari (macOS)	7.0% + all iOS	Built-in
3	JAWS + Chrome (Windows)	24.7% of users	License required
4	TalkBack + Chrome (Android)	27.6% of mobile	Built-in

Desktop Market Share

Screen Reader	Share
JAWS	40.5%
NVDA	37.7%
VoiceOver	9.7%

What to Listen For

When testing with a screen reader, verify:

1. Element identification — role announced correctly ("button", "link", "heading level 2")
2. Accessible name — label is meaningful and matches visible text (WCAG 2.5.3)
3. State changes — expanded/collapsed, checked/unchecked, selected announced on change
4. Reading order — content reads in meaningful sequence
5. Live regions — dynamic updates announced without requiring navigation
6. Heading structure — headings navigable (H key in NVDA), hierarchy logical

Full command reference: ${CLAUDE_SKILL_DIR}/references/screen-reader-commands.md

---

5. WCAG Criteria Requiring Human Judgment

These criteria cannot be verified by automation — they require understanding of meaning, context, and quality.

Criterion	Why Human Judgment Needed
1.1.1 Non-text Content (quality)	Can detect missing alt but not accuracy
1.2.1–1.2.5 Media alternatives	Cannot verify transcript/caption accuracy
1.3.2 Meaningful Sequence	Cannot determine if reading order preserves meaning
1.3.3 Sensory Characteristics	Cannot detect reliance on shape/color/location alone
2.4.5 Multiple Ways	Cannot assess navigation method availability across site
2.4.6 Headings and Labels (quality)	Detects headings, not whether they are descriptive
3.1.2 Language of Parts	Cannot determine where language changes occur
3.2.3 Consistent Navigation	Cannot compare consistency across pages
3.2.4 Consistent Identification	Cannot verify consistent naming across pages
3.3.3 Error Suggestion	Cannot evaluate if error messages are helpful
3.3.4 Error Prevention	Cannot verify confirmation/review for critical transactions

Partially Detectable (flagged but need human confirmation)

Criterion	Automation Flags	Humans Must Verify
1.3.1 Info and Relationships	Missing semantic markup	Whether markup conveys right relationships
1.4.5 Images of Text	Presence of images	Whether images contain text that should be HTML
2.4.4 Link Purpose	Link text exists	Whether text is meaningful in context
4.1.2 Name, Role, Value	ARIA attributes present	Whether names are accurate and useful

Source: Accessible.org

---

6. Writing Acceptance Criteria

Per-Story Template

Add to each user story:

Acceptance Criteria:
- [ ] All interactive elements reachable and operable by keyboard
- [ ] Focus order matches visual/logical order
- [ ] Screen reader announces [element] as [expected announcement]
- [ ] Color contrast meets WCAG AA (4.5:1 text, 3:1 large text)
- [ ] Error messages programmatically associated with fields
- [ ] State changes announced to assistive technology

Component-Level Template

Component: [Name]
ARIA Pattern: [APG reference]

Keyboard:
- [ ] [Key]: [Expected behavior]

Screen Reader:
- [ ] Role announced as: [expected role]
- [ ] Name announced as: [expected name]
- [ ] State changes announced: [list states]

Visual:
- [ ] Focus indicator visible (3:1 contrast)
- [ ] Color contrast meets AA
- [ ] Works at 200% zoom

Definition of Done Checklist

1. Automated scan passes (axe-core, zero violations)
2. Keyboard-only operation verified (full workflow)
3. Focus management correct (modals, route changes, dynamic content)
4. Screen reader announces all content meaningfully (NVDA or VoiceOver)
5. Color contrast meets AA thresholds
6. Content reflows at 200% zoom
7. Error states announced and associated with fields
8. Heading hierarchy logical (no skipped levels)

Full templates: ${CLAUDE_SKILL_DIR}/references/acceptance-criteria-templates.md

Sources: TetraLogical, Section508.gov, TPGi

---

7. Common Mistakes

See: ${CLAUDE_SKILL_DIR}/references/common-mistakes.md

The most frequent testing anti-patterns:

1. "Axe passes, so we're accessible" — Automation catches at most 57% of issues by volume, and only 13% of WCAG criteria with high confidence. (Deque, Accessible.org)

2. Testing only with one screen reader — JAWS, NVDA, and VoiceOver behave differently. A component working in one may fail in another. Test with at least two. (WebAIM Survey #10)

3. Skipping keyboard testing — The fastest, cheapest tier. No tools needed. Catches focus traps, missing operability, and broken tab order that automation misses entirely.

4. Testing at the end — Accessibility debt compounds. Test each tier as part of the PR workflow, not as a pre-release gate.

5. Writing vague acceptance criteria — "Must be accessible" is not testable. Specify what the screen reader should announce, what keyboard interaction is expected, and what contrast ratio applies.

6. Ignoring mobile — 91.3% of screen reader users use one on mobile. VoiceOver (iOS) and TalkBack (Android) have different interaction models than desktop. (WebAIM Survey #10)

7. Not testing with real assistive technology — Browser DevTools accessibility inspectors show the accessibility tree but not what users actually hear. Run NVDA or VoiceOver and listen.

---

8. Cross-References

• a11y-audit — Full audit methodology and WCAG conformance checking
• aria-decision-framework — When and how to use ARIA correctly
• focus-management — Focus trapping, restoration, and roving tabindex patterns
• form-a11y — Form labeling, error handling, and validation patterns
• live-regions — aria-live region patterns and announcement timing