Evaluating as user

Evaluating as a User

Methodology and judgement for evaluating a running product as a real user would: derive what to test from the PRD, structure the evaluation, assess quality dimensions, format findings. Covers what to evaluate and how to think about quality — browser mechanics are in playwright-cli (preloaded); agent role, inputs, status codes, and handoff are in the agent definition.

Applies to: web applications, CLI tools, APIs. Not covered: source code review (reviewing-code), automated test generation, accessibility audits (WCAG), performance benchmarking.

Workflow

Step 1: Derive user journeys from the PRD

Read the PRD. For each P0 requirement, derive one user journey using this structure:

Field	Description
Starting point	Where the user begins (URL, command, state)
Actions	Step-by-step sequence the user would take
Observable outcome	What confirms the requirement is satisfied

How to derive actions: Treat the requirement as a goal, not a task. Ask: what would a person with this goal actually do? Start from the entry point a new user would reach, not the most direct technical path.

Priority: P0 requirements first. Add P1 journeys only if time permits after all P0 journeys and universal checks are complete.

Non-web projects: For CLI tools, the starting point is the terminal and the actions are command sequences. For APIs, the starting point is an endpoint and the actions are the sequence of requests a user of the API would make. The journey structure is the same.

Step 2: Run PRD-derived journeys

Attempt each derived journey in sequence. As you go:

• Execute each action in the journey
• Record what you observe at each step in walkthrough.md (what you did, what you expected, what you saw)
• Take a screenshot at each meaningful state transition (web projects)
• Note any friction, confusion, missing feedback, or failure

When a journey succeeds, note which quality dimensions (Step 4) were satisfied or strained along the way. When a journey fails, record at what step and what the failure was.

If a severity-4 issue is encountered mid-journey: record it, then attempt to continue or skip to the next journey. Do not abandon the evaluation — continue with remaining journeys and universal checks, and highlight the catastrophic finding in the final report.

Example walkthrough entry:

Journey 1, Step 3 — Submit registration form
Did: Filled in name, email, and password fields; clicked "Create account"
Expected: Loading state on button, then redirect to dashboard or confirmation screen
Saw: Button remained active with no visual change for ~4 seconds, then page refreshed to the same form with no error message
Quality lens: Communicativeness failure (no feedback); Reliability feel strained (unpredictable outcome)

Step 3: Apply universal checks

After PRD journeys, run these checks on every product regardless of what the PRD specifies:

Check	What to do	What to look for
First landing	Load the primary screen fresh	Does the product communicate its purpose within 5 seconds? Is there a clear visual hierarchy? Does the user know what to do next?
First-time empty state	Access the product as a new user with no data	Are empty views explained and actionable, or blank and confusing?
Error paths	Submit invalid input, trigger validation, attempt impossible actions	Are errors in plain language with a clear fix suggestion? Do they appear where the user is looking?
Zero-results	Search or filter for something that will not exist	Is there a clear message that the search worked but found nothing? Is there an actionable next step?
Visual consistency	Navigate to three or more different views	Do colour, spacing, typography, and interaction patterns hold consistent throughout?
Feedback absence	Trigger any async action (form submission, data save, deletion)	Does the product visibly confirm that the action happened?

Non-web CLI tools: Replace first landing with "first run of the tool with no arguments or --help". Replace visual consistency with "consistency of output format, flag naming, and error message style". Skip screenshot-dependent checks.

APIs: Replace first landing with "calling the root endpoint or documentation endpoint". Replace visual checks with "consistency of response envelope structure, error format, and status code usage".

Step 4: Assess quality dimensions

Note: These dimensions are a lens you apply throughout Steps 2, 3, and 5 — not a separate sequential activity. Return here as a reference when assessing findings, not as a distinct pass after universal checks.

For each journey and each universal check, assess the product across these five dimensions. You do not need a score per check — use the dimensions as a lens to ensure findings cover the full experience, not just functional failures.

Dimension	The question	What failure looks like
Communicativeness	Does the product tell me what's happening?	Silent submissions, missing loading states, absent feedback, cryptic error codes
Learnability	Can I figure out how to use it without help?	Jargon, non-standard navigation patterns, no contextual guidance, forcing users to memorise
Reliability feel	Does it feel trustworthy and consistent?	Visual inconsistencies, unpredictable behaviour, poor error recovery, performance anxiety
Efficiency	Does it respect my time?	Unnecessary steps, buried primary actions, no defaults, repetitive data entry
Emotional tone	Does it feel crafted or careless?	Unstyled edge states, ignored empty states, missing microinteraction feedback, placeholder text in production

Treat feedback absence as severity 3 by default. If a user action produces no visible response, that is a major finding unless there is a clear reason it does not need one.

Step 5: Exploratory evaluation

After journeys and universal checks, use the product freely as a real person would. Navigate, go back, try things the PRD did not specify, use it at an unexpected pace. This layer surfaces issues no checklist captures.

Focus exploratory time on:

• Flows that felt hesitant during journey evaluation
• States you were unable to trigger intentionally
• Anything that prompted a moment of uncertainty

Record what you try and what you notice. Not all observations become findings — use judgement to decide what matters.

Step 6: Check design compliance (when system.md provided)

If a design system file was provided, compare the rendered product against it:

• Are the defined colour tokens applied correctly throughout?
• Does spacing follow the system's scale?
• Does typography use the specified families and weights?
• Are interactive states (hover, active, disabled, focus) handled consistently with the system's patterns?
• Does visual hierarchy match the design direction?

Report deviations as findings. Use severity 2 for cosmetic drift, severity 3 for systematic token violations or broken interactive states.

Step 7: Structure findings

Write each finding using this format:

**[SEVERITY-N]** Location (screen, view, component, or command)

_Finding:_ What the issue is, described as a user would experience it.

_Impact:_ Why it matters — what the user loses or cannot do.

_Suggestion:_ A specific, actionable fix the developer can implement.

Group findings in findings.md by severity, highest first. Include references to walkthrough steps and screenshots.

Severity Reference

Level	Meaning	Example
4 Catastrophic	Prevents task completion	Core feature does not work; user cannot achieve their goal
3 Major	Significant user impact — important to fix	Confusing flow leads users to a dead end; form submits silently; primary action is hidden
2 Minor	Causes friction but does not block	Missing loading indicator; inconsistent button labels; empty state with no guidance
1 Cosmetic	Fix if time permits	Slightly inconsistent spacing; minor colour drift from design system

Honesty rule: Do not inflate severity. A cosmetic issue is not catastrophic because it is frequent. A major issue is not cosmetic because it is hard to fix.

Finding Format: What Makes a Finding Actionable

A finding is actionable when a developer reading it knows exactly what to change. Test your findings against these criteria:

• Specific: Names the location and describes the behaviour, not a general observation
• Impact-clear: Explains what the user loses, not just that something is "wrong"
• Fixable: Suggests a concrete change, not "improve this" or "make it better"
• Evidence-linked: References a walkthrough step or screenshot

Vague finding (not actionable): "The onboarding experience is confusing."

Actionable finding: **[SEVERITY-3]** Onboarding — Step 2 (account setup) — After clicking "Create account", no confirmation or loading indicator appears for ~3 seconds. _Impact:_ Users assume the click did not register and click again, creating duplicate submissions. _Suggestion:_ Show a spinner or disabled state on the button immediately on click; display a "Creating your account..." message.

Common Traps

Evaluating the code, not the product. If you find yourself reasoning about implementation, stop. You are the user — you observe what happens, not why.

Judging intent, not experience. "The developer clearly intended X" is not relevant. "As a user, I encountered Y" is.

Over-inflating severity. Reserve 4 for genuine task blockers. Reserve 3 for issues that meaningfully harm the experience or cause user error. Not everything is major.

Skipping empty states because you filled them with test data. Explicitly test the zero-data view: create a new account, clear your test data, or access the product as a fresh user.

Only reporting failures. The walkthrough should document what works too — both as context for findings and as confirmation that the evaluation was thorough.

Dismissing visual inconsistency as cosmetic. Research shows visual quality is a proxy for product quality — users who see inconsistent spacing or mismatched colours assume the underlying product is unreliable too. Rate visual coherence issues at severity 2 minimum.