AI prototype user testing: a yes in the meeting does not validate your product
AI prototype user testing: a yes in the meeting does not validate your product
In the meeting, the founder clicks through the screens, explains the labels and smooths over every hesitation. Everyone likes the prototype. Then someone outside the project tries it alone and cannot work out what happens after submitting a request. The team approved a guided demonstration, not an independently usable journey.
User testing an AI-generated prototype means observing a representative person attempt a realistic task without showing them the route. It helps identify barriers before you fund more development. On its own, it does not establish willingness to pay, market demand or the technical reliability of the eventual product.
What do you need to learn before building more?
“Do you like it?” readily produces compliments. “Find yesterday’s quote and request a change” puts the journey to a test. The second instruction gives you something observable and makes failure visible even when the interface looks attractive.
The GOV.UK Service Manual describes moderated usability testing as watching participants attempt specific tasks. It recommends believable, relevant tasks that do not give away the answer. That methodological principle also applies to a mockup generated with Lovable, Bolt or v0; it is not a certification of those products.
Choose a decision before the session: should the team simplify request submission, clarify the status message or revisit the opening promise? Testing the whole product at once makes that decision harder. Your AI prototype PRD holds the wider requirements; this test should resolve a specific uncertainty.
Who should participate, and what should they attempt?
Someone who already knows the project no longer sees it as a first-time user. Recruit actual or likely users whose circumstances reflect your audience: professional vocabulary, device, digital confidence and any assistance needs. GOV.UK’s guidance on learning about user needs treats opinions from other stakeholders as assumptions to investigate.
Illustrative scenario: a small business wants to test a quotation portal. The participant gets a situation: “The quoted amount no longer matches what you need. Request a revised estimate and check that your request has been received.” Avoid “Click Edit, then Confirm.” That instruction supplies the route you are supposed to be testing.
Use coherent dummy data when confidentiality or prototype limitations prevent safe use of real information. Make the simulation explicit. A confirmation screen does not prove that an email was sent, a quotation was saved or a business system was updated.
What should you record when someone gets stuck?
Facilitators naturally want to help. But explaining every screen turns the session into a test of the facilitator’s teaching. Give participants room to explore, ask what they understand and record any assistance. An intervention may be necessary to keep learning; it must not vanish from the findings.
- Observed outcome: was the task completed independently, completed with help or abandoned?
- Breakdown point: record the screen, label or missing information behind the hesitation, rather than a vague judgement such as “needs to be smoother”.
- Understanding: ask what the person believes they submitted and what they expect next. Reaching the final button is not enough if they think a quote was accepted when it was only received.
- Test conditions: retain the prototype version, scenario, participant profile and simulation limits. Without them, findings from a later session are difficult to compare.
Time on task can supplement these observations if tasks and conditions remain comparable. A handful of qualitative sessions does not justify a success-rate claim for the whole market.
How do observations become a product decision?
If a participant cannot find the action, investigate the navigation or wording first. If they understand it but see no reason to return, investigate the underlying need and value. If they expect a confirmation the prototype can only simulate, investigate integration. Those findings call for different investments.
In the quotation example, a submitted request with no understandable status first calls for an interface correction. Rebuilding the entire product would not necessarily solve it. Equally, a clearly understood button does not validate how the sales system processes the request.
Record each decision with its evidence: observation, proposed change, owner and verification criterion. For example: “After submission, the participant can explain that their request is awaiting a response, without help from the facilitator.” Test the revised version rather than treating the new mockup as approved by default.
When is it time to move from prototype to V1?
Keep three kinds of evidence separate: usability — people understand and can act; value — they make a meaningful commitment, tested beyond this session; and execution — data and actions are actually processed. A journey test mainly informs the first. Willingness to pay needs appropriate commercial research or an experiment. Reliability needs technical tests.
A business MVP can then test a hypothesis through a limited real operation. Within the Say Digital Framework, prototype findings become development acceptance criteria. Tests and CI/CD — automated checks before delivery — protect those criteria, followed by a controlled rollout and measurement of actual use.
Two questions before arranging sessions
Is there one correct number of participants?
No. Recruitment depends on the audiences, journeys and decision involved. An initial round can reveal barriers; it does not automatically cover every user group, accessibility need or commercial question. Plan further sessions after making changes.
Must an AI prototype already execute everything?
No, when the question is whether people understand the journey. Its limits must still be disclosed and recorded. Verifying a payment, notification or permission requires additional execution evidence, not just a success screen.
Your next product decision
Before funding more screens, choose the task that puts your promise to a test. Say Digital can scope the research, prepare the prototype and translate observations into V1 decisions: what changes, what waits and what needs technical proof.
Clarify my AI prototype with Say Digital
Methodological sources
- GOV.UK — Using moderated usability testing: realistic tasks, observation, facilitation and the limits of dummy data.
- GOV.UK — Learning about users and their needs: user research and the distinction between needs and assumptions.
Sources checked on 28 October 2026. These are established methods, not a new study of AI performance. The quotation portal is an illustrative example, not a customer result.
