Measurement, control 8 of 17
M-08 Run controlled, versioned prompt benchmarks
AI response measurement must preserve prompts, instruments and context well enough for meaningful comparison. Repeated runs are not automatically independent samples.
- Priority
- High priority.
- Hard gate
- No
- Change class
- READ_ONLY: Inspect authorised evidence.
- Quality bucket
- Measurement and experiment readiness
Why it matters
If the prompts change from run to run, or if one type of question dominates, the results cannot be compared and the trend means nothing. Generative answers also vary by engine, model, date and region, so results without that metadata cannot be reproduced. A fixed, balanced panel makes changes in visibility something you can actually trust.
Evidence to review
- Run manifest, prompt version, raw responses and failure records.
How to implement
- Record engine, interface, model when disclosed, country, language, account state, memory and search mode.
- Keep a stable panel and fixed retry policy.
- Retain unsuccessful runs and raw evidence.
- Version prompt changes and bridge old/new instruments rather than splicing incompatible trends.
How UltimateSEO checks this
Automated. A check in the scan sets PASS or FAIL for this control, and reports UNKNOWN or BLOCKED when its input is missing.
States used: PASS, FAIL, UNKNOWN, NOT_APPLICABLE or BLOCKED. UNKNOWN is never shown as a pass. How states work.
Myth guards that reference this control
Sources
Part of The UltimateSEO Framework v4.3, catalogue dated 2026-10-14. Evidence states, control states and the publish gate are explained on the hub.