Technical eligibility, control 2 of 20
T-02 Separate crawl, indexing, preview and private-access controls
Robots rules govern supported crawling preferences; noindex governs indexing when processed; authentication governs private access. None is a substitute for the others.
- Priority
- Gate. A FAIL stops publish or launch.
- Hard gate
- Yes. A FAIL stops publication, whatever the score.
- Change class
- APPROVAL_REQUIRED: Required exact authority must be valid before the consequential action.
- Quality bucket
- Technical eligibility
Why it matters
Mixing up the two controls leads to pages that stay in search results or content that is exposed. A robots.txt block stops crawlers from seeing a noindex tag, and neither one keeps sensitive material private. Google's technical requirements keep the two jobs separate for this reason.
Evidence to review
- Live robots rules, actual response headers and directive parsing by route class.
- Confirm rules reach the response an engine can fetch.
How to implement
- Build a route-class policy before editing rules.
- Keep pages crawlable when relying on a noindex directive to remove them, unless a separately reviewed strategy applies.
- Test HTML and non-HTML X-Robots-Tag responses; never publish sensitive data because a path is disallowed.
How UltimateSEO checks this
Automated. A check in the scan sets PASS or FAIL for this control, and reports UNKNOWN or BLOCKED when its input is missing.
The check looks at: robots.txt rules against meta robots and X-Robots-Tag headers on crawled URLs, flagging pages that are both disallowed and noindexed or otherwise in conflict.
States used: PASS, FAIL, UNKNOWN, NOT_APPLICABLE or BLOCKED. UNKNOWN is never shown as a pass. How states work.
Sources
Part of The UltimateSEO Framework v4.3, catalogue dated 2026-10-14. Evidence states, control states and the publish gate are explained on the hub.