Technical eligibility, control 11 of 20
T-11 Verify crawler access through CDN and WAF
An allow rule in robots.txt does not demonstrate access through edge security. A user-agent string alone is not trustworthy bot identity.
- Priority
- Gate. A FAIL stops publish or launch.
- Hard gate
- Yes. A FAIL stops publication, whatever the score.
- Change class
- APPROVAL_REQUIRED: Required exact authority must be valid before the consequential action.
- Quality bucket
- Technical eligibility
Why it matters
An open robots.txt means little if the WAF or CDN quietly turns crawlers away with errors or challenges. Search and AI crawlers then cannot read the pages, so the site drops out of results and answers without any visible warning. Robots.txt alone cannot show edge behaviour, which is why logs have to confirm that the OpenAI, Anthropic and Perplexity crawlers are getting through.
Evidence to review
- Verified bot logs, edge decisions, path-scoped policy and a before/after access test.
How to implement
- Inspect 401, 403, 429, challenges, geo blocks and origin failures by verified request class.
- Use current official verification methods where provided.
- Scope any exception to approved public paths; never bypass application authentication or all security checks.
- Label synthetic user-agent tests as simulations, not genuine bot observations.
How UltimateSEO checks this
Partly automated. The scan records page signals, but a passing signal is kept as information: contextual or account evidence is still needed to establish the control.
The check looks at: the crawler fetches with a Googlebot-like user agent and the default user agent and compares status codes; server logs or WAF rules from the customer are needed to confirm no real crawler is blocked.
States used: PASS, FAIL, UNKNOWN, NOT_APPLICABLE or BLOCKED. UNKNOWN is never shown as a pass. How states work.
Sources
Part of The UltimateSEO Framework v4.3, catalogue dated 2026-10-14. Evidence states, control states and the publish gate are explained on the hub.