Rules for search crawlers
See whether robots.txt allows the submitted page for OAI-SearchBot, Googlebot, PerplexityBot and Claude-SearchBot.
Free AI access scan
Can AI crawlers
access your page?
Check your page’s crawler rules and server responses. See the results for each provider separately.
OpenAIGooglePerplexityClaude
An access check. Brand mentions and citations are measured separately.
Start with one page
Your website URL
Free · No account or email
Your AI access record
Access checks complete
robots.txt source and request details
Robots permission and server response are separate. Requests use named user-agents from our server; real crawler visits and AI-answer presence require separate evidence.
AI brand visibility
Where does your brand appear in AI answers?
This scan checks crawler access. Explore a sample report to see an answer, its cited source and what the findings could help you investigate.
What you can inspect
Understand your AI access controls.
Responses from your server
Compare HTTP responses to requests using each crawler’s user-agent, including refusals, redirects and rate limits.
Training and usage controls
Inspect GPTBot, ClaudeBot and Google-Extended separately. Training permission and search access serve different purposes.
How the access checks work
We fetch robots.txt from the submitted page’s origin and evaluate its rules for that exact page path. We also run six bounded HTTPS user-agent probes, which can follow redirects. Google-Extended is a robots control token, so it has no separate HTTP probe. The check stops after 25 seconds; an incomplete request remains unverified.
A robots rule describes a publisher’s instruction. An HTTP response describes how the server treated our request at that time. IP allowlists, firewall rules and other controls can treat the real crawler differently. A 429 response means recheck for rate limiting. A 200 response alone does not verify page content or AI-answer presence.
The robots file is fetched once with the scan’s own user-agent. Sites may vary this file by user-agent or IP. Rules apply to the submitted URL; when a page request redirects, its final URL is shown separately and its own robots permission is unverified. The scan does not check user-triggered assistants, log into a site or query an AI model.
No account, email, analytics or application storage is used. Our server and the destination website receive the request; standard hosting and network logs may apply. Use public page addresses only. Privacy details.
Crawler documentation: OpenAI · Google · Perplexity · Anthropic