File presence
Whether robots.txt exists in the root, what status code it returns, and whether the address is correct.
FREE TOOL · GEO
AI crawler access check
See which AI systems can read your site under the rules in robots.txt.
ROBOTS.TXT
Enter a domain
9 crawlers
-of 9 allowed
Checking
Loading the site's rules.
READY RULE
Allow AI search crawlers
Training bots are not included. Configure their access separately.
robots.txt
User-agent: OAI-SearchBot Allow: / User-agent: Claude-SearchBot Allow: / User-agent: PerplexityBot Allow: /
What the tool checks
Access is read from the actual groups and directives in robots.txt.
Shared rules
Restrictions inherited from User-agent: * when a bot has no rule of its own.
Named bots
Rules for OpenAI, Anthropic, and Perplexity crawlers, plus the Google-Extended control token.
Access type
AI search, user-initiated fetch, and model training are reported separately.
READING THE RESULT
What to allow depends on what you want
AI search
Allow search crawlers if you want to appear in answers with a link back.
User-initiated fetch
These agents open a page when a person explicitly asks an AI system to read it.
Training
Training bots can be restricted without affecting AI search.
QUESTIONS ABOUT THE TOOL
FAQ
If there is no robots.txt, are AI bots blocked+
No. A missing file means no restrictions. Bots read the site under their own rules.
Can I allow AI search and block training+
Yes. They use different user agents, so the rules are set separately.
Does the check account for Cloudflare and WAF+
No. The tool reads robots.txt. A firewall or bot-protection layer can block a crawler that robots.txt allows, and that shows up in server logs, not in this file.
What does partial access mean+
Some of a vendor's bots are allowed and others are blocked. A common case is AI search open and training closed.
Materials and next steps
How AI bots read robots.txt
Keep checking
If access is open but your pages still do not show up in AI answers, look at GEO services.
GEO services ↗Copied