Robots.txt & AI Crawler Checker
Find out which search engines and AI crawlers can visit a website: Googlebot, Bingbot, GPTBot, ClaudeBot, PerplexityBot, Google-Extended and more. The checker reads the site's robots.txt and llms.txt, flags problems, and lets you test rule changes before you publish them.
- 100% free
- No sign-up
- 18 crawlers checked
- robots.txt and llms.txt
Reads the site's robots.txt and llms.txt and shows which search engines and AI crawlers may visit the homepage and the page you entered.
Results update as you type. Nothing is sent anywhere: the test runs in your browser.
- Check accessSee which crawlers may visit the homepage and the page you enter.
- robots.txt and llms.txtBoth files read and explained in one go.
- AI crawlersGPTBot, ClaudeBot, PerplexityBot, Google-Extended and more.
- Rule testerPaste new rules and test them before you publish.
How to use the robots.txt checker
- Enter a domain, or the address of a page you want to test.
- See which crawlers are allowed on the homepage and on that page, the exact rule that decides, and notes on lines Google ignores.
- Click "Test changes to this file" to edit the rules and see the effect at once, then create a clean file with the robots.txt generator.
Crawler access
- GooglebotAllowed
- BingbotAllowed
- GPTBotBlocked
- ClaudeBotAllowed
Search engines, AI search and AI training crawlers
The crawlers fall into four groups, and blocking each has a different effect:
- Search engines (Googlebot, Bingbot): blocking them removes your pages from search results. Almost every site should allow them.
- AI search (OAI-SearchBot, Claude-SearchBot, PerplexityBot and others): they let AI assistants show and link to your pages. Blocking them means fewer mentions and visits from AI answers.
- User request (ChatGPT-User, Claude-User, Perplexity-User): they open a page when someone asks an assistant about it.
- AI training (GPTBot, ClaudeBot, Google-Extended, CCBot and others): they collect content to train models. Blocking them does not affect Google Search.
How robots.txt rules are read
A crawler follows the group that names it; if none does, it follows the User-agent: * group, and ignores every other group. Inside a group, the longest matching rule wins, and Allow wins a tie. * matches any text and $ marks the end of the URL. The checker applies the rules the same way Google does.
robots.txt controls crawling, not indexing. To keep a page out of Google, allow crawling and add a noindex tag; a blocked page can still be indexed from links.
Frequently asked questions
Should I block AI crawlers?
It depends on your goals. Blocking AI training crawlers keeps your content out of future model training without affecting Google Search. Blocking AI search crawlers also removes you from AI answers and the visits they bring, which most businesses want to keep.
Does blocking Google-Extended affect Google rankings?
No. Google-Extended only controls the use of your content for Gemini and Google AI training. Googlebot, which crawls for Search, follows its own rules.
Do all crawlers obey robots.txt?
The big search engines and the major AI companies say their crawlers do. Some smaller or unknown bots ignore it; blocking those needs a firewall or server rule.
What happens if a site has no robots.txt?
Every crawler may crawl every page. That is fine for many sites. If robots.txt returns a server error instead, Google may pause crawling the whole site, so make sure it answers 200 or 404.
Is the rule tester private?
Yes. The "Test robots.txt rules" tab runs entirely in your browser. Only "Check a website" uses the Today SEO World server, to fetch that site's robots.txt and llms.txt.







