Robots.txt & llms.txt Generator
Build a correct robots.txt in a minute: keep search engines crawling, block admin and cart pages, decide which AI crawlers may visit, and list your sitemap. Then create an llms.txt file that points AI tools to your best pages.
- 100% free
- No sign-up
- AI crawler presets
- WordPress ready
Your robots.txt
Upload the file to the root of your site so it opens at yoursite.com/robots.txt. In WordPress, Rank Math and Yoast can edit it for you (Rank Math: General Settings → Edit robots.txt).
Your llms.txt
llms.txt is a proposed standard (llmstxt.org) for giving AI tools a short, clean guide to your site. It sits at yoursite.com/llms.txt. No major AI search engine has confirmed it uses the file, so treat it as a low-effort extra, not a ranking factor.
- Block unwanted pagesAdmin, cart, search and other pages kept out of crawling.
- Control AI crawlersAllow or block GPTBot, ClaudeBot, PerplexityBot and more.
- Add your sitemapPoint search engines and AI tools to your sitemap.
- Create llms.txtA short guide that points AI tools to your best pages.
How to create your robots.txt
- Keep "Allow the whole site" unless this is a staging or private site.
- Tick the common rules you need, such as the WordPress admin area or WooCommerce cart and checkout pages, and add your own paths.
- Choose a preset for AI crawlers, or set each one to allow or block.
- Add your sitemap URL, test a few addresses in the tester, then copy or download the file.
Upload robots.txt to the root of your site, or paste it into your SEO plugin's robots.txt editor. Check the live file afterwards with the robots.txt and AI crawler checker.
robots.txt
User-agent: * Disallow: /wp-admin/ Allow: /wp-admin/admin-ajax.php Sitemap: https://example.com/sitemap.xml
Choosing which AI crawlers to allow
- Allow all: best for visibility. AI assistants can find, quote and link to your pages.
- Block AI training only: keeps AI search and user-request crawlers, so you still appear in AI answers, while asking training crawlers such as GPTBot, ClaudeBot, Google-Extended and CCBot to stay away.
- Block all AI crawlers: keeps search engines only. Your pages will not appear in AI answers from these companies.
robots.txt mistakes to avoid
- Do not block CSS, JavaScript or
/wp-content/: Google needs them to render your pages. - Do not use robots.txt to hide pages from search results; use a noindex tag instead.
- Do not leave "Disallow: /" from a staging site on the live site.
- Google ignores
NoindexandCrawl-delaylines in robots.txt.
Frequently asked questions
Where do I put robots.txt?
In the root folder of your site, so it opens at yourdomain.com/robots.txt. In WordPress, Rank Math and Yoast let you edit it from the dashboard without uploading a file.
Will blocking AI training crawlers hurt my SEO?
No. Training crawlers such as GPTBot, ClaudeBot, Google-Extended and CCBot are separate from search engine crawlers, so blocking them does not change your Google or Bing rankings.
Why does a crawler with its own group ignore my other rules?
A crawler follows only the most specific group that names it. When you block or allow a crawler separately, the generator writes its full rules into its own group so nothing is lost.
Do AI tools read llms.txt?
llms.txt is a proposal, and no major AI search engine has confirmed it uses the file. It is quick to create and does no harm, so treat it as an optional extra.
Is anything I enter saved?
No. Both generators run entirely in your browser; nothing is sent to Today SEO World.







