Free SEO Tool

Robots.txt & llms.txt Generator

Build a correct robots.txt in a minute: keep search engines crawling, block admin and cart pages, decide which AI crawlers may visit, and list your sitemap. Then create an llms.txt file that points AI tools to your best pages.

  • 100% free
  • No sign-up
  • AI crawler presets
  • WordPress ready
Default for all crawlers
Common rules

One per line. Use * for any text and $ for the end of the URL.

Search engines and AI crawlers

Full URLs, one per line. Rank Math and Yoast use /sitemap_index.xml; WordPress on its own uses /wp-sitemap.xml.

Your robots.txt

    Upload the file to the root of your site so it opens at yoursite.com/robots.txt. In WordPress, Rank Math and Yoast can edit it for you (Rank Math: General Settings → Edit robots.txt).

    • Block unwanted pagesAdmin, cart, search and other pages kept out of crawling.
    • Control AI crawlersAllow or block GPTBot, ClaudeBot, PerplexityBot and more.
    • Add your sitemapPoint search engines and AI tools to your sitemap.
    • Create llms.txtA short guide that points AI tools to your best pages.

    How to create your robots.txt

    1. Keep "Allow the whole site" unless this is a staging or private site.
    2. Tick the common rules you need, such as the WordPress admin area or WooCommerce cart and checkout pages, and add your own paths.
    3. Choose a preset for AI crawlers, or set each one to allow or block.
    4. Add your sitemap URL, test a few addresses in the tester, then copy or download the file.

    Upload robots.txt to the root of your site, or paste it into your SEO plugin's robots.txt editor. Check the live file afterwards with the robots.txt and AI crawler checker.

    robots.txt

    User-agent: *
    Disallow: /wp-admin/
    Allow: /wp-admin/admin-ajax.php
    Sitemap: https://example.com/sitemap.xml

    Choosing which AI crawlers to allow

    • Allow all: best for visibility. AI assistants can find, quote and link to your pages.
    • Block AI training only: keeps AI search and user-request crawlers, so you still appear in AI answers, while asking training crawlers such as GPTBot, ClaudeBot, Google-Extended and CCBot to stay away.
    • Block all AI crawlers: keeps search engines only. Your pages will not appear in AI answers from these companies.

    robots.txt mistakes to avoid

    • Do not block CSS, JavaScript or /wp-content/: Google needs them to render your pages.
    • Do not use robots.txt to hide pages from search results; use a noindex tag instead.
    • Do not leave "Disallow: /" from a staging site on the live site.
    • Google ignores Noindex and Crawl-delay lines in robots.txt.

    How to write a useful llms.txt

    Start with your site name and a one-sentence summary, then add sections such as Services, Products, Guides and Optional, each with links to your most useful pages and a short note on what each covers. Keep it short and factual. Make sure those pages are in good shape with the free SEO checker and have clear structured data from the schema markup generator.

    Frequently asked questions

    Where do I put robots.txt?

    In the root folder of your site, so it opens at yourdomain.com/robots.txt. In WordPress, Rank Math and Yoast let you edit it from the dashboard without uploading a file.

    Will blocking AI training crawlers hurt my SEO?

    No. Training crawlers such as GPTBot, ClaudeBot, Google-Extended and CCBot are separate from search engine crawlers, so blocking them does not change your Google or Bing rankings.

    Why does a crawler with its own group ignore my other rules?

    A crawler follows only the most specific group that names it. When you block or allow a crawler separately, the generator writes its full rules into its own group so nothing is lost.

    Do AI tools read llms.txt?

    llms.txt is a proposal, and no major AI search engine has confirmed it uses the file. It is quick to create and does no harm, so treat it as an optional extra.

    Is anything I enter saved?

    No. Both generators run entirely in your browser; nothing is sent to Today SEO World.