Robots.txt Generator

Build a valid robots.txt with a live preview.

Compose a robots.txt with per-bot Allow/Disallow rules, crawl-delay, host and sitemap directives. Quick presets for WordPress, blocking AI bots and staging. Runs fully in your browser.

How to use Robots.txt Generator

  1. Choose A Policy

    Start the free online robots.txt generator by setting a default policy to allow everything, block everything, or build custom rules for your website.

  2. Apply A Preset

    Speed things up with a one-click preset for WordPress, blocking all crawlers, blocking AI bots like GPTBot and ClaudeBot, or locking down a staging site.

  3. Add Bot Groups

    Create User-agent groups for specific crawlers such as Googlebot or Bingbot, then add Allow and Disallow rules with wildcard and end-anchor path patterns.

  4. Add Sitemaps And Extras

    Paste your absolute Sitemap URLs so crawlers can find every page, and optionally set a crawl-delay and preferred host directive for finer control.

  5. Preview Live

    Watch the valid robots.txt build instantly in the live preview pane as you edit, so you can confirm the exact syntax before publishing it to your site.

  6. Copy Or Download

    Copy the generated robots.txt or download the file, then upload it to your domain root at example.com/robots.txt to guide search engine crawlers.

Frequently asked questions

What is a robots.txt generator and why do I need one?

A robots.txt generator builds the robots.txt file that lives at your site root and tells search-engine crawlers which paths they may or may not request. This free online robots.txt generator creates a valid file with a live preview, so you control crawling without hand-writing the syntax.

How do I create a robots.txt file for my website?

Pick a default policy (allow all, block all, or custom), add per-bot User-agent groups with Allow and Disallow rules, optionally add a crawl-delay, host and Sitemap URL, then copy or download the file. Save it as robots.txt at the root of your domain, for example https://example.com/robots.txt.

How do I block all crawlers or just AI bots in robots.txt?

Use the Block all crawlers preset to add User-agent: * with Disallow: / for the whole site, or use the Block AI bots preset to disallow GPTBot, ClaudeBot, CCBot, Google-Extended, PerplexityBot and other AI crawlers while leaving normal search engines allowed.

What is the difference between Allow and Disallow in robots.txt?

Disallow tells a crawler not to request a path, while Allow re-permits a sub-path inside a disallowed area. An empty Disallow: line allows everything, and Disallow: / blocks the entire site for that user-agent.

How do I add a sitemap to my robots.txt file?

Add a Sitemap line with the full absolute URL of your sitemap, such as Sitemap: https://example.com/sitemap.xml. In this generator just click Add sitemap and paste the URL; you can list multiple sitemaps so crawlers can discover all of your pages.

Does robots.txt stop a page from being indexed by Google?

No. robots.txt is advisory and only controls crawling, not indexing — a disallowed URL can still appear in search results if other pages link to it. To keep a page out of the index use a noindex meta tag or HTTP header, and use authentication for true privacy.

Do robots.txt paths support wildcards like * and $?

Yes. Most major crawlers support the * wildcard for any sequence of characters and the $ end-of-URL anchor, so a rule like Disallow: /*.pdf$ blocks every PDF. This generator lets you enter such patterns directly in the path field.

Where should the robots.txt file be placed?

Exactly at the root of your domain — https://example.com/robots.txt. It won't work in a subfolder, and each subdomain (like blog.example.com) needs its own robots.txt at its own root. Generate the file here and upload it to your site's root directory.

What is Crawl-delay in robots.txt?

Crawl-delay asks a crawler to wait a set number of seconds between requests to reduce server load. Google ignores it (control crawl rate in Search Console instead), but Bing, Yandex and some others honour it. Add it here only if a specific bot is hitting your server too hard.

Related tools — SEO · Robots.txt tools