Robots.txt Tester

See if a URL is allowed for a given crawler.

Paste a robots.txt or fetch it live, then test any path against any user-agent to see exactly which rule allows or blocks it — matching real crawler precedence.

How to use Robots.txt Tester

  1. Add Your Robots.txt

    Paste your robots.txt rules into the free online robots.txt tester, or switch to Fetch by URL to pull the live file straight from your domain.

  2. Pick A Crawler

    Choose the user-agent to simulate, such as Googlebot, Bingbot, GPTBot or ClaudeBot, or enter a custom token to test how that specific bot is treated.

  3. Enter URLs To Test

    Type one path or full URL per line that you want to check against your robots.txt rules for the selected search engine or AI crawler.

  4. Run The Test

    Click Test and the tool instantly evaluates each path using Google's longest-match precedence to decide whether the URL is allowed or blocked.

  5. Read The Verdict

    See a clear Allowed or Blocked badge for every path alongside the exact matching Allow or Disallow rule and its line number for easy debugging.

  6. Fix And Retest

    Adjust your robots.txt directives to unblock important pages or tighten access, then re-run the test to confirm crawlers can reach the right URLs.

Frequently asked questions

What is a robots.txt tester and how does it work?

A robots.txt tester checks whether a given URL is allowed or blocked for a chosen crawler by applying the rules in a robots.txt file. This free online robots.txt tester resolves Allow and Disallow rules with the same precedence real crawlers use and shows you the exact matching rule and its line number.

How do I check if a URL is blocked by robots.txt?

Paste your robots.txt or fetch it live, pick a user-agent such as Googlebot, then enter one URL or path per line and click Test. The tester returns an Allowed or Blocked verdict for each path along with the specific rule that decided it.

How does Google decide between Allow and Disallow rules?

Google matches the most specific user-agent group, then within it the longest matching path pattern wins; if an Allow and a Disallow are the same length, Allow wins. This tester mirrors Google's open-source robots.txt parser so you get the same result Googlebot would.

Can I test robots.txt for different crawlers like Bingbot or GPTBot?

Yes. Choose any preset crawler — Googlebot, Bingbot, YandexBot, AhrefsBot, GPTBot, ClaudeBot, PerplexityBot and more — or enter a custom user-agent token. The tester picks the matching group, falling back to the * group when no specific one exists.

Do robots.txt patterns support wildcards like * and $?

Yes. Path patterns support * to match any sequence of characters and $ to anchor the end of the URL, so a rule like Disallow: /*.pdf$ blocks every PDF. This tester evaluates those wildcards exactly as crawlers do.

Does the host or domain part of a URL affect the robots.txt test?

No. Only the path and query string are matched against the rules; the scheme and host are ignored, which is how crawlers apply robots.txt. You can paste a full URL or just a path and get the same verdict.

Is this robots.txt tester free and accurate for SEO debugging?

Yes, it is free and built on Google's robots.txt matching logic, making it reliable for debugging whether Googlebot can reach a URL before you ship. Other engines like Bing and Yandex follow slightly different rules in some edge cases.

Why is Google not crawling a page I didn't block?

If robots.txt shows the URL as Allowed here, the cause is elsewhere — a noindex tag, a canonical pointing away, no internal links, or a crawl-budget limit. Test the URL here first to rule out a robots.txt block, then check the page's meta tags and Search Console coverage.

Does a robots.txt block remove an already-indexed page?

No. Blocking a URL in robots.txt stops crawling but doesn't remove it from the index — it can still appear (often without a description) if other pages link to it. To deindex, allow crawling and add a noindex tag, or use Search Console's removal tool.

Related tools — SEO · Robots.txt tools