Robots.txt Analyzer

Audit a robots.txt for issues and coverage.

Analyse a robots.txt for syntax problems, conflicting rules, missing sitemaps and risky wildcards — with a clear, prioritised report.

How to use Robots.txt Analyzer

  1. Add Your File

    Paste your robots.txt into the free online robots.txt analyzer, fetch it live by entering a site URL, or upload the file straight from your computer.

  2. Load A Sample

    Not sure where to start? Click Load sample to drop in an example robots.txt so you can instantly see how the analyzer parses groups, rules and sitemaps.

  3. Run The Analysis

    Click Analyze and the tool audits your robots.txt for syntax errors, conflicting rules, risky wildcards, missing sitemaps and accidental whole-site blocks.

  4. Review The Summary

    Scan the at-a-glance stat chips showing groups, rules, sitemaps, errors, warnings and file size to gauge the overall health of your robots.txt at a glance.

  5. Fix The Issues

    Work through the prioritised list of errors and warnings, inspect each user-agent group and sitemap, and correct any rule that could block search engines.

  6. Copy Or Download

    Copy the plain-text report or download it to share with your team, then update your robots.txt and re-analyze to confirm every issue is resolved.

Frequently asked questions

What is a robots.txt analyzer and what does it check?

A robots.txt analyzer audits your robots.txt file for syntax errors, conflicting rules, missing sitemaps and risky wildcards, then returns a prioritised report. This free online robots.txt analyzer parses every User-agent group with its Allow, Disallow and Crawl-delay rules so you can catch mistakes before they hurt your SEO.

How do I check my robots.txt for errors?

Paste your robots.txt, fetch it live by entering a site URL, or upload the file, then click Analyze. The tool flags syntax errors, warns about risky patterns like Disallow: / and duplicate user-agent groups, and shows a clear breakdown of groups, rules and sitemaps.

Why is robots.txt accidentally blocking my whole site from Google?

A line like Disallow: / under User-agent: * blocks every crawler from your entire site, which is the most common robots.txt mistake. This analyzer highlights whole-site blocks and other risky rules so you can fix an accidental Disallow before it deindexes your pages.

Can this tool fetch and analyze a live site's robots.txt by URL?

Yes. Enter a domain and the analyzer fetches the site's /robots.txt server-side, then runs the full audit. If the site has no robots.txt it is treated as allow-all, and if the fetch fails you can simply paste the contents instead.

Does the robots.txt analyzer detect AI crawler rules?

Yes. It highlights rules targeting AI crawlers such as GPTBot, ClaudeBot, CCBot, Google-Extended, PerplexityBot and Bytespider, so you can clearly see which AI bots your robots.txt is blocking or allowing.

Are relative sitemap URLs valid in robots.txt?

No. The Sitemap directive must be a full absolute http or https URL, such as Sitemap: https://example.com/sitemap.xml. The analyzer warns when a sitemap line is relative and also notes when no sitemap is declared at all.

Is this robots.txt checker free and is my file kept private?

Yes, it is free to use. The analysis runs in your browser, and the only network request is the optional server-side fetch of a live site's robots.txt — your pasted or uploaded file is not stored.

What are the most common robots.txt mistakes?

The big ones are an accidental Disallow: / blocking the whole site, blocking CSS/JS that Google needs to render the page, a relative (not absolute) Sitemap URL, and conflicting rules across user-agent groups. This analyzer flags each of these with a prioritised report.

Should I block CSS and JavaScript in robots.txt?

No. Google needs to load your CSS and JS to render and understand the page — blocking them can hurt rankings. The analyzer warns when your rules disallow resource folders like /assets or /wp-includes that contain files the crawler needs.

Related tools — SEO · Robots.txt tools