What actually breaks crawling

The most common self-inflicted SEO wound is a stray Disallow: / left over from a staging environment, blocking the entire production site from Google. This tool flags it explicitly, along with directives that do not match the robots.txt spec, which crawlers are free to interpret however they want, or simply ignore.

Why the sitemap matters more than people think

A Sitemap: line in robots.txt is not required, Google can discover pages by following links, but it is the fastest way to tell a crawler exactly what exists and when it changed. A sitemap with malformed XML, a wrong content type, or zero <loc> entries is effectively invisible even though the file itself returns a 200.

What this tool does not check

It validates structure and syntax, not actual crawl behaviour or Google's indexing decisions. For that, robots.txt and sitemap status straight from Google's own crawler data lives in Google Search Console, this tool is what you run before that, to catch the obvious mistakes early.