About the Robots.txt / Sitemap Checker
The robots.txt and sitemap checker retrieves a public site's crawler instructions and looks for declared sitemap locations. It provides a fast technical check when a site is not being crawled as expected or after robots and sitemap changes.
How to use this tool
Enter a website URL or hostname and run the check. Review whether /robots.txt returns successfully, which user-agent groups and directives are published, and whether Sitemap lines point to reachable canonical files. Test the production hostname search engines actually crawl. Remember that robots rules use path-prefix matching and can differ by crawler, so inspect the most specific group that applies to the bot you care about.
How to read the result
User-agent identifies the crawler group. Allow and Disallow directives describe paths that crawlers may or may not request, subject to each crawler's implementation. Sitemap declarations provide absolute sitemap URLs and can list more than one file. A missing robots.txt commonly means crawling is unrestricted, while a site-wide Disallow can prevent crawling. Robots rules control fetching; they are not a reliable way to remove an already known URL from search results.
Limits and privacy
This checker reports the published files and does not emulate every search engine's parser or indexing decision. A valid sitemap does not guarantee indexing, and a robots-allowed URL can still be excluded for quality, canonicalization, noindex, or response reasons. CDN variants, authentication, and bot-specific responses may differ from this server-side request. The submitted public site is processed by buffer.lol; internal targets are blocked.
Common questions
- Does Disallow remove a page from Google?
- Not necessarily. It prevents compliant crawlers from fetching the path, but the URL can still be known or shown. Use an indexable noindex response when removal is the goal.
- Do I need to list the sitemap in robots.txt?
- It is a useful discovery mechanism but not the only one. Sitemaps can also be submitted directly through search-engine webmaster tools.