Fetch and parse any site's robots.txt, and test whether a path is crawlable.
Your robots.txt file tells search engine crawlers which parts of your site they may or may not request. A single wrong Disallow line can deindex an entire section of your site. This tool fetches the live robots.txt, parses every user-agent group, and lets you test whether a specific path is allowed for a specific crawler using Google's longest-match precedence rule.
Disallow: / under User-agent: */robots.txt from that host and lists every user-agent group with its Allow and Disallow rules, any crawl-delay, and the Sitemap lines./blog/my-post) and a user-agent, then press Test Path. Googlebot is the default; try Bingbot, GPTBot or ClaudeBot to see what other crawlers are told.A crawler picks the one group that names it most specifically and falls back to User-agent: * only when no group names it. Inside that group the longest matching path wins, so a longer Allow beats a shorter Disallow. * matches any run of characters and a trailing $ anchors the end of the URL. An empty Disallow: blocks nothing.
Check a page's tags with the meta tag checker, or its structured data with the Schema.org / JSON-LD validator. To see every AI crawler's verdict at once, use the AI crawler checker, or browse all free tools.
The SEO Audit API checks crawlability as one of 80+ checks and returns a scored report. Free tier available.
Enter the domain to load its live robots.txt, then enter a path and a user-agent such as Googlebot and press Test Path. The tool finds the group that applies to that crawler and reports whether the path is allowed or blocked, and which rule decided it.
The longest matching rule wins, which is how Google documents it. With Disallow: /shop and Allow: /shop/sale, the path /shop/sale/shoes is allowed because the Allow rule is the longer match.
No. robots.txt controls crawling, not indexing. A blocked URL can still be indexed without its content if other pages link to it. To keep a page out of search results, let it be crawled and give it a noindex robots meta tag or X-Robots-Tag header.
An asterisk matches any run of characters and a dollar sign at the end anchors the rule to the end of the URL. Disallow: /*.pdf$ blocks every URL that ends in .pdf. This tester supports both.
A missing file (a 404) means every crawler may fetch everything. That is fine for most small sites, though a robots.txt with a Sitemap line still helps crawlers find all your URLs.
No. A crawler follows only the most specific group that names it and ignores the * group entirely. Rules you want Googlebot to follow have to be repeated inside the Googlebot group.