Index 206 tools12 categoriesRatings: ours · Trustpilot · G2Pricing checked on each vendor's sitePress / to search
Search tools/
My stack 0
Saved on this device · no account

    Nothing saved yet. Use Save on any tool.

    Open my stack
    T-02 · free tool

    robots.txt tester

    Paste the file, pick a crawler, test a path. The answer names the rule that decided it, matched the way Google documents: most specific group, longest rule, allow wins a tie.

      How Google matches rules

      1. The crawler uses the group whose User-agent matches its name most specifically; Googlebot-Image falls back to a Googlebot group, then to *. Groups for the same agent are merged.
      2. Within that group, the rule with the longest path that matches wins. * matches any run of characters and $ anchors the end.
      3. If an Allow and a Disallow are equally long, Allow wins.

      What robots.txt does not do

      It stops crawling, not indexing. A blocked URL can still appear in results if other pages link to it — without a description, because Google could not read the page. To keep a page out of the index, let it be crawled and use noindex. The tester flags directives Google ignores, such as crawl-delay and noindex inside robots.txt.

      AI crawlers

      GPTBot, ClaudeBot, PerplexityBot and Google-Extended each read their own group. Test them by name to see whether your file lets AI products train on or quote your pages.

      Compare0 / 3
      Pick one more