Robots.txt Tester
Test whether specific URLs are allowed or blocked by a site's live robots.txt.
About this tool
Fetches a site's robots.txt and tests the paths you give it, for the crawler you choose. For each path it reports whether it is allowed and names the exact rule that decided it.
Matching is not "first rule wins". The longest matching pattern wins, and when an Allow and a Disallow match equally, Allow wins — which is why reading the file top to bottom often gives the wrong answer.
Common questions
Does robots.txt stop a page appearing in search results?
No. It stops a page being crawled. A blocked URL can still be listed — usually with no description — if other pages link to it. To keep a page out of results, allow crawling and use a noindex directive instead.
Why does my Disallow rule not block the URL?
Almost always because a longer Allow rule also matches. Under RFC 9309 the longest matching pattern wins, so Allow: /blog/public/ beats Disallow: /blog/. The deciding rule is shown for every path.
Can I block AI crawlers here?
You can test the common ones — GPTBot, ClaudeBot, Google-Extended, CCBot. Whether a crawler obeys robots.txt is its own choice; the file expresses a preference, it does not enforce anything.