Bulk Robots Directive Audit
Check a batch of URLs for noindex and nofollow, in the meta tag and in the header where nobody looks.
About this tool
Fetches a batch of URLs and reports every robots directive on each — from the meta tag, and from the `X-Robots-Tag` header.
A single noindex is easy to find. The hard one is the page in three hundred that has one nobody put there deliberately: a template variant, a page copied from staging with its directives intact, a plugin rule that matched more than it was meant to. Finding that needs a batch check, and it is the reason this tool exists separately from the single-page indexability checker.
**The header is what people miss.** `X-Robots-Tag: noindex` does exactly what the meta tag does and is invisible in the page source, so a page that looks perfectly indexable when you view source can be excluded by a header set once and forgotten. This reads both.
Canonical tags are shown too, because a page canonicalising somewhere else is excluded from search just as effectively as one carrying noindex, and by a route that looks completely different.
Common questions
What is X-Robots-Tag?
The same directives as the meta tag, sent as an HTTP header. It works on any file type, including PDFs and images, and it is invisible in the page source — which is why a page that looks indexable can be excluded by it.
Which wins if the tag and the header disagree?
The most restrictive. A meta tag saying index and a header saying noindex means the page is not indexed. There is no ordering rule that lets one override the other in the permissive direction.
Does robots.txt stop a page being indexed?
No, and this is the common misunderstanding. Disallow stops crawling; a page linked from elsewhere can still be indexed without its content ever being read. To keep something out of the index you need noindex, which means the page must be crawlable.
Why does a canonical count as excluding a page?
Because it asks a search engine to index a different URL instead. The effect is the same as noindex from the searcher's point of view, and it is reached by a route that looks nothing like it.