HTTP Header Checker
See the response headers a URL returns, including indexing rules that never appear in the HTML.
About this tool
Fetches a URL and lists the headers it returns, flagging the ones that affect indexing, caching, compression and security.
The header worth knowing about is X-Robots-Tag. It carries the same noindex and nofollow directives as a meta tag, but is delivered by the server and never appears in the page source. People check the HTML, find nothing wrong, and cannot work out why the page is missing from search.
Common questions
What is X-Robots-Tag?
The same indexing directives as a robots meta tag, sent as an HTTP header. It is the only way to apply noindex to a non-HTML file such as a PDF — and the reason a page can be excluded from search with nothing visible in its source.
Which wins, the header or the meta tag?
The most restrictive. If either says noindex, the page is not indexed. Setting them in both places makes the outcome harder to reason about, so pick one.
Should responses be compressed?
Yes for text — HTML, CSS, JavaScript, JSON, XML. A missing Content-Encoding on a large HTML response usually means compression is off, which is bytes wasted on every visitor.
Why does this show fewer headers than my browser?
A browser shows headers from every request a page makes. This shows the headers for the URL you gave it, which is the response that decides how that URL is treated.