Crawl Log Analyzer
Paste server access logs to see which crawlers came, what they hit, and what status they got.
About this tool
Reads server access logs and reports what search engine crawlers actually did: which bots, how many requests each, which paths they spent their time on, and what status codes they received.
The access log is the primary record. Search Console reports crawling days later and in aggregate; the log has every request. Most crawl budget questions — why a section is not being indexed, where the crawler is wasting its time, whether a redirect chain is being walked repeatedly — are answerable from here and from nowhere else.
**The user agent is not proof.** Anyone can send `Googlebot` in a header, and plenty of scrapers do. Confirming a request really came from Google means a reverse DNS lookup on the client IP and a forward lookup back again. This tool reports what each request *claims* to be, and never states it as fact.
Combined and common log formats are recognised, along with most variations that keep the fields in the usual order.
Common questions
How do I confirm a request really came from Googlebot?
Run a reverse DNS lookup on the client IP. It must resolve to a googlebot.com or google.com hostname, and a forward lookup on that hostname must return the same IP. Anything else claiming to be Googlebot is not.
Where do I find my access logs?
On shared hosting, usually in a logs directory in your account or downloadable from the control panel. On a server, /var/log/nginx/access.log or /var/log/apache2/access.log. Behind Cloudflare, the origin log shows only what reached it.
What should I look for?
Crawlers spending most of their requests on parameter URLs, paginated archives or redirects. That is time not spent on pages you want indexed, and it is the clearest crawl budget problem a log can show.
Why are there AI crawlers in the list?
Because they are in your logs whether or not you were expecting them, and several ignore robots.txt inconsistently. Seeing the volume is the first step in deciding whether you mind.