Quantcast
Viewing latest article 24
Browse Latest Browse All 35

Google Crawl rate & the log file mystery

According to Google Search Console for a website I am working on, Googlebot crawls ~5000 pages per day (min 2500, max 8500).

However, when looking at the Apache log files, GoogleBot only shows up ~10 times per day ...

For example:

66.249.64.88    [22/Jan/2020:15:09:01   +0100]  [22/Jan/2020:15:09:01 +0100]    GET / HTTP/1.1  200 1358    Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)

It is GoogleBot since doing a reverse dns does point to Google servers:

$ host 66.249.64.8888.64.249.66.in-addr.arpa domain name pointer crawl-66-249-64-88.googlebot.com 

But I am wondering : If GoogleBot appears only 10 times in Apache log files while it crawls 5000 pages per day, where are the remaining 4990 crawls going?

How can I know which resource GoogleBot crawls when it does not appear in the log files ?

Thanks!


Viewing latest article 24
Browse Latest Browse All 35

Trending Articles