Have had a very strange set of httpd access logs today from before a pagespeed mod and apache crash. It seemed that the following bots appeared to be indexing the site, like crazy, all at once.
- googlebot
- googlebot-image
- bingbot
- yahoo! slurp
- yandexBot
- Baiduspider
- Sogu Web spider
- AhrefsBot
- SeznamBot
- msnbot
Plus a couple more I can't locate at once.
This looks highly suspicious to me. In amongst the bot requests there are some odd URL requests that are not in the site, so are not being spidered (things like word press admin URLs and it not even a wp site).
The bot IPs look genuine. Anyone know how someone might be able to trigger a bot-storm? Never seen that before - seems very odd indeed, can't believe this is mere coincidence.
