.htaccess and Robots.txt for blocking bad bots
I am fed up with having huge amount of traffic from bots that serves no purpose whatsoever. These are bots that don’t make sense at all. Its impossible to control these bots using a Robots protocol as they don’t follow them and usually mask their user agent. My site is in English and the content is also targeted a completely different audience than the Chinese yet the site has relentless Chinese traffic.
If you want a better Robots.txt file then this tool might be of help to you.
Going back into the cave!