Agent warfare is something I am really interested in since there are so many ramifications of spiders and automated agents. I have spoken to the developer and I am impressed with the depth of his knowledge on agents and just how diverse the agents are these days- most of it translates into massive bandwidth and resource suckage for no good reason. This is an incredible hit for smaller publishers of multi-media content.

I am not saying spiders are bad but in many cases their value is one-sided- they benefit the entity running the spider.

BotSense is offering a free tool to help analyze and block rogue agents and bots.

From the site:

We welcome you to test out our latest beta tool called BotSense .htaccess. The beta tool will allow you to easily create a .htaccess file to put on your server, blocking bots you specify, saving you time and money, and most importantly, giving you back control. The Beta is free.

Why .htaccess you ask? Because rogue bots don’t give a damn about robots.txt.

They are not good netizens and if you think robots.txt wards off these agents you need to revisit what is going on in your site.

Popularity: 1% [?]

Share This