Naturally you are going to need to use robots.txt file, despite what engines say- this is not always obeyed nor will those block or throttle undesired agents.

Be sure when using that you put opening and closing brackets < and > around the META to use in its proper syntax.

Restrict MSNBot from indexing a page : META NAME=”msnbot” CONTENT=”noindex” /
Restrict all robots from indexing a page: META NAME=”*” CONTENT=”noindex” /

Restrict MSNBot from following links on page: META NAME=”msnbot” CONTENT=”nofollow” /

Restrict all bots from following links on a page: META NAME=”robots” CONTENT=”nofollow” /

Block MSNBot from indexing & following links: META NAME=”msnbot” CONTENT=”noindex,nofollow” /

Stop MSNBot from caching a page: META NAME=”msnbot” CONTENT=”nocache” /

or as an alternative: META NAME=”msnbot” CONTENT=”noarchive” /

Limit MSBOT crawl frequency

If you occasionally get high traffic from MSNBot, you can specify a crawl delay parameter in the robots.txt file to specify how often, in seconds, MSNBot can access your website. To do this, add this syntax to your robots.txt file:

User-agent: msnbot
Crawl-delay: 120

If you still find that MSNBot is placing too high a load on your web server, contact MSN Search Site Owner Support.

When you contact us about an issue, include the following information so that we can help you more quickly:

* The target website address that MSNBot put in the robots.txt file
* The date range when the issue occurred
* The access logs

Consider a free beta trial of botsense.com bot blocking control to control rogue bots and agents using .htaccess and apache.

Popularity: 2% [?]

Share This