Here is useful robots.txt which allows important bots and disallows all other.
Put following rules in robots.txt.
* Change domainname.com to the domain name for which robots.txt is to be set.User-agent: Googlebot
User-agent: Googlebot-Image
User-agent: MSNBot
User-agent: msnbot/2.0b
User-agent: msnbot/1.1
User-agent: Slurp
User-agent: yahoo-mmcrawler
User-agent: psbot
User-agent: yahoo-blogs/v3.9
Allow : /
User-agent: *
Disallow:
Disallow: /cgi-bin/
Sitemap: http://www.domainname.com/sitemap.html






Reply With Quote

Bookmarks