Results 1 to 2 of 2

Thread: How to allow particular bots and disable all others?

  1. #1

    Default How to allow particular bots and disable all others?

    Here is useful robots.txt which allows important bots and disallows all other.
    Put following rules in robots.txt.

    User-agent: Googlebot
    User-agent: Googlebot-Image
    User-agent: MSNBot
    User-agent: msnbot/2.0b
    User-agent: msnbot/1.1
    User-agent: Slurp
    User-agent: yahoo-mmcrawler
    User-agent: psbot
    User-agent: yahoo-blogs/v3.9
    Allow : /

    User-agent: *
    Disallow:
    Disallow: /cgi-bin/
    Sitemap: http://www.domainname.com/sitemap.html
    * Change domainname.com to the domain name for which robots.txt is to be set.

  2. #2
    Join Date
    Sep 2011
    Posts
    12

    Default

    This is a great post and makes me think of where I can fit in.
    I do a little bit of everything mentioned here.

Tags for this Thread

Bookmarks

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •