What is Robot.txt?

[Caleb]

Is it necessary to have robot.txt within my website? Why and How to create it?

[chintan]

Is it necessary to have robot.txt within my website? Why and How to create it?

Robots.txt is a text (not html) file you put on your site to tell search robots which pages you would like them not to visit. The location of robots.txt is very important. It must be in the main directory because otherwise user agents (search engines) will not be able to find it – they do not search the whole site for a file named robots.txt.

There are two important considerations when using /robots.txt:

* robots can ignore your /robots.txt. Especially malware robots that scan the web for security vulnerabilities, and email address harvesters used by spammers will pay no attention.
* the /robots.txt file is a publicly available file. Anyone can see what sections of your server you don't want robots to use.

So don't try to use /robots.txt to hide information.

for more information, you can visit Robotstxt

[Alexander]

Most of the time they are used to hide some pages which include heavy sized images from bots, because there is not benefit if bots crawled you images from SEO point of view.

[Nishant]

Robots.txt is a text (not html) file you put on your site to tell search robots which pages you would like them not to visit. The location of robots.txt is very important. It must be in the main directory because otherwise user agents (search engines) will not be able to find it – they do not search the whole site for a file named robots.txt.

There are two important considerations when using /robots.txt:

* robots can ignore your /robots.txt. Especially malware robots that scan the web for security vulnerabilities, and email address harvesters used by spammers will pay no attention.
* the /robots.txt file is a publicly available file. Anyone can see what sections of your server you don't want robots to use.

So don't try to use /robots.txt to hide information.

for more information, you can visit Robotstxt

Great Information! Thanks Chintan, you have covered all the aspects concerning Robot.txt

Useful

[Travis]

As we know now that the main function of robot.txt is to restrict Search engine bots from visiting particular pages of a website. I would like to know that is there any limit on the urls to be placed in robot.txt?

[jack]

Is it beneficial to hide flash from search engines with the help of 'Robot.txt'?

[Ronan]

Is it beneficial to hide flash from search engines with the help of 'Robot.txt'?

Yes Jack, and the reason behind this may be just because, heavy flash images affect the page-load speed of a webpage very badly, and now a days Google has started to consider this factor very seriously.

[Nishant]

Yes Jack, and the reason behind this may be just because, heavy flash images affect the page-load speed of a webpage very badly, and now a days Google has started to consider this factor very seriously.

Well if your main priority is to take care ranking in Google then it's beneficial to hide such heavy flash images from it, but I have heard that Bing behaves opposite to Google when the things are about flash images. So if anyone preferring Bing over Google then he need not to keep such webpages with flash images within robot.txt.