Robots text file Generation

Why do I need a robots.txt file?

Search engine spiders look for the robots.txt file when they visit your site. Most of them will follow what you write in the robots.txt file. For example, is there an area of the site that is still under construction or perhaps you do not want the robot to index your images directory. The simple fact that the search engines look for it should be enough reason to place a file in your root directory on your webserver. This will prevent many of the 404 errors that not having one will create.

What is a simple robots.txt file?

You could put just a blank text file in the root directory of your site. name it robots.txt. Doing so would prevent the 404 errors from occurring when the serach engines look for the robots.txt file.

How do I prevent the spider from reading a particular file?

Into your robots.txt file add the following two lines where directory is the name of the directory that the file is in and filename is the name of the file you do not want the spider to visit. User-agent: * Disallow: /directory/filename.html

How do I prevent a particular robot from spidering the pages?

Copy the following two lines into the robots.txt file replacing Name of spider with the spiders actual name.
Disallow: /
User-agent: Name of spider

Need the name of the spider? Search Engine spider names

How do I create a robots.txt file?

Enter the pages you wish to exclude below and the tool will create a robots.txt file for you to use.

How can I validate my robots.txt file?

tool.motoricerca.info

Where do I go to get more information?

www.robotstxt.org


	Robots.txt Generator Tool © SEO Chat™ Allowed User Agent Select user agent or use default for all agents Enable All Webpages Yes No Exclude These URL's Enter URLs you wish to exclude /cgi-bin/ /images/

| | | | |