Some search engines may look for and analyse its contents of a text file named robots.txt in your main directory. The robots.txt file is may be used to tell search engines which directory's they should not search in.
If the search engine is able to find robots.txt it will analyse its contents.
To Allow access all directory's:-
User-agent: *
Disallow:
To Disallow access all directory's. (note the back slash):-
User-agent: *To exclude specific directory's. In this example, three directories are excluded e.g. cgi-bin, data and images:-
Disallow: /
User-agent: *
Disallow: /cgi-bin/
Disallow: /data/
Disallow: /images/
Place robots.txt in your main directory.
Some search engines may have specific codes to prevent that particular search engine from searching a website.
Excluding a file from an individual Search Engine. For example for Google:-
If you have a file, keepoutfile.htm, that you do not wish to be indexed by Google. Google uses a spider that Google sends out is called 'Googlebot'. You would add these lines to your robots.txt file:
User-Agent: Googlebot
Disallow: /keepoutfile.htm
View the robots.txt file for his site.
Advice: Recommended use robots.txt. Some search engines may list sites quicker or be able to improve site ranking for sites which use them. Only ever use relevant robots.txt that relate to your web page. They should never be abused or over used. Also see Meta Tag
Don't rely on robots.txt to stop users from viewing information or as a powerful security tool.
Remember to rename the file exactly as "robots.txt" (all lowercase) and Not robot.txt in singular. Place robots.txt in your main directory.
More information:- Robotstxt.org and Database of Web robots, Overview
Also view Google Site Map, Site Maps and Content Doorway Pages
Please Help support A Great Portal with a donation. If you find the Advertising Methods FREE TIPS useful.
![]()
![]()
![]()
![]()
![]()
Robots.txt Generator Tool © SEO Chat™
Report Problem with Tool.
![]()
![]()
![]()
Search Help for Compare Bargains.
A Computer Portal. Freeware, Shareware. Download software. Computer languages and Programming code. Including PERL Scripts and Java Scripts. Webmaster Tools. Internet Marketing, Website promotion. Hardware Help from BIOS to Windows and UNIX.
® © TM are owned by respective authors and websites. There may be a charge for some software.