robots.txt Advertising Methods

Some search engines may look for and analyse its contents of a text file named robots.txt in your main directory. The robots.txt file is may be used to tell search engines which directory's they should not search in.

If the search engine is able to find robots.txt it will analyse its contents.

To Allow access all directory's:-

User-agent: *
Disallow:

To Disallow access all directory's. (note the back slash):-

User-agent: *
Disallow: /

To exclude specific directory's. In this example, three directories are excluded e.g. cgi-bin, data and images:-

User-agent: *
Disallow: /cgi-bin/
Disallow: /data/
Disallow: /images/

Place robots.txt in your main directory.

Some search engines may have specific codes to prevent that particular search engine from searching a website.

Excluding a file from an individual Search Engine. For example for Google:-

If you have a file, keepoutfile.htm, that you do not wish to be indexed by Google. Google uses a spider that Google sends out is called 'Googlebot'. You would add these lines to your robots.txt file:

User-Agent: Googlebot
Disallow: /keepoutfile.htm/

View the robots.txt file for his site.

Advice: Recommended use robots.txt. Some search engines may list sites quicker or be able to improve site ranking for sites which use them. Only ever use relevant robots.txt that relate to your web page. They should never be abused or over used. Also see Meta Tag

Don't rely on robots.txt to stop users from viewing information or as a powerful security tool.

Remember to rename the file exactly as "robots.txt" (all lowercase) and Not robot.txt in singular. Place robots.txt in your main directory.

robots.txt are generally used to restrict search engine or user access in some way.

More information:- Robotstxt.org and Database of Web robots, Overview

Check your robots.txt file

Robots.txt syntax checker

Robots Exclusion Standard

Robots.txt Generator Tool © SEO Chat™
Allowed User Agent
Select user agent or use default for all agents
Enable All Webpages
Yes
No

Exclude These URL's
Enter URLs you wish to exclude
/cgi-bin/ /images/
Enter Captcha To Continue
To prevent spamming, please enter in the numbers and letters in the box below

Report Problem with Tool.

.htaccess Files and .htaccess Help. How use to .htaccess

Server Side Includes. SSI

Windows Rewrites

Apache open-source software and Apache Servers. Mod Rewrite

Also view Follow, Nofollow. Index, Noindex, Meta Tags, Google Site Map, Site Maps and Content Doorway Pages

Please Help support A Great Portal with a donation. If you find the Advertising Methods FREE TIPS useful.

Advertising Methods FREE TIPS. Read in conjunction with the General Guide about Advertising Methods.

A Computer Portal. Freeware, Shareware. Download software. Computer languages and Programming code. Including PERL Scripts and Java Scripts. Webmaster Tools. Internet Marketing, Website promotion. Hardware Help from BIOS to Windows and UNIX.