Traffic Resources For Webmasters!

Traffic Resources For Webmasters!

Home  Library
 

 

Using  the Robots.txt file on your site properly

  
Certain search engines use a program called a spider to comb through your site and add what they believe to be the relevant pages to their archive.  Using the robots.txt file, you can instruct the spider which pages that you would like to be excluded from their spidering.

<Field> ":" <value>

This file must be created using a standard test editor such as notepad.

The first field of each entry is the User-agent:
This field must be entered for each individual spider you want to provide an exclusion for, or you can enter a * to cover all spiders.

User-agent: googlebot

User-agent: *

The second field of each entry is the Disallow:
This field specifies files or directories that you want excluded from the spider.

Disallow: password.htm

Disallow: /images/

Examples:

User-agent: *
Disallow: /cgi-bin/
Disallow: /images/

or

User-agent: googlebot
Disallow: /images/

We currently use the following program "Traffic Seeker" which in addition to submitting to a large number of search engines, also automatically configures all your meta tags and also will create doorway pages!  Click on the link below to get your copy!

stats