| How to Block Spiders From Visiting and Indexing Your Site
: Robots.txt File Tutorial
There are reasons you might not want your Web site to be indexed by search engines. More likely, there are simply certain pages that you don't want indexed by the major search engines.
For instance, maybe you constructed an elaborate direct marketing site that requires the visitor to enter through your main page and then proceed through a highly structured series of links that lead them to a buying decision.
The internal pages would only confuse visitors who entered through those pages and they would be less likely to buy a product or service.
Whatever your reason, there is a standard that you can implement that will keep most of the major search engine spiders from indexing your Web site. |
|
Here's how to block the spiders. Create a file called "robots. txt" that includes the following code:
user-agent: * Disallow: /*
The first line specifies the agents, browsers or spiders that should read this file and adhere to the instructions in the following lines of code. The second line stipulates which files or directories the spider or browser should not read or index. The example above uses the "/*" which means the agent should not read or index anything as the asterisks denotes "everything."
The robots. txt file must be placed in the root directory
of your Web site. What this means is that if you are hosting
your Web site using one of the free services and your domain
looks something like this:
http:// members. aol. com/ Joesmith/ home. htm you cannot
use the robots. txt file to keep out the spiders, since you
don't have a primary domain name. The primary domain name
is aol. com - and America Online will probably not allow you
to block all the search engines spiders from indexing their
site and the Web sites of the 11 million other subscribers.
This robots. txt file could look like this if there were
specific directories and files that you wish the search engines
not to index:
user-agent: * Disallow: /clients/*
Disallow: /products/* Disallow: /pressrelations/*
Disallow: /surveys/ survey. htm
In the above example the robots. txt file asks the search
engines spider to omit all pages within the following directories:
http:// www. yourcompany. com/ clients/ http:// www. yourcompany.
com/ products/
http:// www. yourcompany. com/ pressrelations/
And the following specific page:
http:// www. yourcompany. com/ survey/ survey. htm
If you are one of the millions of people hosting a Web site
on America Online's server or one of the other free or subdirectory
Web site services and you can't place a robots. txt file in
their root directory, you can use a META tag that talks to
some of the spiders:
<META NAME=" ROBOTS" CONTENT=" NOINDEX">
You will need this META tag on every page in your Web site
that you don't want indexed. If your Web site has 30 or 40
pages (or more), this will take a lot of time. Here's another
reason to buy a good HTML editor like Luckman's WebEdit or
Allaire's HomeSite.
These programs allow you to do a global search and replace
and add an HTML tag to every Web page that you open in the
program. As with all META tags, this META tag goes at the
top of your HTML document between the <HEAD> and </
HEAD> tags.
|
