Firstly, we need to define what kind of visitors we want to track on our website. What I have come up with are normal visitors (people like you and me), and robots. Robots include any form of automatic 'web crawling' or 'spidering' program such as search engines, or any automatic program that visits your website.

Tracking Visitors on your Website
Being a webmaster, I want to know as much as possible about the visitors on my website.

There are some ok free website tracking programs out there, but what I really want to know is which customers are buying, and which aren't. Am I losing customers because I'm not giving them what they want, or because there is a broken link on my page?

A visitor tracking program should be able to give you this information, but the traditional 'log file analysis' simply doesn't have the right information. What you want is visitor's movements, their habits and the resulting trends from this information.

What I really want to see is what each visitor does. What page are they looking at? Did they read the whole page? Where did they go next? WHY DIDN'T THEY BUY?

I'll be adding more info to this section soon.

Tracking Robots / Search Engines on your Website :
Robots can be a variety of things, ranging from the big search engines like Google, Yahoo or Altavista to Spam (email) Harvesting robots or Link Checking robots. The one constant for all robots is that they are automated.

The most important thing with robots is to know that they are robots.

I might think I am getting 800 visits a day (great!), but in reality 750 of those might actually be robots. I.e. not paying customers.

Quite often robots that are up to no good (email harvesting programs etc) don't bother to, or deliberately hide the fact that they are robots. Even some search engines such as Inktomi hide the fact that they are a robot, to make sure that I am showing them the same information on my website as a normal user.

Here's a list I have compiled of the different robots I know about:

Search Engines :
These are the most common, and usually don't mind letting you know that they are a search engine (with the exception of Inktomi). The robots are used to update the search engines 'cache' of a website, so that searches made on the search engine are up-to-date and accurate.

Examples are: Googlebot 2.1 (www.google.com), Zyborg 1.0 (www.looksmart.com), Slurp (www.inktomi.com), Scooter 3.3 (www.altavista.com)
The Good: Bring in visitors, especially the bigger search engines.
The Bad: Nothing I can think of.
Would I want this robot on my site: YES!!

Link Checkers :
The aim of these robots is to follow the links on your website and make sure none of them are broken. Often accompanied by a feature that emails you if it finds a broken link. Can be run on your computer or online.

Examples are: Link Walker (www.seventwentyfour.com), Watchfire® WebQA™ 2.0 (www.watchfire.com), The Google Directory has more...
The Good: Makes sure your site is working properly
The Bad: Can be used as marketing tool (I have had a few emails from people saying 'we found a broken link on your website, would you like to pay for this service on a regular basis')
Would I want this robot on my site: Only if you specifically asked for it

Website Availability
Similar to Link Checkers, except they monitor your website and alert you if it suddenly becomes unavailable (i.e. your host/server has died). Can be useful, but if your host is unreliable enough for you to use a third-party service like this, swap your hosting ASAP.

Examples are:

Website Availability (www.websiteavailability.com), NetMechanic (www.netmechanic.com/monitor.htm), The Google Directory has more...
The Good: Makes sure your site is available
The Bad: Can 'bombard' your website with hundreds of hits in a few minutes, slowing everything down and costing you in bandwidth.
Would I want this robot on my site: Only if you specifically asked for it

Spam / Email Harvesters
The worst of all robots, whose pure purpose is to gather email addresses on website's to be used in mass email marketing (or spam). As a webmaster you have nothing to gain from them, and as a consequence they nearly always try to hide who/what they are.

Examples are: Mail Harvester (www.arkysoft.com), Atomic Harvester (www.getyoursoftware.com).
The Good: None
The Bad: Your email is added to a spam list, and you get hundreds of junk emails a day. Costs you bandwidth.
Would I want this robot on my site: NEVER
For more information on robots, how they work and how to use them, see the official website: www.robotstxt.org

Yari McGauley - September 2003