I learned a long time ago that search engines can be a powerful tool in the arsenal of anyone whether they are a business man or woman, a child learning about dinosaurs in school or a regular Joe who wants to find out about the latest medical breakthrough or the hottest gossip about Paris Hilton. The internet has a myriad of uses and the better you understand how to use these search engines the easier it will be for you to find the info that you really want or need to have when you need to have it.

When I ask my students if they understand how search engines gather their information. The most common answer is that they know what they are and understand the importance of being indexed and listed on them - well some do - but the minute you start talking about spiders, robots and the like, they glaze over. Poof they are gone, just like that. And if you mention the word Boolean, they have not the foggiest idea what it could mean. And why would they?

Most of us use the different search engines for such basic things that we don't need to really learn all of the different functions or all of the best ways to find certain things. We just type in the basics of what we are looking for and we are off and running with what the search engines dredges up. But for many people this isn't enough. If you are trying to get something for your business or you have a thesis due or a term paper then you need to get as much relevant data as possible and you can learn a lot about search engines that will make them an even more powerful ally than they are now.
This article aims to shed some light on search engines and their inner-workings. After all, if you want to benefit from being listed on search engines, you should know how they work, or at least know enough to be dangerous.
So as Lewis Carroll said "Let's start at the beginning..."

Most people use the term search engine to generically describe, crawler based search engines, human powered directories, and the engines that are a combination of the two called or hybrid , but these three search engines gather their information in completely different ways. Let's take a quick look at these types of engines and how they collect their data.

The first one we will discuss is the crawler based search engine. The crawler based search engine is composed of three parts. They are: the spider or crawler, the index and the software engine.  The spider visits a web page, examines it, and then follows the links to other pages that are within the same site. So if you here someone mentioning there site has been 'spidered' or 'crawled' this is what they meant. Generally spiders return to the site on a regular basis, set by the program, to look for changes.