83 8.6 The Deep Web

There are billions of web pages connected through the Internet. However, estimates are that traditional search engines on the web index just a small fraction of those pages. The rest has come to be known as the “deep web.” This “deep web ” consists of material in databases embedded in websites, Flash animations, information that is password protected or behind registration screens that cannot be “spidered,”, and “dynamic” content of various sorts such as airline flight schedules and news pages that are updated minute-by-minute.

Notably, this information is not necessarily completely “buried” and inaccessible to you as a user. Rather, much of this information is that which cannot be accessed through broad search engines such as Google or Yahoo!. It is important for you to know, then, that the material accessible to you through a web search site is significantly limited in scope – there may well be valuable information out there that you will have to dig a little more deeply to find.

If you need to do a thorough search on a topic, particularly one for which important information would be found inside a database, a directory, or in an Excel spreadsheet, you will have to look into the hidden nooks and crannies of the web to find what you need.

Some of the search engines that claimed to provide results from the deep web are no longer around. The experts now suggest that you identify some of the specialized search engines in the specific field or area of interest and use those to locate materials that the huge general search engines can’t reach. Examples include Economagic for data files, charts and Excel files for a vast array of economic data or Guidestar for information from and about the hundreds of thousands of non-profit organizations in the U.S.