An Insight Into Google’s Indexing Procedure

Whenever a net user approaches [tag]Google[/tag] for a search, Google conducts an [tag]indexing of WebPages[/tag] based on the [tag]keyword phrases[/tag] outlined by the net researcher. The index, in turn, is based on a logical algorithm that functions through a “bot” – a piece of software built to go through each web Page of available websites, categorize them and place it into Google’s Database.

Google functions through three bots, namely the [tag]Adsense bot[/tag], the [tag]Freshbot[/tag] and the [tag]DeepCrawl[/tag]. While Google Adsense bot is restricted to Adsense subscribed websites, the Freshbot and DeepCrawl play a role in the generic indexing procedure.

The Freshbot visits the websites based on the popularity of the websites and the consequent frequency of change expected over them. For example, the Freshbot crawls through certain pages of Amazon and CNN every ten minutes. A typical website is crawled through every one to fourteen days, subject to popularity.

During each visit, Freshbot identifies all deeper links in a website to be crawled later by DeepCrawl. DeepCrawl visits all the links found by Freshbot once a month. This also explains the time span taken by Google in indexing an entire site, even if the site includes a Google sitemap.