Crawlers (or bots) are used to gather data out there on the web. By using web site navigation menus, and studying inner and external hyperlinks, the bots start to know the context of a web page. Of course, the words, pictures, and different information on pages additionally help search engines like