Web Crawler

A “spider” or “Web crawler” – is a software application. It’s a “bot.” It runs automated tasks to download webpages for the purpose of recording their content and how they are connected. A search engine can then search that evolving set of data.