Meaning

An automated software program that systematically browses the internet to collect and index content for search engines.

Examples

• The search engine deployed a web crawler to update its index with the latest news articles.
• Web crawlers follow hyperlinks from one page to another to map the structure of the internet.