An application that systematically and continuously traverses the World Wide Web for some purpose. Web search engines, for example, use web crawlers to refresh their data by visiting every website periodically. Performing such traversals is called spidering.