Online algorithms for estimating change rates of web pages

• Published in: Performance evaluation Vol. 153; p. 102261
• Main Authors: Avrachenkov, Konstantin, Patil, Kishor, Thoppe, Gugan
• Format: Journal Article
• Language: English
• Published: Elsevier B.V 01.02.2022; Elsevier
• Series: SI: ValueTools 2020
• Subjects: Change rate; Computer Science; Database synchronization; Heavy ball; Information Retrieval; Machine Learning; Mathematics; Momentum; Probability; Social and Information Networks; Statistics; Stochastic approximation; Web crawling; Stochastic approximation; Momentum; Web crawling; Change rate; Database synchronization; Heavy ball

A search engine maintains local copies of different web pages to provide quick search results. This local cache is kept up-to-date by a web crawler that frequently visits these different pages to track changes in them. Ideally, the local copy should be updated as soon as a page changes on the web. H...