System and method for prioritizing websites during a webcrawling process

A system and method for prioritizing a fetch order of web pages. The method comprises extracting by a web crawler a set of candidate web pages to be crawled. Each web page in the set of candidate web pages is associated with a website in a computer network. A determination is made to determine if a...

Full description

Saved in:
Bibliographic Details
Main Author BLACKMAN DAVID L.,CHING MICHAEL,DILL STEPHEN,GONZALEZ IVAN E.,MARCUS ADAM,MEREDITH DANIEL N.,NGUYENLINDA A. L
Format Patent
LanguageEnglish
Published 03.10.2007
Subjects
CALCULATING
COMPUTING
COUNTING
ELECTRIC DIGITAL DATA PROCESSING
PHYSICS
Online AccessGet full text

Cover

More Information
Summary:A system and method for prioritizing a fetch order of web pages. The method comprises extracting by a web crawler a set of candidate web pages to be crawled. Each web page in the set of candidate web pages is associated with a website in a computer network. A determination is made to determine if a first website score for the website is in a website score database. The first website score is associated with web pages in the set of candidate web pages if the first website score exists in the website score database. The set of candidate web pages is prioritized with respect to an associated website score for each web page in the candidate set of web pages. Content is retrieved from the set of candidate web. Hyperlinks are extracted from the content. The hyperlinks are stored in a memory unit.
Bibliography:Application Number: CN200710091563