Architecture Design of Subject-Oriented Web Crawler

In response to the defects of traditional search engines of which it will return a large amount of information when users search keywords, and it's hard to get the useful information, thus propose the thought of subject division, that is the subject-oriented search engine.Subject crawler is the...

Full description

Saved in:
Bibliographic Details
Published in2013 Fourth International Conference on Intelligent Systems Design and Engineering Applications pp. 174 - 177
Main Authors Cao Xin, Zhang Yong, Zhang Fuyan, Ni Changbao
Format Conference Proceeding
LanguageEnglish
Published IEEE 01.11.2013
Subjects
Astronomical Image
Computer architecture
Crawlers
HTML
Image Segmentation
Instruction sets
Memory
Mutual Information
PCNN
Search engines
Sockets
Online AccessGet full text

Cover

More Information
Summary:In response to the defects of traditional search engines of which it will return a large amount of information when users search keywords, and it's hard to get the useful information, thus propose the thought of subject division, that is the subject-oriented search engine.Subject crawler is the key and unique components of the subject-based search engine. The structure of the crawler has a significant impact on the speed of Web resources, as well as the multi-machine distributed extended functionality. This paper studies and designed an architecture of subject crawler, which has flexible modular scalability and multi-machine distributed scalability, and elaborated it.
DOI:10.1109/ISDEA.2013.444