面向BBS的主题爬虫系统的分析与设计

BBS is an important platform for the current network users to make comments and exchange views. There is lots of commercial value, user needs and other important information here. Focused crawler is a topic-oriented information collection system, which can collect information relevant to the subject automatically from the Internet according to user needs, It is used more and more widely in the design of subject search engine and analysis of site structure. This paper describes the principle, main modules and the key technologies of focused Web crawler. A general scheme of the BBS information extraction Web crawler is designed by analyzing the directory structure of dynamic web pages and text structure of BBS. After that, the analysis and design of the focused crawler is described in detail. Finally, Contrast the scheme crawler with the general Web crawler program.