一种基于枚举的网络实体爬取方法研究

Nowadays the information is increasing on the Internet, and obtaining web-scale entities has become an important topic. However, traditional relation-based crawling strategy has some shortages. To fix them, this paper first analysis the enumerability of URL, and then raise an enumeration-based algorithm framework to do the crawling, including na?ve feature incorporation, clustering, feature incorporation after clustering, feature completion and enumeration sequence optimization. The experiment on Sina Weibo Place and Baidu Wenku shows that this approach has good performance on sample coverage and accuracy, which is better than traditional approach in many cases.