基于PMI的变体短信分词方法研究

With the development of mobile communication network,the growth and spread of spam messages is plaguing mobile phone users .This situation is bring a great challenge to the governance of spam messages.In the spam messages governance, reasonable segmentation of SMS text is the prerequisites for recognition ,classification and intercept. Illegal SMS is ungrammatical, variability and those kinds of things,which leads to the problem of the segmentation accuracy deteriorated.In this paper,a revised PMI is combined with the proposed cross-skip-bi-grams model to resolve the problem.In addition,optimal segmentation,segmentation merge,increment training and feedback training are proposed to improve the usability and robustness of this segmentation method.Experimental results show that this method can improve the accuracy of segmentation on Illegal SMSinformation.