基于Spark Streaming的网管系统告警过滤算法的设计与实现

With the rapid development of information technology, big and high-speed data is produced in more and more scenarios. So how to process and analyze big data in real time becomes the focus problem in the Big Data Era. In this context, distributed data stream processing emerged. Among various stream processing technologies, Spark Streaming is quasi-realtime, consistent, fault-tolerant, high throughput, and easy to integrate with other libraries of Spark. Spark Streaming provides operations such as sliding windows and incremental processing. The thesis uses Spark Streaming to design and implement the alarm filtering algorithms in network management system based on practical need, in order to filter out duplicate alarms.