一种基于Kafka/Hadoop/Hive平台的大规模智慧路灯的数据采集与处理系统

With the rapid development of information technology, smart street poles appear in people\'s vision. Different from ordinary street poles, smart street poles integrate a lot of intelligent modules. These modules report a large amount of monitoring data to the server in real time. As the number of smart street poles increases and the running time stretches, thesemonitoring data will be accumulated into mass data, and the TCP connection counts between the street pole and the server will be overload. In order to solve the problem of large-scale TCP long-link problem and the massive data collection, storage and processing caused by large-scale smart street poles, the paper uses Kafka, Hadoop, Hive and other technologies to design and implement a distributed smart street poles data Acquisition, storage and processing system.