基于Kafka的视频流采集系统

In the recent years, the data generated by human activities are growing explosively because of the rapid development of the Internet and the decline in the price of storage devices and the increasing number of electronic devices. The increase in data leads to the rapid development of big data technology. Batch computing and stream computing are main forms of big data computing, and the research and practical experience on stream computing which emphasizes real-time are comparatively less. LinkedIn presents a distributed message queue named Kafka as a solution for stream data storage. Kafka has been widely recognized by the industry because of its high throughput and low latency and it is widely used in text stream data storage. Nowadays, our experience in applying Kafka to video stream data storage is insufficient in the industry. Therefore, this paper utilizes the advantages of Kafka. We design and implement a video stream collection system based on Kafka. In our method, we present a Kafka storage solution towards the features of video stream data and combines with streaming media technology and video encoding technology. The system has high efficiency and practicability.