基于Fluentd和HDFS的日志收集系统设计与实现

Log system is an important part of the information system. While Log collection tools are commonly used in stand-alone environment, when collecting logs from several machines, it becomes complicated, and it's hard to manage and process the logs effectively. In this paper, based on Fluentd and HDFS, a distributed log collection system is designed and implemented. This system can collect logs from a variety of heterogeneous platforms and different applications, and store the logs on HDFS. After testing, the system has the characteristics of high performance, high throughput , and it's easy to extend, and supports offline processing of the collected log data. This system is of high practicality.