Hadoop剩余有效时间调度算法

Parallel computing can be used to solve complex distributed computing and large processing problems, by the use of multiple machines computing power at the same time,we can achieve the goal of fast parallel computing. Hadoop as parallel computing, open source platform, is widely used in the web log analysis, data mining and image processing, etc. With the deepening of the use of hadoop, various kinds of services are on the bottleneck of the processing timeliness. Existing hadoop scheduling focused on reducing execution time, while ignoring the ageing response requirements. In order to improve the ability of hadoop processing timeliness task, this paper designs a hadoop remaining valid time scheduling algorithm. When scheduling, remaining valid time real-time scheduler will estimate the remaining valid time of the task, And on this basis to update the task execution order. Results show that the remaining time effective scheduling can improve the ability of timing hadoop processing tasks.