Hadoop中网络感知的Reduce任务放置算法
Network-Aware Reduce Task Placement Algorithm In Hadoop
针对MapReduce中shuffle阶段的数据传输时延成为分布式计算应用性能瓶颈问题，本文提出一种网络感知的Hadoop任务放置算法，该算法通过将Reduce任务放置在和Map任务通信带宽充足的节点来对shuffle阶段数据传输进行优化。仿真结果表明，通过与原生的任务放置机制相比，采用网络感知的任务放置算法能够使作业的平均完成时间降低42.11%。
In Hadoop, Reduce tasks have to read a lot of data across racks in shuffle stage. To optimize the transmission time during the shuffle stage, this paper proposes a network-aware task placement algorithm, which places the Reduce task with consideration of its data sources and network utilization. Simulation results show that, compared to original algorithm, the proposed network aware algorithm can reduce the job completion time by 42.11%.
郭宏翔、张东旭、屈戈
计算技术、计算机技术
HadoopMapReduce任务放置网络感知大数据
HadoopMapReduceTask PlacementNetwork AwareBig Data
郭宏翔,张东旭,屈戈.Hadoop中网络感知的Reduce任务放置算法[EB/OL].(2015-12-08)[2025-08-04].http://www.paper.edu.cn/releasepaper/content/201512-411.点此复制
评论