MapReduce中Shuffle优化与重构

Hadoop has become the most mainstream cloud computing platform. In the Hadoop distributed computing platform, how to optimize the MapReduce computational performance is a hot issue in the present study. In addition to writing high-performance Map and Reduce functions, enhance operator performance mainly from the optimization of the system framework. This paper describes the details of the MapReduce programming framework, and analyzes the shuffle-stage process clearly. In order to optimize and reconstruct Shuffle in MapReduce, the paper has taken the following measures: compress the output of map end, reconstruct the protocol which is used to copy the data form map end to reduce end and optimize memory allocation on reduce end. Finally, through building a Hadoop cluster, using the MapReduce distributed algorithm test experimental data. Experimental results show that optimization of the reconstructed Shuffle can significantly improve performance MapReduce computing.