基于Spark的两表等值连接过程优化

he equivalence connection between tables in the statistical analysis of data is one of the commonly used operations, but the price is relatively high. In big data environment, the connection of large scale data tables is less efficient. In order to solve this problem, this paper proposed a method for optimization of two-table equivalent connection process based on Spark: first, constructed the Bloom Filter to complete the filtering operation according to the low density of data density; secondly combined the advantages of Simi-Join method and Partition Join methods, the he greedy algorithm Splitting methods is used for the filtered unilateral table; lastly joined the split subsets . Then the connection process of two big tables was changed into two stages of the two small table connection, Cost analysis and experiments show that the proposed algorithm has improved performance compared with the existing Spark-based connection operation performance and data tilt.