基于Spark的改进K-means算法的并行实现

iming at the deficiency of K-means clustering algorithm, this paper proposes an improved algorithm with the use of simplified silhouette coefficient as the evaluation criterion to measure the k value in K-means to boost the algorithm performance. The K-means++ algorithm is used to choose the initial center points in the K-means algorithm. After setting the k value and the initial center point, morphology similarity distance is used as the similarity measurement standard to assign the data points to the cluster formed by the closest center point. And finally calculate the average silhouette coefficient to determine the appropriate k value. The improved algorithm in this paper is implemented on Spark. Experiments on accuracy, run-time and speedup of four standard datasets show that the improved K-means algorithm not only improves the quality of clustering division compared with the traditional K-means algorithm and SKDK-means algorithm, but also shortens the calculation time, showing good parallel performance in a multi-node cluster environment. The experimental results suggest that the improved algorithm proposed in this paper can effectively improve the algorithm execution efficiency and parallel computing ability.