|国家预印本平台
首页|LLMEval-3: A Large-Scale Longitudinal Study on Robust and Fair Evaluation of Large Language Models

LLMEval-3: A Large-Scale Longitudinal Study on Robust and Fair Evaluation of Large Language Models

LLMEval-3: A Large-Scale Longitudinal Study on Robust and Fair Evaluation of Large Language Models

来源:Arxiv_logoArxiv
英文摘要

Existing evaluation of Large Language Models (LLMs) on static benchmarks is vulnerable to data contamination and leaderboard overfitting, critical issues that obscure true model capabilities. To address this, we introduce LLMEval-3, a framework for dynamic evaluation of LLMs. LLMEval-3 is built on a proprietary bank of 220k graduate-level questions, from which it dynamically samples unseen test sets for each evaluation run. Its automated pipeline ensures integrity via contamination-resistant data curation, a novel anti-cheating architecture, and a calibrated LLM-as-a-judge process achieving 90% agreement with human experts, complemented by a relative ranking system for fair comparison. An 20-month longitudinal study of nearly 50 leading models reveals a performance ceiling on knowledge memorization and exposes data contamination vulnerabilities undetectable by static benchmarks. The framework demonstrates exceptional robustness in ranking stability and consistency, providing strong empirical validation for the dynamic evaluation paradigm. LLMEval-3 offers a robust and credible methodology for assessing the true capabilities of LLMs beyond leaderboard scores, promoting the development of more trustworthy evaluation standards.

Ming Zhang、Yujiong Shen、Jingyi Deng、Yuhui Wang、Yue Zhang、Junzhe Wang、Shichun Liu、Shihan Dou、Huayu Sha、Qiyuan Peng、Changhao Jiang、Jingqi Tong、Yilong Wu、Zhihao Zhang、Mingqi Wu、Zhiheng Xi、Mingxu Chai、Tao Liang、Zhihui Fei、Zhen Wang、Mingyang Wan、Guojun Ma、Tao Gui、Qi Zhang、Xuanjing Huang

计算技术、计算机技术

Ming Zhang,Yujiong Shen,Jingyi Deng,Yuhui Wang,Yue Zhang,Junzhe Wang,Shichun Liu,Shihan Dou,Huayu Sha,Qiyuan Peng,Changhao Jiang,Jingqi Tong,Yilong Wu,Zhihao Zhang,Mingqi Wu,Zhiheng Xi,Mingxu Chai,Tao Liang,Zhihui Fei,Zhen Wang,Mingyang Wan,Guojun Ma,Tao Gui,Qi Zhang,Xuanjing Huang.LLMEval-3: A Large-Scale Longitudinal Study on Robust and Fair Evaluation of Large Language Models[EB/OL].(2025-08-12)[2025-08-18].https://arxiv.org/abs/2508.05452.点此复制

评论