Measuring the Gap Between Human and LLM Research Ideas
Ziyu Chen Yilun Zhao Arman Cohan
作者信息
Abstract
LLMs are increasingly used to brainstorm research ideas, but existing evaluations mostly judge individual ideas by novelty, feasibility, or expert preference. We instead ask: how far are current LLM-generated ideas from human researchers? To characterize this gap, we build a large-scale evaluation framework for ideation from high-quality human research papers. For each paper, we reverse-engineer a small set of closely related prior works that likely inspired its core idea. LLMs are then prompted to generate a new idea from the set of paper titles and summaries. We introduce a two-axis research-taste taxonomy to profile each idea by its opportunity pattern and research paradigm, and use it to quantify the divergence between human and LLM ideas. Across idea sets generated by different LLMs, we observe a consistent distributional gap: LLM ideas are disproportionately concentrated around bridge-like opportunities and synthesis methods, whereas the human paper reference distribution spreads more broadly across ways of framing gaps and constructing contributions. This result suggests that strong LLMs can produce a range of reasonable ideas, but that range remains narrower than, and systematically shifted relative to, human research taste.引用本文复制引用
Ziyu Chen,Yilun Zhao,Arman Cohan.Measuring the Gap Between Human and LLM Research Ideas[EB/OL].(2026-07-01)[2026-07-04].https://arxiv.org/abs/2607.01233.学科分类
计算技术、计算机技术
|国家预印本平台