Evaluation Method of Embedded RAG in Scientific Literature Discovery Systems
陈阳 1孟旭阳 1陈熙1
作者信息
1. 中国科学技术信息研究所，北京 100038
折叠
摘要
检索增强生成（Retrieval-Augmented Generation，RAG）技术已广泛应用于科技文献发现系统。但当前发现系统测评方法仍存在人工成本高，难以适配RAG架构特性等问题，因此，本研究提出一种内嵌RAG的科技文献发现系统测评方法，从检索质量、生成质量和溯源质量三个维度构建测评指标体系。首先利用自动化测评获得各指标的量化结果，再采用融合大模型智能评分与人工核对的混合评估方法进行评价，最后通过斯皮尔曼、肯德尔相关性检验验证指标量化结果与混合评估结果的一致性。本研究以国家科技图书文献中心（National Science and Technology Library，NSTL）发现系统为实证对象进行验证。研究结果表明，本研究提出的测评方法能有效识别系统在科技文献检索、生成和溯源方面的差异，且自动化测评结果与混合评估结果呈显著正相关，验证了本研究所构建指标体系的可靠性与有效性，为科技文献发现系统测评与迭代升级提供了实践支撑。
Abstract
Retrieval-Augmented Generation (RAG) technology has been widely applied to scientific literature discovery systems. However, current evaluation methods for these systems are still costly and difficult to adapt to the RAG architecture. This study proposes an evaluation method for embedded RAG in scientific literature discovery systems, constructing an evaluation index system from three dimensions: retrieval quality, generation quality, and traceability quality. The study automates the evaluation to obtain the quantitative results of each index and uses a hybrid evaluation method combining intelligent scoring by a large model and manual verification. The study verifies the consistency of the quantitative results with the hybrid evaluation results through Spearman and Kendall correlation tests. This study validates the reliability and effectiveness of the index system by using the National Science and Technology Library (NSTL) discovery system as a case study, providing practical support for the evaluation and iterative upgrade of scientific literature discovery systems.
关键词
科技文献/检索增强生成/指标体系/自动化测评/NSTL
Key words
scientific literature/ retrieval-augmented generation/ index system/ automated evaluation/ NSTL
