laude 3 Opus和Kimi在生物医学期刊同行评议中的能力评价

Purposes] This study aimed to evaluate the capabilities of Claude 3 Opus and Kimi in peer review for biomedical journals. [Methods] Peer reviews were conducted using Claude 3 Opus and Kimi on 29 papers from China Oncology that were at the final review stage before publication, and were examined according to the disclosure checklist in the Biomedical Research Reporting Guidelines. All authors declared their consent to use AIGC for peer review. The results of the AIGC peer reviews were scored using a 5-point Likert scale. Count data were analyzed using one-way ANOVA or Fishers exact probability test, and multiple paired measurement data were analyzed using the Friedman M test. Sensitivity, specificity, positive predictive value, negative predictive value, and accuracy of the two types of AIGC assessments were calculated using a four-fold table method, and ROC curves were plotted and assess their predictive capability. [Findings] Of the 29 articles, after the final review meeting, 6 were published, 15 were published after revisions, and 8 were rejected. Claude 3 Opuss peer review conclusions led to the publication of 19 articles and the publication of 10 after revisions. Kimis peer review conclusions resulted in the publication of 9 articles, 16 after revisions, and 4 rejections. The Friedman M test showed no statistically significant difference between Kimis peer review conclusions and the experts conclusions at the final review meeting (M=0.241, adjusted P=1.000). The Likert scale results indicated that the experts at the final review meeting had a higher level of agreement with Kimis peer review results than with those of Claude 3 Opus (3.850.47 vs 3.480.73, F=10.017, P=0.002). In terms of reviewing according to the Biomedical Research Reporting Guidelines disclosure checklist, Claude 3 Opuss evaluation accuracy was 77.5%, sensitivity was 76.9%, and specificity was 64.0%; Kimis evaluation accuracy was 75.2%, sensitivity was 77.5%, specificity was 70.1%. ROC curve analysis showed that the areas under the curve for Claude 3 Opus and Kimi were 0.818 and 0.841, respectively, indicating good predictive capability. Upon testing, there was no statistically significant difference between Kimis review results and the editors review results (M=-0.152, adjusted P=0.061). [Conclusions] Claude 3 Opus and Kimi have shown good capabilities in reviewing the disclosure checklist of biomedical research reports, with high consistency compared to editorial reviews. However, these two AIGC models have not yet reached the level of expert peer review, exhibiting issues such as:Inaccuracy in generated content;Lack of personalization in generated content;Insufficient extrapolation of external content;Ambiguity in review content;Overly coarse-grained generated content;Bias towards positive evaluations in review conclusions. Despite these issues, they still demonstrate certain potential for application. To further enhance their effectiveness, it is recommended that medical experts, medical editors, and AIGC developers work together to establish relevant standards, ensure data security and quality, increase transparency, reduce review bias, adhere to publishing ethics, and establish effective supervision and feedback mechanisms to ensure the accuracy of AIGC in peer review for biomedical journals.