基于多粒度一致性与图像底层取证的多模态虚假信息检测模型
Multimodal False Information Detection Model Based on Multi-Granularity Consistency and Low-Level Image Forensics
张凌 1齐阳天 2沈晓川3
作者信息
- 1. 武汉科技大学管理学院，武汉 430081;武汉科技大学湖北产业政策与管理研究中心，武汉 430081
- 2. 武汉科技大学管理学院，武汉 430081
- 3. 武汉启云方科技有限公司，武汉，430081
- 折叠
摘要
针对社交媒体多模态虚假信息检测中图文语义不一致、细粒度实体错配和图像篡改痕迹难识别问题，提出MGC-Forensic检测模型。基于Weibo和Twitter数据集，利用CLIP提取图文语义特征，并引入LoRA进行参数微调；结合多粒度一致性模块、实体级图文匹配模块和SRM-CNN图像取证分支，融合语义一致性特征与图像底层取证特征进行真伪分类。模型在Weibo数据集上的准确率为90.1%，F1-Real和F1-Fake分别为90.4%和89.9%；在Twitter数据集上的准确率为83.3%，F1-Real和F1-Fake分别为78.2%和86.5%。消融实验表明，LoRA、多粒度一致性模块以及图像取证分支均能提升模型准确率。模型仍可能将经过压缩、滤镜等无害处理的真实图像误判为虚假线索，对事实相关性弱但情绪或场景相近的图文样本识别能力仍有限。多粒度图文一致性建模与图像底层取证特征具有互补作用，能够提升多模态虚假信息检测效果。
Abstract
To address text-image semantic inconsistency, fine-grained entity mismatch, and the difficulty of identifying image manipulation traces in social media multimodal false information detection, this study proposes the MGC-Forensic detection model. Based on the Weibo and Twitter datasets, CLIP is used to extract text-image semantic features, and LoRA is introduced for parameter fine-tuning. A multi-granularity consistency module, an entity-level text-image matching module, and an SRM-CNN image forensic branch are further integrated to fuse semantic consistency features and low-level image forensic features for authenticity classification. The model achieves an accuracy of 90.1% on the Weibo dataset, with F1-Real and F1-Fake reaching 90.4% and 89.9%, respectively. On the Twitter dataset, it achieves an accuracy of 83.3%, with F1-Real and F1-Fake reaching 78.2% and 86.5%, respectively. Ablation experiments show that LoRA, the multi-granularity consistency module, and the image forensic branch all improve model accuracy. The model may still misclassify authentic images that have undergone harmless processing, such as compression or filtering, as false cues, and its ability to identify text-image samples with weak factual relevance but similar sentiment or scenes remains limited. Multi-granularity text-image consistency modeling and low-level image forensic features are complementary and can improve the performance of multimodal false information detection.关键词
多模态虚假信息检测/多粒度一致性/SRM/CLIP/LoRAKey words
multimodal fake information detection/multi-granularity consistency/SRM/CLIP/LoRA引用本文复制引用
张凌,齐阳天,沈晓川.基于多粒度一致性与图像底层取证的多模态虚假信息检测模型[EB/OL].(2026-07-27)[2026-07-30].http://www.paper.edu.cn/releasepaper/content/202607-31.学科分类
计算技术、计算机技术
