Robustness of Multimodal Large Language Models in Facial Expression Recognition: A Human-Comparative Study
Robustness of Multimodal Large Language Models in Facial Expression Recognition: A Human-Comparative Study
王甦菁 1李婧婷 1赵琳 1孙梦依 1马骏驰 2付双杰3
作者信息
- 1. 中国科学院大学心理系;中国科学院心理研究所, 认知科学和心理健康全国重点实验室
- 2. 江苏科技大学计算机学院;中国科学院心理研究所, 认知科学和心理健康全国重点实验室
- 3. 中国科学院大学计算机科学与技术学院;中国科学院计算技术研究所，智能算法安全全国重点实验室
- 折叠
摘要
多模态大语言模型（MLLMs）在标准面部表情识别（FER）基准测试中已取得优异表现。然而，在典型表情识别中获得较高准确率，并不一定意味着模型在面部信息存在冲突或不完整时仍具备鲁棒性。本研究考察了在语义冲突和信息匮乏两种受控挑战下，MLLMs与人类反应模式的一致性。我们提出了一种受生物学启发的双任务评估框架，将人类观察者与当前最先进的（SOTA）MLLMs进行比较。首先，利用复合面孔范式，考察模型面对冲突情绪线索时的鲁棒性，以检验其对语义干扰的易感性。其次，采用眼部刺激材料，评估模型仅依据局部特征（仅眼睛、眼睛与眉毛）推断基本情绪的能力，并将其与基于完整面部信息的表现进行比较。结果表明，MLLMs对情绪不一致复合面孔的识别准确率高于情绪一致复合面孔，这一模式与人类相反；随着可见面部信息受到限制，模型准确率的下降幅度也大于人类。在匹配的动态呈现条件及不同提示词模板下，主要的面部区域效应模式得到了复现，但其具体幅度因模型和提示词而异。此外，人类在群体层面的校准误差低于所评估的MLLMs。我们提出将该框架作为一套基准评估工具，用于对下一代情感计算系统的感知鲁棒性与认知对齐进行压力测试。
Abstract
Multimodal Large Language Models (MLLMs) have achieved strong performance on standard facial expressionrecognition (FER) benchmarks. However, high accuracy on canonical expressions does not necessarily indicaterobustness when facial information is conflicting or incomplete. This study investigates the alignment betweenMLLMs and human response patterns under two controlled challenges: semantic conflict and informationscarcity. We introduce a bio-inspired dual-task evaluation framework comparing human observers with stateof-the-art (SOTA) MLLMs. First, utilizing the Composite Face paradigm, we examine model robustness againstconflicting emotional cues to test susceptibility to semantic interference. Second, employing ocular stimuli,we evaluate the models capacity to infer basic emotions from isolated local features (isolated eyes, eyes andeyebrows) versus the full facial information. MLLMs showed higher accuracy for incongruent than congruentcomposite faces, opposite to the human pattern, and their accuracy declined more sharply than human accuracyas visible facial information was restricted. The main regional pattern was reproduced under matched dynamicpresentation and alternative prompt templates, although its exact magnitude varied across models and prompts.Humans also showed lower group-level calibration error than the evaluated MLLMs. We propose this frameworkas a benchmark-style evaluation suite for stress-testing perceptual robustness and cognitive alignment innext-generation affective computing systems.关键词
面部表情识别/多模态大语言模型/认知对齐/鲁棒性/整体加工/语义冲突Key words
Facial expression recognition/Multimodal large language models/Cognitive alignment/Robustness/Holistic processing/Semantic conflict引用本文复制引用
王甦菁,李婧婷,赵琳,孙梦依,马骏驰,付双杰.Robustness of Multimodal Large Language Models in Facial Expression Recognition: A Human-Comparative Study[EB/OL].(2026-11-02)[2026-11-05].https://chinaxiv.org/abs/202611.00052.学科分类
计算技术、计算机技术
