基于跨模态特征解耦的多模态基础模型去偏见方法

Machine learning models have been proven to inherit biases from their training datasets, especially the multimodal foundation models trained on large-scale, unprocessed datasets crawled from the In-ternet. These biases will be amplified and propagated to downstream tasks after fine-tuning. This study proposes a general method for debiasing multimodal foundation models by decoupling the bi-as attribute features and the target attribute features in the multimodal embedding through orthogo-nal projection during the fine-tuning process. This enables the model to learn a fair representation that does not contain bias information. Moreover, the fairness research on multimodal foundation models often involves multiple bias attributes. We demonstrate the effectiveness of our method, which can decouple multiplDebiasing Method for Multimodal Foundation Models Based on Cross-Modal Feature Decouplinge bias attributes simultaneously in a single fine-tuning with only a small amount of computing resources.