多模态数据的知识增强深度语义学习方法

he main purpose of image-text retrieval is to facilitate the retrieval of text by image and vice versa. However, these data often show the characteristics of multi-mode, multi-source heterogeneous, including text, image, audio, video and other different forms of information, so to effectively manage and use these multi-mode data, need to use the knowledge graph, graph neural network, contrast learning, adversarial learning and multi-modal large model and other emerging technologies. Therefore, this paper proposes a deep semantic learning method for cross-modal education big data with multi-modal knowledge enhancement. In this study, multi-modal knowledge graph is applied to cross-modal retrieval task. The unity of image and text representation is realized, and the contrast learning loss function within and between modes is added. These measures are helpful for representation learning of multimodal data in a unified semantic space, thus improving the accuracy and efficiency of retrieval. Experimental results show that the proposed method achieves remarkable performance improvement in multi-modal retrieval tasks, which verifies its effectiveness and practicability.