多模态深度学习综述

modality refers to the way in which something happens or is experienced, such as word, language, sound, picture and so on. Multimodality is a combination of two or more modalities. Multimodal learning refers to learning the information of each modality in the multimodality, and realizing the exchange and conversion of information of each modality. Thus, Multimodal deep learning is the establishment of a neural network model that can accomplish multimodal learning tasks. The universality of multimodal learning and the intensification of deep learning lead to the vitality of multimodal deep learning. This paper aims to summarize the current multimodal deep learning, find common problems in the implementation of multimodal deep learning under different multimodal and learning objectives, as well as making common problems classify and describing methods for solving various problems at the early development of multimodal deep learning. Specifically, this paper summarizing the current multimodal deep learning that study on natural language, visual, auditory, and considering the research direction such as language translation, event detection, information description, emotion recognition, voice recognition and synthesis, and multimedia retrieval and so on, which further concludes that there are four types of common problems: multimodal representation, multimodal interpretation, multimodal fusion, and multimodal alignment. Meanwhile, each common multimodal learning problem is sub-categorized and discussed, and the neural network models generated for solving the problems are listed. Finally, it introduce some actual multimodal system, list baseline datasets and evaluation criteria used in multimodal deep learning, and conclude with perspectives and directions for future research.