多模态营养知识图谱构建

Diet is a crucial aspect of peoples lives. In recent years, with the improvement of living standards, people have increasingly focused on the health and personalization of their diets. To provide precise, effective, and intuitive nutritional and dietary recommendations for different populations, it is essential to construct a multimodal nutrition knowledge graph. This paper constructs a multimodal nutrition knowledge graph that includes entities such as food, nutrition, and diseases by integrating nutritional data from nutrition books, literature, and websites. By referencing the OneRel model, joint extraction of Chinese entity relationships is completed. The perceptual hash (pHash) algorithm is used to filter the acquired food image data, and the RoBERTa-ResNet model is employed to learn the features of text and image data separately. The text and image feature vectors are concatenated to form a fused vector, and a fully connected layer is added to learn the superficial features between modalities, which aids in the construction of the multimodal knowledge graph. Finally, the Neo4j graph database is utilized to store and visually display the multimodal nutrition knowledge graph. The multimodal nutrition knowledge graph constructed using the cross-modal knowledge graph construction method proposed in this paper not only systematically integrates multimodal knowledge in the field of nutrition and enables good visual query capabilities but also serves as the underlying support for downstream tasks such as intelligent question answering and nutrition recommendation systems.