|国家预印本平台
首页|F-ViTA: Foundation Model Guided Visible to Thermal Translation

F-ViTA: Foundation Model Guided Visible to Thermal Translation

F-ViTA: Foundation Model Guided Visible to Thermal Translation

来源:Arxiv_logoArxiv
英文摘要

Thermal imaging is crucial for scene understanding, particularly in low-light and nighttime conditions. However, collecting large thermal datasets is costly and labor-intensive due to the specialized equipment required for infrared image capture. To address this challenge, researchers have explored visible-to-thermal image translation. Most existing methods rely on Generative Adversarial Networks (GANs) or Diffusion Models (DMs), treating the task as a style transfer problem. As a result, these approaches attempt to learn both the modality distribution shift and underlying physical principles from limited training data. In this paper, we propose F-ViTA, a novel approach that leverages the general world knowledge embedded in foundation models to guide the diffusion process for improved translation. Specifically, we condition an InstructPix2Pix Diffusion Model with zero-shot masks and labels from foundation models such as SAM and Grounded DINO. This allows the model to learn meaningful correlations between scene objects and their thermal signatures in infrared imagery. Extensive experiments on five public datasets demonstrate that F-ViTA outperforms state-of-the-art (SOTA) methods. Furthermore, our model generalizes well to out-of-distribution (OOD) scenarios and can generate Long-Wave Infrared (LWIR), Mid-Wave Infrared (MWIR), and Near-Infrared (NIR) translations from the same visible image. Code: https://github.com/JayParanjape/F-ViTA/tree/master.

Jay N. Paranjape、Celso de Melo、Vishal M. Patel

计算技术、计算机技术高电压技术

Jay N. Paranjape,Celso de Melo,Vishal M. Patel.F-ViTA: Foundation Model Guided Visible to Thermal Translation[EB/OL].(2025-04-03)[2025-07-17].https://arxiv.org/abs/2504.02801.点此复制

评论