Can Learning to Answer Spatial Questions Improve Novel-View Generation?
Jahyeok Koo Hyeonseo Yu Seungho Jang Jaewoo Jung Jiwon Kang Seokju Cho Seungryong Kim
作者信息
Abstract
Generating novel views from limited observations requires preserving both the appearance and spatial structure of a scene. Although recent approaches exploit fine-grained geometric cues, they do not explicitly model high-level semantic and spatial properties such as object identity, attributes, and inter-object relationships. In this work, we investigate whether 3D language understanding can provide complementary supervision for novel-view generation. We introduce Ask4NV, built on a pretrained unified multimodal model with a Mixture-of-Transformers (MoT) architecture. Our model incorporates camera geometry into both VAE and ViT representations and leverages understanding features during generation through joint self-attention with a modality-aware attention masking strategy. Experiments demonstrate strong image quality and geometric consistency in novel-view generation, alongside competitive 3D language understanding. Further analysis shows that joint training improves cross-view correspondence in VAE representations and similarity in a 3D-aware feature space.引用本文复制引用
Jahyeok Koo,Hyeonseo Yu,Seungho Jang,Jaewoo Jung,Jiwon Kang,Seokju Cho,Seungryong Kim.Can Learning to Answer Spatial Questions Improve Novel-View Generation?[EB/OL].(2026-10-14)[2026-10-16].https://arxiv.org/abs/2610.20610.学科分类
工程设计、工程测绘
|国家预印本平台