MoTrans: Customized Motion Transfer with Text-driven Video Diffusion Models
MoTrans: Customized Motion Transfer with Text-driven Video Diffusion Models
计算技术、计算机技术
Haiwen Diao,Xu Jia,Pengxiang Li,Mengmeng Ge,Xiaomin Li,Huchuan Lu,You He,Qinghe Wang.MoTrans: Customized Motion Transfer with Text-driven Video Diffusion Models[EB/OL].(2024-12-02)[2025-09-18].https://arxiv.org/abs/2412.01343.点此复制
Existing pretrained text-to-video (T2V) models have demonstrated impressive
abilities in generating realistic videos with basic motion or camera movement.
However, these models exhibit significant limitations when generating
intricate, human-centric motions. Current efforts primarily focus on
fine-tuning models on a small set of videos containing a specific motion. They
often fail to effectively decouple motion and the appearance in the limited
reference videos, thereby weakening the modeling capability of motion patterns.
To this end, we propose MoTrans, a customized motion transfer method enabling
video generation of similar motion in new context. Specifically, we introduce a
multimodal large language model (MLLM)-based recaptioner to expand the initial
prompt to focus more on appearance and an appearance injection module to adapt
appearance prior from video frames to the motion modeling process. These
complementary multimodal representations from recaptioned prompt and video
frames promote the modeling of appearance and facilitate the decoupling of
appearance and motion. In addition, we devise a motion-specific embedding for
further enhancing the modeling of the specific motion. Experimental results
demonstrate that our method effectively learns specific motion pattern from
singular or multiple reference videos, performing favorably against existing
methods in customized video generation.
展开英文信息
评论