OmniVinci: Enhancing Architecture and Data for Omni-Modal Understanding LLM
NVIDIA · NVIDIA Research · University of Hong Kong · Kyoto University · CMU, Carnegie Mellon University · University of California San Diego · Massachusetts Institute of Technology · Department of Computer Science, Princeton University · NVIDIA Corporation · Nvidia · UC San Diego · Salesforce AI Research · Walmart Labs · NVidia · National Taiwan University & NVIDIA · MIT / NVIDIA
PDF 由论文原始站点提供，PaperCompass 不保存论文文件。
摘要
Advancing machine intelligence requires developing the ability to perceive across multiple modalities, much as humans sense the world. We introduce OmniVinci, an initiative to build a strong, open-source, omni-modal LLM. We carefully study the design choices across model architecture and data curation. For model architecture, we present three key innovations: (i) OmniAlignNet for strengthening alignment between vision and audio embeddings in a shared omni-modal latent space; (ii) Temporal Embedding Grouping for capturing relative temporal alignment between vision and audio signals; and (iii) Constrained Rotary Time Embedding for encoding absolute temporal information in omni-modal embeddings. We introduce a curation and synthesis pipeline that generates 24M single-modal and omni-modal conversations. We find that modalities reinforce one another in both perception and reasoning. Our model, OmniVinci, improves over Qwen2.5-Omni with +19.05 on DailyOmni (cross-modal understanding), +1.7 on MMAR (audio), and +3.9 on Video-MME (vision), while using just 0.2T training tokens — a 6× reduction compared to Qwen2.5-Omni’s 1.2T. We finally demonstrate omni-modal advantages in downstream applications spanning robotics, medical AI, and smart factory.