Scaling Unified Multimodal Models via Large Pre-trained Vision-Language Initializations
Initialize a unified multimodal model backbone using a large pre-trained vision-language model and adapt it via parameter-efficient fine-tuning.
Source paper
Yichen Zhang, Da Peng, Zonghao Guo, Zijian Zhang, Xuesong Yang, Tong Sun, Shichu Sun, Yidan Zhang, Yanghao Li, Haiyan Zhao, Wang Xu, Qi Shi, Yangang Sun, Chi Chen, Shuo Wang, Yukun Yan, Xu Han, Qiang Ma, Wei Ke, Liang Wang, Zhiyuan Liu, Maosong Sun
Generated by AI. The paper’s authors did not write these.
Initialize a unified multimodal model backbone using a large pre-trained vision-language model and adapt it via parameter-efficient fine-tuning.
A method that extends a unified vision tokenizer to accept spatial-temporal tubes instead of 2D patches, enabling video-text generation models to process sequential video datasets like WebVid.
A method that replaces VAE pixel reconstruction with a specialized latent-to-latent projection network, mapping VAE latents directly into the SigLIP2 feature space for efficient OCR processing.
The paper introduces CHEERS, a unified multimodal model that seamlessly integrates both visual comprehension and high-fidelity image generation within a single transformer framework. Its primary contribution is a novel architecture that decouples patch-level high-frequency visual details from low-frequency semantic representations, stabilizing multimodal reasoning while enhancing image synthesis through adaptive detail injection.
The model employs a unified vision tokenizer combining a VAE and SigLIP2-ViT with 2D token compression to extract semantic features. These features are processed by a 1.5B parameter LLM backbone that uses autoregressive decoding for text and a cascaded flow-matching head for continuous-time image generation, dynamically injecting high-frequency detail residuals during the synthesis process.