Decoupled Text Representation Distillation
Train a lightweight feature distillation adapter to align mobile text encoder representations with a frozen high-capacity teacher model.
Source paper
Abdelrahman Shaker, Ahmed Heakl, Jaseel Muhammad, Ritesh Thawkar, Omkar Thawakar, Senmao Li, Hisham Cholakkal, Ian Reid, Eric P. Xing, Salman Khan, Fahad Shahbaz Khan
Generated by AI. The paper’s authors did not write these.
Train a lightweight feature distillation adapter to align mobile text encoder representations with a frozen high-capacity teacher model.
Introduce a self-consistent training phase that conditions understanding loss on model-generated images rather than ground truth data.
Integrate a lightweight super-resolution module optimized for mobile neural engines to upscale low-resolution latents to high-density display requirements.
This paper introduces Mobile-O, a lightweight unified multimodal model designed for mobile devices that performs both visual understanding (image captioning, VQA) and text-to-image generation. The authors propose a novel 'Mobile Conditioning Projector' (MCP) to efficiently bridge a compact Vision-Language Model (FastVLM) with a diffusion generator (DiT), alongside a three-stage training pipeline utilizing a new 'quadruplet' data format to jointly optimize both tasks with minimal data.
The model architecture couples a FastVLM-0.5B (using FastViT and Qwen2-0.5B) with a SANA-0.6B-based diffusion transformer. The core technical contribution is the Mobile Conditioning Projector (MCP), which uses layerwise feature fusion and depthwise-separable convolutions to project VLM tokens into the diffusion conditioning space. Training involves a specific 'quadruplet' scheme (generation prompt, image, question, answer) to align understanding and generation tasks simultaneously.