Skip to content

Source paper

Penguin-VL: Exploring the Efficiency Limits of VLM with LLM-based Vision Encoders

Boqiang Zhang, Lei Ke, Ruihan Yang and 5 more

Boqiang Zhang, Lei Ke, Ruihan Yang, Qi Gao, Tianyuan Qu, Rossell Chen, Dong Yu, Leoweiliang

arXiv 2603.06569 · 202682 upvotes on Hugging Face

Research ideas from this paper (3)

Generated by AI. The paper’s authors did not write these.

AI summary

This paper introduces Penguin-VL, a compact vision-centric multimodal foundation model that challenges the reliance on contrastively pretrained vision encoders like CLIP. By initializing its vision encoder directly from a text-only large language model, Penguin-VL preserves fine-grained visual cues and achieves state-of-the-art performance in dense perception and reasoning among parameter-efficient models.

How it works (AI summary)

The core technique involves adapting a text-only LLM (Qwen3-0.6B) into a bidirectional vision encoder equipped with 2D rotary positional embeddings. Training proceeds through a three-stage pipeline utilizing novel feature reconstruction losses (including a unique relation loss) to preserve spatial cues, alongside Temporal Redundancy-Aware (TRA) token compression for efficient long-video processing.