Skip to content
AI-generated research idea🟠Medium💡Moderate InnovationRelated work found

Enhancing Abstract Visual Logic via Synthetically Augmented Multi-Step Math SFT

Enhance abstract multi-step visual reasoning by synthetically generating logic-heavy visual puzzles and fine-tuning models with process reward models.

Time

3-5 days

Compute level

Large cluster

Resources

A cluster of multiple A100/H100 GPUs to fine-tune the 8B VLM on augmented math datasets.

What the paper used

Penguin-VL uses 2B and 8B model variants trained on a large-scale pretraining dataset of 121M samples and 39M SFT image samples.

Prerequisites

Data curation, multi-modal fine-tuning, Chain-of-Thought prompting

Scope

large-scale data collection and modify existing code

The Gap

During the benchmark evaluations in Section 4.3.1, both the Penguin-VL 8B and 2B models underperformed relative to Qwen3-VL and InternVL-3.5 on abstract multi-step reasoning tasks such as MathVerse and LogicVista.

“On MathVerse and LogicVista, Penguin-8B is outperformed by Qwen3-VL and InternVL-3.5, which suggests that while the model has excellent perceptual grounding for math, it may still be refining the deeper logical chains required for advanced visual puzzles.”
From the paper, Page 19. Matched against the PDF text.

Limitation

The model's reasoning capabilities are mildly constrained by a limited proportion of math-focused Supervised Fine-Tuning (SFT) data, causing it to struggle with forming the deep logical chains required for complex, visually grounded mathematical puzzles.

Proposed Solution

Implement a targeted data synthesis pipeline using stronger teacher models to generate multi-step, logic-heavy visual puzzles and procedural math charts. Fine-tune the model using Process Reward Models (PRMs) or visual Chain-of-Thought (CoT) distillation specifically specialized for multi-hop abstract visual logic.

Why This Matters

Closing this gap will allow the model to excel in highly rigorous academic and engineering tasks, making it a fully well-rounded open-source model capable of advanced structural logic.

How to Get Started

How to test it: We hypothesize that augmenting SFT data with synthetically generated multi-step visual logic problems will improve performance on MathVerse and LogicVista compared to the baseline. We measure accuracy on MathVerse and LogicVista; if accuracy does not improve after fine-tuning, we stop.

  1. Set up a data synthesis pipeline using a strong teacher model to generate multi-step visual logic puzzles and procedural math charts.
  2. Construct a fine-tuning dataset incorporating visual Chain-of-Thought reasoning steps.
  3. Train or adapt a Process Reward Model to evaluate intermediate reasoning steps in visual contexts.
  4. Fine-tune Penguin-VL on the synthesized dataset and evaluate performance on MathVerse and LogicVista benchmarks.

Code linked to this paper on Hugging Face: github.com/tencent-ailab/Penguin-VL

Suggested by AI. Adapt it to your setup.

Has This Been Done?

Related work found

SD-MAR: Multi-image Analytical Reasoning via Synthetic Data and Reinforcement Learning and Unlocking Multimodal Mathematical Reasoning via Process Reward Model explore synthetic data and process supervision for multimodal and mathematical reasoning, but they do not specifically target abstract multi-step visual logic puzzles with chart-based SFT and visual CoT distillation.

We searched OpenAlex and arXiv for “Enhance abstract multi-step visual reasoning by synthetically generating logic-heavy visual puzzles and fine-tuning models with process reward models.” and “Enhancing Abstract Visual Logic via Synthetically Augmented Multi-Step Math SFT” and an AI compared the idea with the top 10 results. A small search can miss relevant work.

Source Paper

Penguin-VL: Exploring the Efficiency Limits of VLM with LLM-based Vision Encoders

Boqiang Zhang, Lei Ke, Ruihan Yang +5 more · 2026

This paper introduces Penguin-VL, a compact vision-centric multimodal foundation model that challenges the reliance on contrastively pretrained vision encoders like CLIP. By initializing its vision encoder directly from a text-only large language model, Penguin-VL preserves fine-grained visual cues and achieves state-of-the-art performance in dense perception and reasoning among parameter-efficient models.

Related Ideas

Is this a good idea?

Share onXLinkedIn