Vector-Based Editable Diagram Generation via GUI Agents
Drive professional vector software via GUI agents to generate editable SVG and PDF diagrams directly from textual style plans.
Time
Compute level
Resources
Prerequisites
Scope
The Gap
The current `Visualizer` agent generates raster images (pixels), specifically at 4K resolution to mitigate blurriness. However, as noted in the Limitations (Section 9), this fails to meet the academic standard for vector graphics (SVG/PDF), making post-generation editing of specific text or nodes impossible without regeneration.
Limitation
The inability to produce editable vector graphics limits the usability of the tool for final manuscript submission, where manual tweaking is often required.
Proposed Solution
Replace the raster generation module with a GUI Agent capable of driving professional vector software (e.g., Adobe Illustrator or Inkscape) or fine-tune a model to output robust TikZ/SVG code directly. This would involve training the agent to map the 'Stylist's' textual plan into sequences of API calls or vector primitives rather than pixel diffusion.
Why This Matters
This would transform the tool from a prototyping utility into a production-grade asset, allowing researchers to tweak labels or move nodes in the final output, a strict requirement for many top-tier conferences.
How to Get Started
- Set up a headless environment with vector graphics software such as Inkscape or access to a TikZ code generation pipeline.
- Construct a dataset of textual style plans paired with corresponding ground-truth SVG or TikZ code files.
- Fine-tune an open-source Vision-Language Model or GUI agent to translate stylist plans into sequences of vector API calls.
- Evaluate the generated vector files for editability, rendering correctness, and visual appeal against raster image baselines.
Code linked to this paper on Hugging Face: github.com/dwzhu-pku/PaperBanana
Suggested by AI. Adapt it to your setup.
Has This Been Done?
From Pixels to Paths: A Multi-Agent Framework for Editable Scientific Illustration explores using multi-agent setups for editable vector-based illustrations. Other works like OS-Helix and Agent Banana explore general GUI interactions and image editing with agentic tooling, but do not fully solve specific vector software integration or direct TikZ/SVG fine-tuning pipelines.
- From Pixels to Paths: A Multi-Agent Framework for Editable Scientific Illustration · 2025 · 6 citations
- OS-Helix: Unifying GUI Interaction and Code Execution in Native Computer-Using Agents · 2026 · 0 citations
- Agent Banana: High-Fidelity Image Editing with Agentic Thinking and Tooling · 2026 · 27 citations
We searched OpenAlex for “Drive professional vector software via GUI agents to generate editable SVG and PDF diagrams directly from textual style plans.” and “Vector-Based Editable Diagram Generation via GUI Agents” and an AI compared the idea with the top 10 results. A small search can miss relevant work.
Source Paper
PaperBanana: Automating Academic Illustration for AI ScientistsDawei Zhu, Rui Meng, Yale Song +4 more · 2026
This paper introduces PaperBanana, an agentic framework designed to automate the generation of publication-ready academic illustrations, specifically methodology diagrams and statistical plots. The system orchestrates five specialized agents (Retriever, Planner, Stylist, Visualizer, and Critic) powered by Large Language Models and image generation models to convert textual methodology descriptions into high-fidelity, aesthetically compliant figures.
Related Ideas
- Democratizing Scientific Illustration: Fine-Tuning Open-Weight Models for Spatial Reasoning and SVG GenerationAutoFigure-Edit: Generating Editable Scientific Illustration🔴Hard
- Cross-Modal Verification and Layout Optimization for Autonomous Scientific DiagrammingPaperOrchestra: A Multi-Agent Framework for Automated AI Research Paper Writing🟠Medium
- Visual Feedback Loop for Frontend DebuggingOpenDevin: An Open Platform for AI Software Developers as Generalist Agents🟠Medium
- Instruction-Guided Iterative Editing ModuleOmniLottie: Generating Vector Animations via Parameterized Lottie Tokens🟡Fairly easy
- Scaling Physics Data Synthesis via Rasterized 3D Simulation EnginesInternVL-U: Democratizing Unified Multimodal Models for Understanding, Reasoning, Generation and Editing🔴Hard