Instruction-Guided Iterative Editing Module
Enable iterative vector graphic editing by fine-tuning a model on a synthetic dataset of original animations, edit instructions, and modified outputs.
Time
Compute level
Resources
Prerequisites
Scope
The Gap
While the introduction claims Lottie is 'editing-friendly,' Section 4.2 and the experiments focus entirely on generating animations from scratch. There is no mechanism to refine existing outputs (e.g., 'Make the red circle blue') without regenerating the entire sequence, which discards the user's preferred structural elements.
“Future work could explore constrained decoding, reinforcement learning with renderability rewards, or agentic integration with professional tools like After Effects to improve reliability and practical applicability.”
Limitation
The current autoregressive architecture is one-shot; it lacks the capability to accept an existing Lottie file and an edit instruction to perform localized updates.
Proposed Solution
Develop a synthetic 'edit instruction' dataset by programmatically perturbing the MMLottie-2M data (e.g., shifting hue, scaling duration). Fine-tune the model to accept a concatenated input of [Original Lottie Tokens, Edit Prompt] and output [Edited Lottie Tokens], enabling iterative refinement similar to InstructPix2Pix but for vector data.
The paper’s authors list this direction as future work, so they may already be pursuing it.
Why This Matters
This transforms the tool from a novelty generator into a practical workflow asset, allowing designers to tweak results rather than relying on random regeneration.
How to Get Started
How to test it: Hypothesis: Fine-tuning on synthetic edit instruction pairs enables robust localized Lottie updates. Baseline: One-shot generation from scratch given the edit prompt. Metric: CLIP similarity and structural preservation score on edited regions. Stopping condition: Superior edit preservation compared to full regeneration.
- Programmatically generate a synthetic edit dataset from the MMLottie-2M corpus by applying parameter perturbations and corresponding textual instructions.
- Format the training data to accept concatenated input sequences containing both the original Lottie tokens and the edit prompt.
- Fine-tune an open-source sequence generation model on the synthetic edit dataset using a standard GPU setup.
- Evaluate the editing model on qualitative user instructions to verify structural preservation and target modifications.
Code linked to this paper on Hugging Face: github.com/OpenVGLab/OmniLottie
Suggested by AI. Adapt it to your setup.
Has This Been Done?
Works such as SVGenius explore LLMs in SVG editing and generation, while others like RefVideo-6M and SEED-Data-Edit focus on instructional editing datasets for images or video. However, none of the listed papers specifically implement programmatic perturbation of vector tokens on datasets like MMLottie-2M for instruction-guided Lottie animation editing.
- SVGenius: Benchmarking LLMs in SVG Understanding, Editing and Generation · 2025 · 29 citations
- SEED-Data-Edit Technical Report: A Hybrid Dataset for Instructional Image Editing · 2024 · 121 citations
We searched OpenAlex for “Enable iterative vector graphic editing by fine-tuning a model on a synthetic dataset of original animations, edit instructions, and modified outputs.” and “Instruction-Guided Iterative Editing Module” and an AI compared the idea with the top 10 results. A small search can miss relevant work.
Source Paper
OmniLottie: Generating Vector Animations via Parameterized Lottie TokensYiying Yang, Wei Cheng, Sijin Chen +5 more · 2026
This paper presents OmniLottie, an end-to-end framework leveraging a multimodal Large Language Model (Qwen2.5-VL) to generate Lottie vector animations from text, images, or video instructions. The authors introduce a specialized "Lottie Tokenizer" to compress complex JSON structures into efficient command sequences and release MMLottie-2M, a large-scale dataset of vector animations, to train the model for high visual fidelity and semantic alignment.
Related Ideas
- Developing an Open-Source, Fine-Grained Automated Evaluator for Instruction-Based Image EditingFireRed-Image-Edit-1.0 Techinical Report🟡Fairly easy
- Native Video-Based Instructive Editing PipelineOmniVCus: Feedforward Subject-driven Video Customization with Multimodal Control Conditions🔴Hard
- Vector-Based Editable Diagram Generation via GUI AgentsPaperBanana: Automating Academic Illustration for AI Scientists🔴Hard
- Infinite-Context Dual-Autoregressive Architecture for Unbounded Audiobook SynthesisFish Audio S2 Technical Report🔴Hard
- Synthetic Trajectory Distillation for Specialized Open-Weight ModelsDeepCode: Open Agentic Coding🟠Medium