Native Multimodal Operator Support via Late Fusion Storage
Extend data storage abstractions to reference binary image blobs directly and introduce multimodal operators that interface with vision-language models for late-fusion pipelines.
Time
Compute level
Resources
Prerequisites
Scope
The Gap
As described in Section 4.3 ('Modality Dimension'), DataFlow processes non-text modalities by first converting them into text (e.g., using MinerU for PDFs/images) before applying downstream transformations. This 'text-first' architecture prevents the creation of true multimodal pipelines, such as those requiring interleaved image-text manipulation (e.g., 'remove images with watermarks' or 'crop image based on caption') without breaking the abstraction.
Limitation
The current framework relies on early conversion of non-text data to text, losing raw signal (pixel data, layout information) required for multimodal data synthesis and refinement tasks.
Proposed Solution
Extend the `DataFlowStorage` abstraction to support referencing binary blobs (tensors/images) alongside tabular text data. Implement a new class of 'MultimodalOperators' that accept tuple inputs (image_tensor, text_prompt) and interface with Vision-Language Models (e.g., CLIP, LLaVA) directly within the pipeline, allowing for late-fusion operations like visual quality filtering or image-text consistency scoring.
Why This Matters
This would transform DataFlow from a text-processing engine into a general-purpose multimodal data factory, essential for training next-generation VLM (Vision-Language Models).
How to Get Started
- Update the core storage class to support tensor and image references alongside tabular text.
- Design a multimodal operator base class capable of accepting tuple inputs consisting of image tensors and text prompts.
- Implement specific operators for visual quality filtering and image-text consistency scoring using pre-trained vision-language models like CLIP or LLaVA.
- Benchmark the new pipeline on multimodal document datasets to verify abstraction integrity.
Code linked to this paper on Hugging Face: github.com/OpenDCAI/DataFlow
Suggested by AI. Adapt it to your setup.
Has This Been Done?
None of the provided results focus on extending dataflow storage and data operator abstractions to integrate native binary blobs and late-fusion multimodal operators with vision-language models for tabular and unstructured pipelines. While papers like SemPipes discuss optimizable semantic data operators and others cover multimodal storage generally, none of them execute the specific proposal of unifying binary blob referencing and late-fusion vision-language operators within a dataflow storage framework.
- SemPipes -- Optimizable Semantic Data Operators for Tabular Machine Learning Pipelines · 2026 · 6 citations
- Multimodal Data Storage and Retrieval for Embodied AI: A Survey · 2025 · 17 citations
- SciForge: An AI-Native, Multimodal Workbench for Scientific Discovery · 2026 · 2 citations
We searched OpenAlex and arXiv for “Extend data storage abstractions to reference binary image blobs directly and introduce multimodal operators that interface with vision-language models for late-fusion pipelines.” and “Native Multimodal Operator Support via Late Fusion Storage” and an AI compared the idea with the top 10 results. A small search can miss relevant work.
Source Paper
DataFlow: An LLM-Driven Framework for Unified Data Preparation and Workflow Automation in the Era of Data-Centric AIHao Liang, Xiaochen Ma, Zhou Liu +32 more · 2025
This paper presents DataFlow, a unified, open-source framework for LLM-driven data preparation that replaces ad-hoc scripts with composable, Python-based operators and pipelines. It features a novel agentic orchestration layer that can synthesize executable data workflows from natural language commands, demonstrating superior performance in generating high-quality synthetic data for math, code, and text tasks compared to existing baselines.
Related Ideas
- Decoupled Asymmetric Modality Branching in Vision-Language TransformersMulti-Branch Transformer🔴Hard
- Extending Decoupled Semantic-Detail Representations to Video ModalitiesCheers: Decoupling Patch Details from Semantic Representations Enables Unified Multimodal Comprehension and Generation⛔Very hard
- Native 1-bit Multimodal ArchitectureBitNet b1.58 2B4T Technical Report🔴Hard
- Dual-Stream Vision Fusion for Concurrent Motion Dynamics and Text/OCR RecognitionV-JEPA-LM: Video-Language Models That Read Dynamics Beyond Static, Text-Aligned Representations🟡Fairly easy
- End-to-End Joint Optimization of MLLM and MMDiTUniVideo: Unified Understanding, Generation, and Editing for Videos🔴Hard