Contrastive Modality-Aware Re-ranking to Mitigate Text Bias
Train a contrastive cross-encoder to re-rank retrieved text and image candidates based on modal necessity and resolve conflicts caused by text-centric retrieval bias.
Source paper
Zirui Guo, Xubin Ren, Lingrui Xu, Jiahao Zhang, Chao Huang
Generated by AI. The paper’s authors did not write these.
Train a contrastive cross-encoder to re-rank retrieved text and image candidates based on modal necessity and resolve conflicts caused by text-centric retrieval bias.
Fine-tune a vision-language model to directly output graph adjacency matrices from raw image crops, bypassing fragile rule-based table parsers and text conversions.
Develop a lazy indexing strategy that trains a lightweight distillation model to defer expensive vision-language model inference until query time for top-ranked visual candidates.
This paper introduces RAG-Anything, a multimodal Retrieval-Augmented Generation framework designed to treat non-textual elements (images, tables, equations) as first-class knowledge entities alongside text. It overcomes the limitations of text-centric RAG systems by constructing a unified dual-graph (cross-modal and text-based) that preserves structural relationships and visual semantics, enabling precise reasoning over heterogeneous documents.
The framework employs a Dual-Graph Construction strategy: a 'Cross-Modal Knowledge Graph' links visual/tabular elements via structural relationships (e.g., belongs_to) and VLM-generated descriptions, while a 'Text-Based Knowledge Graph' captures semantic entity relations. Retrieval is performed using a hybrid approach combining 'Structural Knowledge Navigation' (following graph edges) and 'Semantic Similarity Matching' (dense vector search), followed by multi-signal fusion scoring to rank evidence for a VLM-based answer generator.