Adversarial Hypothesis Testing for Visual Grounding
Generate competing visual grounding hypotheses during tool use to prevent early fixation and robustly select evidence-consistent paths.
Source paper
Xiaoxi Li, Wenxiang Jiao, Jiarui Jin, Shijian Wang, Guanting Dong, Jiajie Jin, Hao Wang, Yinuo Wang, Ji-Rong Wen, Yuan Lu, Zhicheng Dou
Generated by AI. The paper’s authors did not write these.
Generate competing visual grounding hypotheses during tool use to prevent early fixation and robustly select evidence-consistent paths.
Route visual verification tasks through a high-fidelity object-aligned encoder using gated cross-attention to reduce perception errors in open-source agents.
Optimize tool-use efficiency by applying online reinforcement learning directly inside the benchmark environment with pass-fail rewards and step penalties.
This paper introduces OmniGAIA, a benchmark designed to evaluate native omni-modal AI agents on tasks requiring multi-hop reasoning and tool use across video, audio, and image modalities. Additionally, it proposes OmniAtlas, a foundation agent trained via a novel pipeline involving hindsight-guided tree exploration, trajectory-level supervised fine-tuning with masked supervision, and OmniDPO for fine-grained error correction.
The authors construct an 'Omni-modal Event Graph' by extracting fine-grained signals from raw media using foundation models, which is then expanded with external tools and fuzzified to generate complex QA pairs. The OmniAtlas agent is trained using a 'Tool-Integrated Reasoning' paradigm that allows it to autonomously interleave internal reasoning with external tool calls (including active perception tools like `read_video`), optimized via Direct Preference Optimization (DPO) on synthesized trajectories.