Skip to content
AI-generated research idea🟡Fairly easy💡Moderate InnovationRelated work found

Uncertainty-Aware Verification for VLM-Driven Semantic Labeling

A system that cross-verifies foundational vision-language model labels with geometric features and uncertainty estimators to reject low-confidence semantic tags.

Time

4-6 days

Compute level

Large cluster

Resources

8-16 NVIDIA H800 GPUs for running VLM inference ensembles and evaluating semantic labeling pipelines.

What the paper used

1.2M 3D grounding pairs and 1.25M spatial QA pairs

Prerequisites

basic ML knowledge

Scope

modify existing code

The Gap

During Image-level Perception, the system queries a foundational VLM to sequentially generate open-vocabulary categories and maintain a dynamic class-label memory across frames.

“Open-vocabulary semantic labeling may also inherit biases or errors from foundation models, making robust verification and uncertainty estimation important future directions.”
From the paper, Impact Statement, page 13. Matched against the PDF text.

Limitation

The pipeline is highly susceptible to inheriting semantic biases, hallucinations, or misclassifications from the foundational VLM, which then propagate irreversibly into the final 3D bounding boxes and QA pairs.

Proposed Solution

Introduce an uncertainty estimation module during the VLM labeling phase. This involves prompting the VLM for log-probabilities or using an ensemble of different VLMs to vote on labels, cross-verifying them against geometric features to reject low-confidence semantic tags.

The paper’s authors list this direction as future work, so they may already be pursuing it.

Why This Matters

Improves the reliability and safety of the curated datasets by actively filtering out foundation model hallucinations, which is absolutely critical for training trustworthy downstream embodied AI models.

How to Get Started

How to test it: We hypothesize that introducing an uncertainty estimation module during VLM labeling will reduce semantic classification errors. We will compare against baseline VLM labeling without uncertainty filtering, measuring 3D bounding box semantic accuracy (AP50). The experiment will stop if classification error reduction is less than 5% relative improvement.

  1. Integrate log-probability extraction or multi-VLM voting mechanisms into the image-level perception querying pipeline.
  2. Extract geometric features and spatial point clouds corresponding to the targeted open-vocabulary category regions.
  3. Design a verification heuristic that combines VLM uncertainty scores with geometric consistency checks.
  4. Benchmark the filtered semantic labeling accuracy against standard open-vocabulary segmentation baselines.

Code linked to this paper on Hugging Face: github.com/Visionary-Laboratory/holi-spatial

Suggested by AI. Adapt it to your setup.

Has This Been Done?

Related work found

Works like VISA: VLM-Guided Instance Semantic Auditing for 3D Occupancy World Models touch upon auditing and verifying VLM-driven semantic labels within 3D spaces. However, the listed results leave open the specific use of VLM log-probabilistic ensembles cross-verified against geometric features for confidence rejection.

We searched OpenAlex and arXiv for “A system that cross-verifies foundational vision-language model labels with geometric features and uncertainty estimators to reject low-confidence semantic tags.” and “Uncertainty-Aware Verification for VLM-Driven Semantic Labeling” and an AI compared the idea with the top 10 results. A small search can miss relevant work.

Source Paper

Holi-Spatial: Evolving Video Streams into Holistic 3D Spatial Intelligence

Yuanyuan Gao, Hao Li, Yifei Liu +14 more · 2026

This paper introduces Holi-Spatial, a fully automated pipeline designed to convert raw video streams into high-fidelity 3D spatial annotations without human intervention. The primary contribution is the creation of Holi-Spatial-4M, a massive, spatially-aware multimodal dataset that significantly enhances the spatial reasoning and 3D grounding capabilities of Vision-Language Models.

Related Ideas

Is this a good idea?

Share onXLinkedIn