Scaling Spatial Test-Time Training to In-the-Wild Videos via Self-Supervised Depth Estimation
Replace explicit 3D scene-description supervision with self-supervised monocular depth pseudo-labels to provide dense gradient signals for spatial test-time training fast weights.
A very similar idea came first from another paper: Self-Supervised Geometric Distillation for Long-Horizon Data Walls
Time
Compute level
Resources
What the paper used
Prerequisites
Scope
The Gap
The model relies heavily on a custom dense scene-description dataset derived from explicit object-centric 3D scene graphs (ScanNet, ARKitScenes) to provide rich, high-coverage gradient signals for learning the fast-weight dynamics during the first training stage.
“Unlike short-form answers, the target output is formatted as a coherent scene walkthrough, which is derived from object-centric 3D scene graphs (Jia et al., 2024).”
Limitation
The requirement for detailed 3D scene graphs and dense walkthroughs restricts the training data to indoor, pre-mapped environments, limiting the model's generalizability and ability to scale to unconstrained, in-the-wild outdoor videos.
Proposed Solution
Replace or augment the explicit 3D scene-description supervision with a self-supervised auxiliary task, such as monocular depth estimation using pseudo-labels from models like Depth Anything, to provide dense gradient signals for the TTT fast weights without requiring manual 3D annotations.
Why This Matters
Removing the bottleneck of expensive, domain-specific 3D annotations allows the spatial memory capabilities to scale to massive, diverse video datasets, dramatically improving real-world generalization.
How to Get Started
How to test it: Hypothesis: Stage-1 supervision using monocular depth pseudo-labels (e.g., from Depth Anything) on unconstrained videos teaches fast-weight update dynamics without 3D scene-graph walkthroughs. Compare depth-supervised Stage-1 + Stage-2 Spatial-TTT-2B against the original Spatial-TTT-2B (64.4 Avg.) and w/o Dense Data ablation (61.3 Avg.) on VSI-Bench and an outdoor/in-the-wild spatial video split; stop if it fails to beat the w/o Dense Data baseline (61.3 Avg.) on VSI-Bench or shows no gain on outdoor spatial reasoning.
- Generate pseudo-labels for in-the-wild video datasets using off-the-shelf monocular depth estimators like Depth Anything.
- Adapt the Spatial-TTT training objective to accept self-supervised depth pseudo-labels in place of 3D scene graph annotations.
- Train the fast-weight dynamics on unannotated video sequences using the modified self-supervised loss function.
- Evaluate generalization performance on diverse in-the-wild video benchmarks compared to models trained exclusively on ScanNet or ARKitScenes.
Code linked to this paper on Hugging Face: github.com/THU-SI/Spatial-TTT
Suggested by AI. Adapt it to your setup.
Has This Been Done?
Not checked against the literature yet.
Source Paper
Spatial-TTT: Streaming Visual-based Spatial Intelligence with Test-Time TrainingFangfu Liu, Diankun Wu, Jiawei Chi +7 more · 2026
Spatial-TTT introduces a novel framework for streaming visual-based spatial intelligence in Multimodal Large Language Models (MLLMs) by utilizing Test-Time Training (TTT). The authors' main contribution is enabling MLLMs to selectively maintain, progressively update, and reason over 3D spatial evidence from potentially unbounded long-horizon video streams.
Related Ideas
- Extending Spatial Curation to Dynamic Scenes via 4D RepresentationHoli-Spatial: Evolving Video Streams into Holistic 3D Spatial Intelligence🔴Hard
- Extending Decoupled Semantic-Detail Representations to Video ModalitiesCheers: Decoupling Patch Details from Semantic Representations Enables Unified Multimodal Comprehension and Generation⛔Very hard
- Adaptive Early-Stopping and Feed-Forward Alternatives for 3DGS OptimizationHoli-Spatial: Evolving Video Streams into Holistic 3D Spatial Intelligence🟠Medium
- Natural Predictive Pre-training on Web-Scale VideoIn Pursuit of Pixel Supervision for Visual Pre-training⛔Very hard
- Fully Automated Metric Depth Estimation for Dynamic Small ObjectsStepping VLMs onto the Court: Benchmarking Spatial Intelligence in Sports🔴Hard