Fisher-Regularized Training for Drift Prevention
Incorporate Fisher Information regularization directly into training to prevent fast weights from drifting and corrupting previously learned geometries.
Time
Compute level
Resources
What the paper used
Prerequisites
Scope
The Gap
Supplementary Section A discusses 'Selective Update of Fast Weights in AR Setting,' noting that naively updating weights with new streaming views causes 'drifting'—where the model corrupts previously reconstructed geometry. The authors experimented with an inference-time heuristic but suggested integrating it into training.
“The fast weight update can also consider historical gradients and fast weights to mitigate drifting”
Limitation
The current autoregressive update lacks a mechanism to protect critical weights associated with previous views, leading to catastrophic interference during streaming reconstruction.
Proposed Solution
Integrate a differentiable Elastic Weight Consolidation (EWC) loss directly into the training loop. Calculate the Fisher Information Matrix for the fast weights during the sequence processing and penalize updates to weights that were crucial for reconstructing earlier frames, rather than relying on post-training heuristics.
The paper’s authors list this direction as future work, so they may already be pursuing it.
Why This Matters
Essential for robust real-time SLAM and robotics applications where the agent must maintain a consistent world model without revisiting old locations.
How to Get Started
How to test it: We hypothesize that integrating an EWC loss with the Fisher Information Matrix will prevent catastrophic forgetting during streaming updates. We will use the standard autoregressive update rule as our baseline, measuring PSNR degradation across long video streams, and halt the experiment if drift is not reduced by at least 15%.
- Formulate the Elastic Weight Consolidation loss function tailored for streaming view updates.
- Modify the training loop to compute the Fisher Information Matrix over fast weights during sequence processing.
- Add the regularization penalty to the optimization objective to restrict updates on critical parameters.
- Benchmark reconstruction drift and preservation of earlier frames against post-processing heuristic baselines.
Code linked to this paper on Hugging Face: github.com/cwchenwang/tttLRM
Suggested by AI. Adapt it to your setup.
Has This Been Done?
None of the listed papers discuss integrating an Elastic Weight Consolidation (EWC) loss or Fisher Information Matrix calculation directly into the training loop for sequence processing to prevent drift. While works like SUV and Mask the Target address unlearning or forgetting, they do not cover the specific Fisher-regularized training mechanism proposed here.
We searched OpenAlex and arXiv for “Incorporate Fisher Information regularization directly into training to prevent fast weights from drifting and corrupting previously learned geometries.” and “Fisher-Regularized Training for Drift Prevention” and an AI compared the idea with the top 10 results. A small search can miss relevant work.
Source Paper
tttLRM: Test-Time Training for Long Context and Autoregressive 3D ReconstructionChen Wang, Hao Tan, Wang Yifan +6 more · 2026
The paper presents tttLRM, a Large Reconstruction Model that utilizes Test-Time Training (TTT) layers to perform 3D reconstruction from long sequences of images with linear computational complexity. It enables both feedforward and autoregressive (streaming) reconstruction of explicit 3D representations like Gaussian Splats, scaling efficiently to large context windows where traditional Transformers fail.
Related Ideas
- Adaptive Chunking Mechanisms for Event-Driven Test-Time Training in Spatial MLLMsSpatial-TTT: Streaming Visual-based Spatial Intelligence with Test-Time Training🟡Fairly easy
- Global Geometric Memory for Loop ClosureWorldWarp: Propagating 3D Geometry with Asynchronous Video Diffusion🟠Medium
- Adaptive Token Sparsity and Early Exiting for Real-Time VLM InferencePenguin-VL: Exploring the Efficiency Limits of VLM with LLM-based Vision Encoders🔴Hard
- Non-Linear Latent Tracking for Robust Depth Estimation in Unconstrained Long VideosDVD: Deterministic Video Depth Estimation with Generative Priors🔴Hard
- Adaptive Reference-Free Extrapolation for Video World Models under Sparse GroundingGrounding World Simulation Models in a Real-World Metropolis🟠Medium