Linear-Complexity Backbone via State Space Models
Replace transformer self-attention layers with selective state space model blocks so time-series foundation models scale linearly with patch length during inference.
Time
Compute level
Resources
What the paper used
Prerequisites
Scope
The Gap
The model uses a 'Stacked Transformer' (Section 4) with multi-head self-attention, which scales quadratically with the number of patches. Appendix A.1 explicitly suggests trying 'efficient linear state space models like Mamba'.
Limitation
While patching reduces sequence length, the quadratic complexity of Attention still limits the ability to process extremely long historical contexts (e.g., high-frequency tick data) efficiently.
Proposed Solution
Replace the Transformer attention layers with Selective State Space Model (SSM) blocks (e.g., Mamba). Retain the patching mechanism and the decoder-only training objective. Pre-train this 'TimesMamba' architecture on the same 100B token corpus to benchmark the trade-off between inference throughput (linear scaling) and zero-shot forecasting accuracy.
The paper’s authors list this direction as future work, so they may already be pursuing it.
Why This Matters
A linear-time foundation model could handle vastly longer context windows, potentially capturing long-term dependencies (e.g., year-over-year seasonality in hourly data) that strictly windowed Transformers miss.
How to Get Started
How to test it: We hypothesize that replacing self-attention with Mamba blocks scales linearly while maintaining zero-shot MAE on ETT datasets within 5 percent of TimesFM. If accuracy degrades beyond 5 percent, we stop.
- Replace the stacked transformer attention layers in the base architecture with Mamba state space model blocks.
- Retain the original patch embedding mechanism and decoder-only pre-training objective.
- Pre-train the new state space architecture on the standard token corpus using a distributed compute cluster.
- Benchmark inference throughput and zero-shot forecasting accuracy against the original transformer-based model.
Code linked to this paper on Hugging Face: github.com/google-research/timesfm
Suggested by AI. Adapt it to your setup.
Has This Been Done?
State space models such as Mamba and MambaTS explore replacing transformer architectures with linear-complexity backbones for time series forecasting. However, none of the listed papers specifically evaluate the exact 100B token pre-training and zero-shot benchmark proposed under the 'TimesMamba' architecture name.
- Mamba: Linear-Time Sequence Modeling with Selective State Spaces · 2023 · 11823 citations
- MambaTS: Improved selective state space models for long-term time series forecasting · 2026 · 84 citations
We searched OpenAlex for “Replace transformer self-attention layers with selective state space model blocks so time-series foundation models scale linearly with patch length during inference.” and “Linear-Complexity Backbone via State Space Models” and an AI compared the idea with the top 10 results. A small search can miss relevant work.
Source Paper
A decoder-only foundation model for time-series forecastingAbhimanyu Das, Weihao Kong, Rajat Sen +1 more · 2023
This paper introduces TimesFM, a 200M-parameter decoder-only foundation model for time-series forecasting pretrained on a massive corpus of 100 billion real and synthetic time-points. The authors demonstrate that by using a patch-based decoder architecture and extensive pretraining, the model achieves zero-shot performance on unseen datasets that rivals or exceeds state-of-the-art supervised models trained specifically for those tasks.
Related Ideas
- Homogeneous Mamba-Based Audio TokenizationMOSS-Audio-Tokenizer: Scaling Audio Tokenizers for Future Audio Foundation Models🔴Hard
- Adaptive Attention-Based Memory CompressionHelios: Real Real-Time Long Video Generation Model🟠Medium
- Hardware-Efficient Backbone for Scalable DeploymentUtonia: Toward One Encoder for All Point Clouds⛔Very hard
- Causal-SLA for Autoregressive Video GenerationTurboDiffusion: Accelerating Video Diffusion Models by 100-200 Times🔴Hard
- Non-Autoregressive Local Decoding for Real-Time InferenceHeartMuLa: A Family of Open Sourced Music Foundation Models🟠Medium