Covariate-Aware Foundation Model Architecture
Extend univariate time-series foundation models to accept parallel streams of dynamic covariates and static attributes through fused patch embeddings.
Source paper
Generated by AI. The paper’s authors did not write these.
Extend univariate time-series foundation models to accept parallel streams of dynamic covariates and static attributes through fused patch embeddings.
Replace deterministic point forecast output layers with multi-head quantile projection layers so models produce calibrated probabilistic predictions.
Replace transformer self-attention layers with selective state space model blocks so time-series foundation models scale linearly with patch length during inference.
This paper introduces TimesFM, a 200M-parameter decoder-only foundation model for time-series forecasting pretrained on a massive corpus of 100 billion real and synthetic time-points. The authors demonstrate that by using a patch-based decoder architecture and extensive pretraining, the model achieves zero-shot performance on unseen datasets that rivals or exceeds state-of-the-art supervised models trained specifically for those tasks.
The model uses a decoder-only Transformer architecture modified with input and output patching. Input time-series are broken into patches (tokens), processed by residual blocks, and fed into a causal transformer. Uniquely, the model predicts output patches that can be longer than input patches to reduce autoregressive inference steps. It was trained on a mixed corpus of Google Trends, Wiki Pageviews, and synthetic data using a standard Mean Squared Error (MSE) loss.