Skip to content
AI-generated research idea🟠Medium💡Moderate InnovationClosely matching work found

Covariate-Aware Foundation Model Architecture

Extend univariate time-series foundation models to accept parallel streams of dynamic covariates and static attributes through fused patch embeddings.

Time

3-5 days

Compute level

Large cluster

Resources

A TPUv5e or multi-GPU cluster with at least 16 accelerators for fine-tuning on multivariate datasets over several days.

What the paper used

200M parameters, O(100B) timepoints, TPUv5e setup with 16 tensor-cores

Prerequisites

transformer architectures, multi-modal fusion techniques

Scope

modify existing code, new model architecture, partial retraining

The Gap

The current TimesFM model is strictly univariate (f: y1:L → ŷL+1:L+H), as noted in Appendix A.1. It processes the target time-series history but ignores dynamic covariates (e.g., price, weather) and static attributes (e.g., product category), which are critical in domains like retail and energy.

“Currently the model is not pretrained with covariates as one of the key challenges is finding large volumes of pretrained data with meaningful covariates”
From the paper, Appendix A.1, page 13. Matched against the PDF text.

Limitation

The inability to ingest exogenous variables limits the model's applicability in causal forecasting scenarios where the target variable is driven by external factors rather than just its own history.

Proposed Solution

Extend the architecture to accept a parallel stream of covariate patches. Modify the `InputResidualBlock` to fuse the time-series patch embedding with covariate embeddings (via concatenation or cross-attention). Fine-tune the pre-trained model on multivariate datasets (like M5 or Traffic) where covariates are available, or generate synthetic covariates (e.g., random 'price' shocks correlated with the synthetic 'demand') to continue pre-training.

The paper’s authors list this direction as future work, so they may already be pursuing it.

Why This Matters

Enabling covariate support would transition the model from a pure extrapolator to a causal forecasting tool, significantly increasing its utility in industrial applications like supply chain planning.

How to Get Started

How to test it: We hypothesize that adding covariate patches improves MAE on multivariate datasets like M5 compared to univariate baselines. If the MAE does not decrease by at least 2 percent after fine-tuning, we stop.

  1. Modify the input residual block of the base univariate architecture to accept covariate patch embeddings alongside target history.
  2. Implement cross-attention or concatenation mechanisms to fuse time-series features with external price and weather variables.
  3. Curate multivariate benchmark datasets such as M5 or Traffic with well-defined dynamic and static covariates.
  4. Fine-tune the expanded model architecture on the multivariate datasets and evaluate forecasting accuracy against univariate baselines.

Code linked to this paper on Hugging Face: github.com/google-research/timesfm

Suggested by AI. Adapt it to your setup.

Has This Been Done?

Closely matching work found

Works like CoRA: Covariate-Aware Adaptation of Time Series Foundation Models and UniCA: Unified Covariate Adaptation for Time Series Foundation Model specifically investigate adapting time-series foundation models to incorporate covariates. These papers explore architectures and adaptation strategies for fusing covariates with pre-trained embeddings for multivariate forecasting. Therefore, the proposed architectural modifications and fine-tuning strategies have essentially been explored in the recent literature.

We searched OpenAlex and arXiv for “Extend univariate time-series foundation models to accept parallel streams of dynamic covariates and static attributes through fused patch embeddings.” and “Covariate-Aware Foundation Model Architecture” and an AI compared the idea with the top 10 results. A small search can miss relevant work.

Source Paper

A decoder-only foundation model for time-series forecasting

Abhimanyu Das, Weihao Kong, Rajat Sen +1 more · 2023

This paper introduces TimesFM, a 200M-parameter decoder-only foundation model for time-series forecasting pretrained on a massive corpus of 100 billion real and synthetic time-points. The authors demonstrate that by using a patch-based decoder architecture and extensive pretraining, the model achieves zero-shot performance on unseen datasets that rivals or exceeds state-of-the-art supervised models trained specifically for those tasks.

Related Ideas

Is this a good idea?

Share onXLinkedIn