Natural Predictive Pre-training on Web-Scale Video
A predictive pre-training framework that replaces artificial spatial image masking with natural future frame and optical flow reconstruction tasks using web-scale video datasets.
Source paper
Lihe Yang, Shang-Wen Li, Yang Li, Xinjie Lei, Dong Wang, Abdelrahman Mohamed, Hengshuang Zhao, Hu Xu
Generated by AI. The paper’s authors did not write these.
A predictive pre-training framework that replaces artificial spatial image masking with natural future frame and optical flow reconstruction tasks using web-scale video datasets.
Stabilize online hard example mining by using an exponential moving average teacher model for targets, a warm-up period, and batch-level loss statistics to prevent early-stage reconstruction noise from disrupting training.
Apply a hierarchical multi-scale masking strategy across encoder layers and combine it with a pyramid reconstruction loss to bridge local textural details with global semantic representations.
Introduce a feature-level alignment loss between visible and masked latent representations to eliminate visual quality disparities caused by loss computation restricted only to masked tokens.
This paper presents Pixio, an enhanced Masked Autoencoder (MAE) framework that demonstrates the continued viability of pixel-space self-supervised learning for large-scale pre-training. By scaling to 2 billion web-crawled images and introducing architectural improvements like deeper decoders and multiple class tokens, Pixio achieves performance competitive with latent-space methods like DINOv3 on tasks such as depth estimation and semantic segmentation.
The methodology builds on the MAE paradigm by increasing pre-training difficulty through larger masking blocks (4x4 patches) and increasing model capacity via a significantly deeper ViT decoder (32 blocks) and 8 global class tokens. A loss-based soft self-curation strategy is used to filter 2B uncurated web images, prioritizing visually complex data by sampling images based on their reconstruction error from a baseline model.