Skip to content

Source paper

In Pursuit of Pixel Supervision for Visual Pre-training

Lihe Yang, Shang-Wen Li, Yang Li and 5 more

Lihe Yang, Shang-Wen Li, Yang Li, Xinjie Lei, Dong Wang, Abdelrahman Mohamed, Hengshuang Zhao, Hu Xu

arXiv 2512.15715 · 20257 upvotes on Hugging Face

Research ideas from this paper (4)

Generated by AI. The paper’s authors did not write these.

Natural Predictive Pre-training on Web-Scale Video

A predictive pre-training framework that replaces artificial spatial image masking with natural future frame and optical flow reconstruction tasks using web-scale video datasets.

In Pursuit of Pixel Supervision for Visual Pre-training
⛔Very hard🔬Significant ResearchLarge clusterNot yet checked#Training#Vision

Stabilized Online Hard Example Mining via Momentum-based Difficulty Estimation

Stabilize online hard example mining by using an exponential moving average teacher model for targets, a warm-up period, and batch-level loss statistics to prevent early-stage reconstruction noise from disrupting training.

In Pursuit of Pixel Supervision for Visual Pre-training
🟡Fairly easy💡Moderate InnovationSingle GPU or lessNo close match found#Training#Vision

Multi-Scale Hierarchical Masking for Dense Prediction Tasks

Apply a hierarchical multi-scale masking strategy across encoder layers and combine it with a pyramid reconstruction loss to bridge local textural details with global semantic representations.

In Pursuit of Pixel Supervision for Visual Pre-training
🟠Medium💡Moderate InnovationLarge clusterRelated work found#Vision#Training

Latent Feature Alignment to Resolve Mask-Visible Disparity

Introduce a feature-level alignment loss between visible and masked latent representations to eliminate visual quality disparities caused by loss computation restricted only to masked tokens.

In Pursuit of Pixel Supervision for Visual Pre-training
🟠Medium💡Moderate InnovationMultiple GPUsNo close match found-2#Vision#Training

AI summary

This paper presents Pixio, an enhanced Masked Autoencoder (MAE) framework that demonstrates the continued viability of pixel-space self-supervised learning for large-scale pre-training. By scaling to 2 billion web-crawled images and introducing architectural improvements like deeper decoders and multiple class tokens, Pixio achieves performance competitive with latent-space methods like DINOv3 on tasks such as depth estimation and semantic segmentation.

How it works (AI summary)

The methodology builds on the MAE paradigm by increasing pre-training difficulty through larger masking blocks (4x4 patches) and increasing model capacity via a significantly deeper ViT decoder (32 blocks) and 8 global class tokens. A loss-based soft self-curation strategy is used to filter 2B uncurated web images, prioritizing visually complex data by sampling images based on their reconstruction error from a baseline model.