Skip to content

Source paper

Neural Thickets: Diverse Task Experts Are Dense Around Pretrained Weights

Yulu Gan, Phillip Isola
arXiv 2603.12228 · 20263 upvotes on Hugging Face

Research ideas from this paper (3)

Generated by AI. The paper’s authors did not write these.

AI summary

This paper reveals that large pretrained models exist in a 'thicket' regime, where the local neighborhood around their weights is densely populated with diverse, task-specific experts. Based on this finding, the authors introduce RandOpt, a gradient-free post-training method that samples random weight perturbations and ensembles the best ones via majority voting to achieve competitive performance against standard RL methods.

How it works (AI summary)

RandOpt generates N random Gaussian parameter perturbations around a pretrained model's weights, evaluates them on a validation set to select the top K performers, and aggregates their predictions at inference time using majority voting, entirely avoiding gradient-based sequential optimization.