Distilling Open-Ended Generation from RandOpt Ensembles
Develop a sequence-level knowledge distillation framework that aligns student output distributions with averaged token-level probability distributions from an ensemble of teachers.
Source paper
Generated by AI. The paper’s authors did not write these.
Develop a sequence-level knowledge distillation framework that aligns student output distributions with averaged token-level probability distributions from an ensemble of teachers.
Implement a representation-space ensembling method using hidden-state interpolation or a reward model verifier to execute Best-of-K selection for continuous structured predictions.
Train a suite of language models on controlled data mixtures to track spectral discordance and solution density, mapping the exact threshold where neural thickets emerge.
This paper reveals that large pretrained models exist in a 'thicket' regime, where the local neighborhood around their weights is densely populated with diverse, task-specific experts. Based on this finding, the authors introduce RandOpt, a gradient-free post-training method that samples random weight perturbations and ensembles the best ones via majority voting to achieve competitive performance against standard RL methods.
RandOpt generates N random Gaussian parameter perturbations around a pretrained model's weights, evaluates them on a validation set to select the top K performers, and aggregates their predictions at inference time using majority voting, entirely avoiding gradient-based sequential optimization.