Distilling Open-Ended Generation from RandOpt Ensembles
Develop a sequence-level knowledge distillation framework that aligns student output distributions with averaged token-level probability distributions from an ensemble of teachers.
Time
Compute level
Resources
What the paper used
Prerequisites
Scope
The Gap
The distillation approach described in Section 7 is explicitly tailored to LLM reasoning tasks that end in a categorical final prediction (e.g., math answers), making it difficult to apply to code generation or creative writing tasks.
“A catch with RandOpt, compared to standard post-training, is that good performance requires K forward passes at test time. To address this, we explore distilling the top-K models into a single model.”
Limitation
RandOpt incurs a high inference-time cost by requiring K forward passes, and the paper's proposed distillation fix cannot currently handle open-ended text where strict majority voting on identical output sequences is impossible.
Proposed Solution
Develop a sequence-level knowledge distillation framework that aligns the student model's output distribution with the averaged token-level probability distributions of the K teacher models (the RandOpt ensemble) at each autoregressive decoding step, rather than relying on exact trace matches.
The paper’s authors list this direction as future work, so they may already be pursuing it.
Why This Matters
This would allow the performance benefits of RandOpt to be deployed efficiently at inference time for a much broader range of generative tasks, drastically reducing latency while maintaining ensemble-level reasoning.
How to Get Started
How to test it: Hypothesis: Aligning the student model to the averaged token-level probability distribution of the K teacher models achieves performance close to the ensemble. Baseline: The un-distilled base model or discrete trace-matching distillation. Metric: Accuracy on GSM8K and MATH-500. Stopping condition: If the distilled student achieves at least 95% of the ensemble's accuracy while requiring only a single forward pass.
- Set up an ensemble of diverse teacher models to generate multiple reasoning traces for open-ended code generation and creative writing tasks.
- Compute averaged token-level probability distributions across the ensemble outputs during autoregressive generation steps.
- Implement a sequence-level divergence loss function to align the student model's token distribution with the ensemble average.
- Evaluate the distilled student model on standard open-ended text and code generation benchmarks.
Code linked to this paper on Hugging Face: github.com/sunrainyg/RandOpt
Suggested by AI. Adapt it to your setup.
Has This Been Done?
Papers like Logit-Based Ensemble Distribution Distillation for Robust Autoregressive Sequence Uncertainties and Universal Cross-Tokenizer Distillation via Approximate Likelihood Matching discuss ensemble distillation and logit matching for language models. However, none of the listed works specifically propose distilling an open-ended RandOpt ensemble using token-level probability averaging at each autoregressive step.
- Logit-Based Ensemble Distribution Distillation for Robust Autoregressive Sequence Uncertainties · 2023 · 11 citations
- Universal Cross-Tokenizer Distillation via Approximate Likelihood Matching · 2025 · 59 citations
We searched OpenAlex and arXiv for “Develop a sequence-level knowledge distillation framework that aligns student output distributions with averaged token-level probability distributions from an ensemble of teachers.” and “Distilling Open-Ended Generation from RandOpt Ensembles” and an AI compared the idea with the top 10 results. A small search can miss relevant work.
Source Paper
Neural Thickets: Diverse Task Experts Are Dense Around Pretrained WeightsYulu Gan, Phillip Isola · 2026
This paper reveals that large pretrained models exist in a 'thicket' regime, where the local neighborhood around their weights is densely populated with diverse, task-specific experts. Based on this finding, the authors introduce RandOpt, a gradient-free post-training method that samples random weight perturbations and ensembles the best ones via majority voting to achieve competitive performance against standard RL methods.
Related Ideas
- Distilling Explicit Reasoning into Latent SpaceRobust-R1: Degradation-Aware Reasoning for Robust Visual Understanding🟡Fairly easy
- BitDistill for Reasoning-Intensive DomainsBitNet Distillation🟠Medium
- Distillation-Enhanced ARLM-to-DLM ConversiondLLM: Simple Diffusion Language Modeling🟠Medium
- Process-Supervised Reasoning-Targeted Distillation for Routing-Replaced MoEsThe Other Half of the Memory Wall: Serving 35B MoEs from SSD with Trained Routing Prediction🟠Medium
- Distillation of Reasoning Models for Efficient Route PlanningMobilityBench: A Benchmark for Evaluating Route-Planning Agents in Real-World Mobility Scenarios🟡Fairly easy