Skip to content

Source paper

K2-Horizon: A Frontier Fleet of Open Source Language Models

K2 Team, Zhengzhong Liu, Haonan Li and 92 more

K2 Team, Zhengzhong Liu, Haonan Li, Xuezhe Ma, Alexander Moreno, Subham Sekhar Sahoo, Rupesh Kumar Srivastava, Mikhail Yurochkin, Hadi Abdine, Yash Akhauri, Adharsh Babu, David Bellamy, Jalaj Bhandari, Dani Bouch, Wail Bouhedja, Weili Cao, Ahmad Chamma, Po-Chun Chang, Junlin Chen, Lingjie Chen, Zhoujun Cheng, Kyle Chickering, Ryandito Diandaru, Evan Dufraisse, Chaitanya Dwivedi, Bilal El Bouardi, Desai Fan, Richard Fan, Chengqian Gao, Rifo Ahmad Genadi, Jonathan Geuter, Charles Guille-Escuret, Willis Guo, Dhawal Gupta, Seungwook Han, Tingxu Han, Shibo Hao, Besher Hassan, Guowei He, Ming Shan Hee, Joel Hestness, Yu Huang, Yuan Huang, Haolong Jia, Linghao Jin, Junu Kim, Lingjing Kong, Nurkhan Laiyk, Andrej Leban, Liqun Ma, John Maggs, Amr Mohamed, Mukhesh Narra, Omkar Pangarkar, Aldo Pareja, Khiem Pham, Varad Pimpalkhute, Nikhil Ranjan, Elizaveta Rebrova, Hector Ren, Shaurya Rohatgi, Guokan Shang, Jianshu She, Chufan Shi, Aaryamonvikram Singh, Daria Soboleva, Kartik Sreenivasan, Suqi Sun, Yuekai Sun, Bowen Tan, Liping Tang, Shengkun Tang, Muzi Tao, Mallikarjuna Tupakula, Micah Villmow, Hongyi Wang, Huijuan Wang, Renxi Wang, Yongxin Wang, Yuqi Wang, Zhenting Wang, Shicheng Wen, Yingquan Wu, Anze Xie, Andrew Xing, Matthew Yang, Shuanghai Yu, Kerem Zaman, Zenan Zhai, Hongli Zhan, Xuezhou Zhang, Anshun Asher Zheng, Zhe Zhou, Richard Zhuang, Eric P. Xing

arXiv 2611.21003 · 202686 upvotes on Hugging Face

Research ideas from this paper (3)

Generated by AI. The paper’s authors did not write these.

AI summary

This technical report presents K2-Horizon, a family of six fully open language models ranging from 0.9B dense to a 375B mixture-of-experts (MoE) flagship activating 23B parameters, optimized for agentic and reasoning tasks. The authors open-source the complete development artifacts, including intermediate checkpoints, pre-training corpora with 10T synthetic tokens, the xLLM training framework, and the RL360 reinforcement learning stack. Key architectural innovations include Grouped RMSNorm, Mixture-of-Value Attention (MoVA), softplus attention gating, and Uno discrete diffusion adapters for accelerated decoding.

How it works (AI summary)

The architecture incorporates fine-grained sparse MoE feed-forward layers, Grouped RMSNorm, Mixture-of-Value Attention (sparse routing over value projections in attention), and softplus-gated attention. Models undergo a multi-stage training pipeline comprising pre-training across up to 22T tokens, a multi-phase mid-training curriculum extending context up to 512k tokens, domain-specialized reinforcement learning via a single-step GRPO variant across asynchronous environments, parameter-level model merging using ISO-RAM (ISO-merger for attention/shared experts and RAM for routed experts), and targeted supervised fine-tuning.