Skip to content

Source paper

BitNet b1.58 2B4T Technical Report

Shuming Ma, Hongyu Wang, Shaohan Huang and 5 more

Shuming Ma, Hongyu Wang, Shaohan Huang, Xingxing Zhang, Ying Hu, Ting Song, Yan Xia, Furu Wei

arXiv 2504.12285 · 202576 upvotes on Hugging Face

Research ideas from this paper (4)

Generated by AI. The paper’s authors did not write these.

AI summary

This paper introduces BitNet b1.58 2B4T, a 2-billion parameter Large Language Model trained from scratch with 1.58-bit ternary weights ({-1, 0, 1}) and 8-bit activations. The authors demonstrate that this native 1-bit model achieves performance parity with full-precision open-source models (like LLaMA and Gemma) of similar size while significantly reducing memory footprint, energy consumption, and inference latency.

How it works (AI summary)

The model utilizes a Transformer architecture where standard linear layers are replaced with `BitLinear` layers. Weights are quantized to ternary values using absolute mean quantization, and activations are quantized to 8-bit integers. Training involves a high learning rate followed by a cooldown phase on high-quality data, employing purely SFT and Direct Preference Optimization (DPO) without reinforcement learning.