Skip to content

Source paper

Cautious Weight Decay

Lizhang Chen, Jonathan Li, Kaizhao Liang and 6 more

Lizhang Chen, Jonathan Li, Kaizhao Liang, Baiyu Su, Cong Xie, Nuo Wang Pierse, Chen Liang, Ni Lao, Qiang Liu

arXiv 2510.12402 · 20259 upvotes on Hugging Face

Research ideas from this paper (3)

Generated by AI. The paper’s authors did not write these.

AI summary

The paper introduces Cautious Weight Decay (CWD), a one-line, optimizer-agnostic modification that applies decoupled weight decay only when the parameter's sign aligns with the optimizer's update direction. By preserving the original loss objective, CWD improves convergence, lowers final validation loss, and increases zero-shot accuracy across large-scale language and vision models without requiring hyperparameter retuning.

How it works (AI summary)

CWD utilizes a sign-selective binary mask (an indicator function) during the parameter update step, applying weight decay selectively. Theoretically, this is supported by Lyapunov analysis and Filippov differential inclusions, proving that standard optimizers with CWD undergo sliding-mode dynamics to reach locally Pareto-optimal points on the stationary manifold of the unregularized objective.