Skip to content

Source paper

LiveTalk: Real-Time Multimodal Interactive Video Diffusion via Improved On-Policy Distillation

Ethan Chern, Zhulin Hu, Bohao Tang and 4 more

Ethan Chern, Zhulin Hu, Bohao Tang, Jiadi Su, Steffi Chern, Zhijie Deng, Pengfei Liu

arXiv 2512.23576 · 202555 upvotes on Hugging Face

Research ideas from this paper (3)

Generated by AI. The paper’s authors did not write these.

AI summary

The paper presents LiveTalk, a real-time system for interactive multimodal avatar video generation using a novel video diffusion distillation recipe. By refining multimodal data quality, ensuring full ODE convergence before distillation, and using an aggressive optimization schedule, the authors achieve 20x acceleration and sub-second latency while maintaining visual quality comparable to larger bidirectional models.

How it works (AI summary)

The core technique is an improved 'Self Forcing' on-policy distillation method for Distribution Matching Distillation (DMD). The system uses a causal, block-wise autoregressive video diffusion model conditioned on audio, text, and images, integrated with 'Anchor-Heavy Identity Sinks' (AHIS) in the attention mechanism to preserve speaker identity over long streaming sessions.