Skip to content

Source paper

Gamer: A Generalist Vision-Language-Action Foundation Model for Long-Horizon Real-Time Game Agent

Run Luo, Kuan Zhang, Qi Deng and 24 more

Run Luo, Kuan Zhang, Qi Deng, Yukun Chen, Zhihao Yang, Yufei Zhang, Renrong Hu, Yi Zhang, Guiyu Ma, Tianyu Fan, Longze Chen, Hao Sun, Jiaming Li, Jiaming Yang, Xiaohao Liu, Tao Tan, Minzheng Wang, Hangtian Jia, Shuai Yuan, Lvdi Wang, Xiao Liu, Chao Ma, Yuting Zou, Yinhe Zheng, Xiaobo Xia, Dawei Jiang, Shaoqing Ren

arXiv 2611.18397 · 2026119 upvotes on Hugging Face

Research ideas from this paper (3)

Generated by AI. The paper’s authors did not write these.

AI summary

This paper presents Gamer, a generalist Vision-Language-Action (VLA) foundation model designed for real-time, long-horizon interactive 3D and 2D gaming environments using raw screen observations and standard keyboard-mouse actions. The system employs a three-stage training pipeline comprising continued pre-training (CPT) with trajectory debiasing, multi-level supervised fine-tuning (SFT) with sparse reasoning and memory compaction, and Temporal-Aware Group Relative Policy Optimization (TA-GRPO) using game-native verifiers. To achieve sub-100ms latency necessary for real-time closed-loop control and rollout generation, the authors develop Streaming-vLLM, featuring stateful KV-cache reuse, incremental RoPE cache updates, and FSM-guided speculative decoding.

How it works (AI summary)

The framework unifies game interaction into a serialized 30 Hz keyboard-mouse action format decoupled from a 5 Hz visual observation rate. It trains dense and MoE backbones across CPT, SFT, and online RL stages: CPT utilizes random key remapping and repeat-aware loss decay over multi-game datasets; SFT integrates hierarchical task instructions (operations, subgoals, tasks) with sparse autoregressive reasoning and FIFO summary-based context memory compaction; and TA-GRPO optimizes memory-linked segment rollouts using verifiable game-state outcomes and temporally discounted credit assignment. Serving and rollouts are accelerated via Streaming-vLLM with stateful KV caching, in-place rotary phase shifts, and FSM-guided speculative drafting.