Mask-Guided Spatiotemporal Token Pruning for Inference
A method that prunes diffusion transformer tokens outside the face mask region during video generation to accelerate face-swapping inference without sacrificing visual fidelity.
Time
Compute level
Resources
Prerequisites
Scope
The Gap
Section 3.2 describes the Diffusion Transformer (DiT) treating the entire video volume as tokens. The 'Spatio-Temporal Context Module' injects the background, which often remains static or changes minimally compared to the face.
Limitation
Processing background tokens with the same computational intensity as facial region tokens is inefficient. Since the goal is face swapping, a large portion of the video (the background) requires minimal semantic alteration, yet the DiT consumes massive compute resources processing these redundant tokens.
Proposed Solution
Implement a dynamic token pruning mechanism (e.g., ToMe) guided by the face mask provided in the condition. During inference, progressively merge or prune tokens falling outside the dilated face mask region in the later layers of the DiT, effectively focusing compute solely on the high-frequency changes required for the face swap.
Why This Matters
DiT video generation is notoriously slow. This optimization could speed up inference by 2-3x without quality loss, making the high-fidelity model actually viable for consumer applications or real-time editing workflows.
How to Get Started
- Set up an open-source Diffusion Transformer codebase like Open-Sora and load a pre-trained face-swapping model checkpoint.
- Implement a dilated face-mask extraction pipeline using MediaPipe or similar lightweight face landmark detectors on benchmark video datasets like VoxCeleb2.
- Modify the transformer's self-attention layers to dynamically drop or merge tokens falling outside the dilated face mask during the later denoising steps.
- Evaluate the modified model on reconstruction metrics such as PSNR, SSIM, and Fréchet Video Distance (FVD) against the unpruned baseline while measuring inference speedup.
Code linked to this paper on Hugging Face: github.com/bytedance/DreamID-V
Suggested by AI. Adapt it to your setup.
Has This Been Done?
Papers like FastSTAR: Spatiotemporal Token Pruning for Efficient Autoregressive Video Synthesis and Token Pruning for In-Context Generation in Diffusion Transformers discuss general spatiotemporal token pruning, but none address mask-guided dynamic token pruning specifically tailored to face masks for identity-preserving video generation.
- FastSTAR: Spatiotemporal Token Pruning for Efficient Autoregressive Video Synthesis · 2026 · 3 citations
- Token Pruning for In-Context Generation in Diffusion Transformers · 2026 · 4 citations
We searched OpenAlex for “A method that prunes diffusion transformer tokens outside the face mask region during video generation to accelerate face-swapping inference without sacrificing visual fidelity.” and “Mask-Guided Spatiotemporal Token Pruning for Inference” and an AI compared the idea with the top 10 results. A small search can miss relevant work.
Source Paper
DreamID-V:Bridging the Image-to-Video Gap for High-Fidelity Face Swapping via Diffusion TransformerXu Guo, Fulong Ye, Xinghui Li +6 more · 2026
The paper introduces DreamID-V, a video face-swapping framework that addresses the lack of temporal consistency and paired training data in existing methods. The authors bridge the gap between image and video domains by creating a synthetic data pipeline (SyncID-Pipe) that generates high-fidelity video-face pairs and employing a Diffusion Transformer (DiT) aimed at preserving identity and motion continuity.
Related Ideas
- Spatiotemporal SpotEdit for Video DiTsSpotEdit: Selective Region Editing in Diffusion Transformers🔴Hard
- BitDance-Video: Spatiotemporal Binary TokenizationBitDance: Scaling Autoregressive Generative Models with Binary Tokens🔴Hard
- Semantic-Aware Variable-Rate TokenizationNextFlow: Unified Sequential Modeling Activates Multimodal Understanding and Generation🔴Hard
- Distilling Generative Video Depth Priors into Lightweight Linear-Complexity ModelsDVD: Deterministic Video Depth Estimation with Generative Priors🔴Hard
- Spatiotemporal Prism: Unified Video AutoencodingThe Prism Hypothesis: Harmonizing Semantic and Pixel Representations via Unified Autoencoding🔴Hard