Skip to content
AI-generated research idea🟠Medium🔬Significant ResearchRelated work found

Latent Space Disentanglement for Robust Global Style Transfer in Real-World Grounded Video Generation

A method that combines Low-Rank Adaptation with cross-attention control to explicitly disentangle spatial geometry conditioning from global style conditioning in real-world world models.

Time

Approximately 1 week (3-5 days for training and ablating LoRA rank and style-regularization weights across 24 H100 GPUs, plus 1-2 days for benchmark and style-prompt evaluation).

Compute level

Large cluster

Resources

24 NVIDIA H100 GPUs to train the 2B-parameter Cosmos-Predict2.5-2B backbone with LoRA and cross-attention regularization for 10K iterations (batch size 48) on the 440K street-view, 12.7K synthetic, and Waymo datasets.

What the paper used

Fine-tunes Cosmos-Predict2.5-2B (2B-parameter DiT) using AdamW for 10K iterations at batch size 48 across 24 NVIDIA H100 GPUs (plus 6K ODE init steps on 1K pairs and 10K fine-tuning iterations for the Self-Forcing variant; inference runs at 15.2 fps on 1 H100 GPU) on 440K Seoul street-view images (from 1.2M panoramas), 12.7K CARLA synthetic videos, Waymo Open Dataset, and 10K stylized videos.

Prerequisites

transformer architectures

Scope

modify existing code

The Gap

SWM fine-tunes a base pretrained world model on real-world street-view images, which predominantly feature standard daytime lighting and weather conditions.

“However, prompts involving global style changes such as day-to-night transitions or weather variations tend to be less faithfully followed after fine-tuning on street-view data.”
From the paper, Appendix A.2, page 22. Matched against the PDF text.

Limitation

The model exhibits catastrophic forgetting of the base model's ability to handle global style changes (e.g., day-to-night transitions, severe weather conditions). To compensate, the authors currently have to artificially augment the training data using a computationally expensive external video stylization pipeline.

Proposed Solution

Implement a Low-Rank Adaptation (LoRA) module combined with a cross-attention control mechanism that explicitly disentangles spatial geometry conditioning (from street-view images) from global style conditioning (from text prompts). Train the base spatial layout weights on the street-view data while strictly regularizing the style-mapping layers to preserve the pre-trained model's stylistic and atmospheric priors.

Why This Matters

This removes the reliance on synthetic, offline video stylization pipelines during training, allowing the model to generate highly diverse, zero-shot environmental variations seamlessly, which is critical for edge-case simulation.

How to Get Started

How to test it: Hypothesis: LoRA with regularized cross-attention disentangling spatial layout from global style preserves the pretrained model's atmospheric priors without needing the 10K external stylized video augmentations. Compare against SWM (TF) trained with stylized video augmentation using text-style alignment (CLIP/VLM/human preference) on global style prompts and FID, FVD, mPSNR, mLPIPS on Busan-City-Bench and Ann-Arbor-City-Bench; stop if style adherence underperforms the stylized-augmentation baseline or mPSNR degrades by >0.5 dB.

  1. Set up a pretrained world model codebase capable of conditioning on real-world street-view image inputs.
  2. Implement a Low-Rank Adaptation (LoRA) module coupled with cross-attention control to separate layout and style pathways.
  3. Train spatial layout weights on real-world driving datasets while applying regularization constraints to style-mapping layers.
  4. Evaluate generated video samples across diverse text prompts to ensure stylistic priors are preserved without degrading geometric accuracy.

Code linked to this paper on Hugging Face: github.com/naver-ai/seoul-world-model

Suggested by AI. Adapt it to your setup.

Has This Been Done?

Related work found

Papers such as "Attention Calibration for Disentangled Text-to-Image Personalization" and "Disco: Disentangled Control for Realistic Human Dance Generation" discuss disentangled control and latent space adjustments, but none explicitly detail the combination of LoRA and cross-attention control to isolate spatial geometry conditioning from global style conditioning for grounded video generation.

We searched OpenAlex for “A method that combines Low-Rank Adaptation with cross-attention control to explicitly disentangle spatial geometry conditioning from global style conditioning in real-world world models.” and “Latent Space Disentanglement for Robust Global Style Transfer in Real-World Grounded Video Generation” and an AI compared the idea with the top 10 results. A small search can miss relevant work.

Source Paper

Grounding World Simulation Models in a Real-World Metropolis

Junyoung Seo, Hyunwook Choi, Minkyung Kwon +10 more · 2026

The paper introduces the Seoul World Model (SWM), a city-scale video generative world model that anchors simulated environments to the actual physical layout of a real metropolis. The main contribution is a retrieval-augmented generation framework that leverages cross-temporal pairing, synthetic urban data, and a Virtual Lookahead Sink to synthesize spatially faithful, temporally consistent, and long-horizon videos.

Related Ideas

Is this a good idea?

Share onXLinkedIn