Some random thoughts I've been having about video world model/long video generation since working on Mixture of Contexts (whose title could also be "Learnable Sparse Attention for Long Video Generation"):
🚨Semi-long Post Alert🚨
1. Learnable sparse attention is still underrated for video, 3D/4D, and world models.
- Different from text: text often hinges on single-token dependencies; video almost never does. Visual signals of interest form patch/tube structures that persist and evolve across frames.
- The wrong mental model: the “needle-in-a-haystack” token recall test for LLMs doesn’t map to video. Long video rarely needs recalling a lone token from ages ago. And because viewpoint, lighting, scale, occlusion, articulation, motion blur, and even edits change substantially, there is no invariant single "token" to recover.
- Visual contents are physically structural: continuity, locality, bounded acceleration, and limited parallax—these drastically shrink the search space. Targets always reappear across multiple frames and move predictably.
- Compression vs Sparsity? Compression is blunt for space-time recurrence. Learnable sparsity directly routes computation to the recurring, structured signal instead of risking loss of fine but persistent cues. For visual domains, learnable sparsity might be more suitable than compression-centric strategies. But they are not orthogonal; we use some sort of naive "attention sink" in MoC, which is a form of compression, and it helps.
2. What should “memory/context/state/history” mean for long video generation or video world models?
- We want context that supports a self-evolving world state (in spirit with @ylecun 's view).
- After scaling up, merely achieving scene/character consistencies becomes a relatively trivial task. Our MoC works, Context-as-Memory works, TTT/LaCT works, nano-banana also works.
- What we need is a more expressive context ability, like the following simple behavioral test: a car enters from the left; the camera looks away; when it returns, the car should have advanced plausibly. That requires a state that evolves off-screen that enables the deduction of what is happening.
- This requires something beyond 3D caches. Pure 3D memory (geometry/appearance) doesn’t carry ongoing events through occlusions or FOV changes. We need an evolving 4D latent state tracking identity, pose, momentum, interactions, and constraints—i.e., “what’s going on” even when unseen. This also means we need more than a memory bank. Consistency of characters/assets isn’t enough; we need state transitions that continue (even while unobserved).
- This doesn't mean using an SSM, but means placing a deductive step in the model. Full attention can do it well given sufficient data since they are essentially dynamic graphs, but it becomes intractable at long contexts, so learnable sparsity matters—it's a core motivation for us to do MoC.
3. Algorithms are not a problem to handle "memory" in video world models, (video-action paired) data is.
- We largely know how to represent and route long-range visual context. The hard part is data: we need video–action/interaction-paired data that stresses long-horizon prediction: persistent identity, occlusions, off-screen dynamics, multi-agent interactions.
- This mirrors the difficult VLA challenge: scalable, high-quality interaction data is the real rate limiter for grounded state evolution and robust deduction. Luckily, we may not have that much of a Sim2Real gap under the context of Video World Models.
4. What is the role of explicit/3D then?
I side with purely implicit, data-driven approaches, so explicit/3D stuff will be in data and alignment, but not as the model's foundation.
5. The future is a unified model.
- A unified model is the most direct way to put that deduction step in the right place—the semantic representation space—and train it end-to-end.
- Borrow more, borrow better: shared representations let the model transfer motion priors, physics, and identity persistence across tasks/modalities. And it will be easier to borrow MORE stuff BETTER from the years of efforts from the LLM community ;)
- Consistent routing/compression: unified training yields stable sparsity policies (what to attend to, when, and how) across tasks.
- Richer supervision: multi-task signals sharpen the evolving latent state and improve long-horizon deduction ability.
There is still much to be done.
How do we generate videos on the scale of minutes, without drifting or forgetting about the historical context?
We introduce Mixture of Contexts. Every minute-long video below is the direct output of our model in a single pass, with no post-processing, stitching, or editing.
1/4
