LingBot-World: an open-source video world simulator with long memory and real-time control

The paper starts from a mismatch. Many video models can make scenes look believable, but they do not reliably behave like a world with lasting objects, cause and effect, or consequences for action. That is why the authors call them dreamers rather than simulators.

Two bottlenecks matter most here. First, interactive world data are scarce, especially data that connect an agent’s decision to the environment’s response. Second, standard diffusion systems struggle to keep long clips coherent over minutes and can suffer catastrophic forgetting when the sequence gets long.

The paper’s first major move is a hybrid data engine. It combines three sources: general videos, game recordings with W/A/S/D controls, and synthetic clips rendered in Unreal Engine. That mix is meant to supply both visual variety and the kind of action-and-reaction structure that ordinary web video lacks.

The data are then filtered and labeled more carefully than a raw video dump. Clips are checked for quality, segmented in time, and given layered captions: a broad narrative, a scene-level description that separates static setting from action, and dense captions tied to exact time windows. The point is to make the same clip useful for both scene understanding and temporal control.

The model, LingBot-World, is trained in three stages because the paper is trying to solve two different problems at once: keep long videos coherent, and make them controllable in real time. Stage II stretches the model toward longer sequences so it learns memory over time rather than only short clip completion.

A mixture-of-experts design appears in the middle stage, with about 28B parameters total while keeping inference cost closer to a dense 14B model. Action control is added with inputs such as camera rotation embeddings and multi-hot keyboard actions like W, A, S, and D. The paper also describes a later shift to causal block attention and few-step distillation so the model can move from a heavy training setup toward a streaming-style generator.

The resulting system is presented as a promptable world generator. It can produce long scenes from language and discrete actions, and the paper shows examples of global changes such as winter or pixel-art style as well as local insertions like fireworks, fish, or birds.

The strongest visible claim is persistence: the authors report that landmarks can reappear after being out of view for up to 60 seconds, and they show examples of stable scenes lasting up to ten minutes. They also report a 16 fps, 480p version on one GPU node. Interpretation: if these results hold broadly, the model could be useful for interactive simulation tools, but the paper also says drift still accumulates and that the action space remains limited to navigation and basic movement.

The paper’s broader claim is not that it has solved general world simulation, but that it has moved open-source video generation closer to an interactive simulator. The open-source angle matters because the authors explicitly contrast it with a remaining gap to proprietary systems.

Its most plausible uses, based on the paper’s own examples, are interactive world-building, action-conditioned video generation, embodied AI training, and video-based 3D reconstruction. The boundary is important: memory is still emergent rather than stored in a separate memory module, multi-agent interaction is not yet supported, and high compute remains necessary.