Series note. To mark the completion of my ECE 5864 Critical Engineering capstone, I’m publishing a seven-part adaptation for a special week of ITE 135: AI Awareness. These posts are longer and more formal than my usual entries because they began as a graduate research paper. This is Part 2 of 7.
Technical Summary
A. Introduction
In “Understanding World or Predicting Future? A Comprehensive Survey of World Models,” Jingtao Ding and eleven colleagues at Tsinghua University (published in ACM Computing Surveys 58, 3, 2025) provide the first comprehensive peer-reviewed survey of world-model research. Its central move is to resolve the term’s contested definition by organizing the entire literature around two functions: i. building internal representations that help a machine understand the world and ii. predicting future states of the world to guide decisions.
B. Description
The problem that Ding et al. address is that “world model” has come to mean different things to different research communities, all hiding beneath the marketing hype. One lineage, descending from Ha and Schmidhuber’s 2018 work and from model-based reinforcement learning, treats a world model as a compressed internal representation that an agent consults to make decisions. Ha and Schmidhuber used learned latent representations to produce the first reported solution of the CarRacing benchmark. In a separate VizDoom experiment, they trained a policy entirely inside an environment generated by the model and transferred that policy successfully to the original game (Ha and Schmidhuber 2018).
A second lineage, energized by video generators such as Sora and FLUX, treats a world model as a simulator that renders plausible future states of the physical world. The “Joint Embedding Predictive Architecture,” a.k.a. JEPA, proposed by Yann LeCun, former Chief AI Research Scientist at Meta, argues that a machine should both represent the world abstractly and use that representation to imagine outcomes before acting. The embedding in that name points to a mechanism both lineages increasingly share: instead of learning only from text data or reproducing every image pixel, many current systems compress video, lidar, motion, and other sensor data into a ‘latent’ or abstract space, which consists of numerical vector representations intended to preserve important event information and dynamic relationships while omitting overwhelming raw detail. Prediction then happens in that compressed space rather than at the pixel level, which is what makes the approach computationally tractable at scale, but also what makes its omissions consequential.
The authors conduct a systematic review in three passes. First, they survey the internal-representation branch: world models inside model-based reinforcement learning agents, and the more recent finding that large language models (LLMs) appear to acquire latent world knowledge, including spatial and temporal structure, without being explicitly trained for it. Second, they survey the future-prediction branch, tracing a progression from video generation toward interactive embodied environments, from producing footage of the world to producing worlds an agent can act in. Third, they examine three application domains (autonomous driving, robotics, and social simulacra), showing that each domain draws on their two functions in different proportions. The authors attempt to unify these two fields as two functions of one underlying idea: understanding the present and predicting the future.
The paper closes with an open problem and findings that are largely negative. Current video generators achieve visual realism but fail systematic tests of physical law: the authors cite diagnostic studies showing errors in gravity, fluid, and thermal dynamics, and evidence that scaling produces case-based rather than rule-based generalization. So far world models ‘memorize’ situations without inducing the physics beneath them. Finally, the authors document that evaluation is fragmented: because the field pursues divergent goals with heterogeneous methods, no canonical benchmark or metric for “being a good world model” yet exists.
References
Ding, Jingtao, Yunke Zhang, Yu Shang, Yuheng Zhang, Zefang Zong, Jie Feng, Yuan Yuan, et al. 2025. “Understanding World or Predicting Future? A Comprehensive Survey of World Models.” ACM Computing Surveys 58 (3): Article 57.
Ha, David, and Jürgen Schmidhuber. 2018. “Recurrent World Models Facilitate Policy Evolution.” In Advances in Neural Information Processing Systems 31 (NeurIPS 2018), 2450–2462.
Leave a Reply