Markov Chains, Rewards & Rules

This article explores LLM-Sim, a benchmark designed to test whether large language models can serve as “world simulators” in text-based environments. By framing the problem as a goal-conditioned partially observable Markov decision process (POMDP), the study evaluates how LLMs model both action-driven and environment-driven transitions, track object properties, and assess game progress. Using human- and AI-generated context rules, the research measures prediction accuracy across object states and rewards, providing insight into how well LLMs can reason about dynamic systems beyond simple text prediction.