Are Large Language Models the Future of Game State Simulation?

Large language models hold promise as simulators of virtual environments, but new benchmarking with BYTESIZED32 shows that even GPT-4 falls short. While LLMs can generate plausible outcomes, they often fail at capturing complex state transitions requiring arithmetic, common sense, or scientific reasoning. This research highlights both their potential and current limitations, offering a novel benchmark for tracking progress as models evolve.