Why GPT-4 Struggles with Complex Game Scenarios

This study evaluates GPT-4’s ability to simulate game state transitions in the LLM-Sim task. Results show GPT-4 performs best on action-driven and static transitions but struggles with environment-driven dynamics, arithmetic, and common-sense reasoning. While GPT-4 can predict game progress with high accuracy when given rules, it still lags behind humans, who achieve ~80% accuracy compared to GPT-4’s ~50% in challenging cases. Findings highlight both the promise and current limitations of LLMs in complex simulation tasks.