How AI Models Are Evaluated for Language Understanding

This appendix details how researchers screened English-speaking participants, piloted survey designs, and compared Google and OpenAI language models (LaMDA, PaLM, Flan-PaLM, GPT-3.5, GPT-4) under different prompt conditions. Findings show consistent model performance across prompt types, with GPT-4 and Flan-PaLM outperforming others on reasoning and factual tasks. The study highlights methodological challenges, such as token biases and API differences, while emphasizing fair human-to-AI comparison.