Why CriticBench Refuses GPT & LLaMA for Data Generation

CriticBench uses Google’s PaLM-2 model family to generate benchmark data for tasks like GSM8K, HumanEval, and TruthfulQA. By avoiding GPT and LLaMA due to licensing constraints, the project ensures a more open and compliant evaluation framework. Its methodology employs chain-of-thought prompting, code sandbox testing, and principle-driven prompting to create high-quality responses that capture both final answers and underlying reasoning, making it a valuable resource for critique-based AI evaluation.