Code-Augmented Counter-Example Verification for Auto-Graders
Integrate an automated code-execution loop using Python and SymPy to verify mathematical derivation steps and catch logical errors in auto-graders.
Source paper
Thang Luong, Dawsen Hwang, Hoang H. Nguyen, Golnaz Ghiasi, Yuri Chervonyi, Insuk Seo, Junsu Kim, Garrett Bingham, Jonathan Lee, Swaroop Mishra, Alex Zhai, Clara Huiyi Hu, Henryk Michalewski, Jimin Kim, Jeonghyun Ahn, Junhwi Bae, Xingyou Song, Trieu H. Trinh, Quoc V. Le, Junehyuk Jung
Generated by AI. The paper’s authors did not write these.
Integrate an automated code-execution loop using Python and SymPy to verify mathematical derivation steps and catch logical errors in auto-graders.
Use robustified mathematical problem reformulations as a data augmentation strategy during fine-tuning to prevent models from overfitting to specific phrasings.
Precede proof generation with a strategy classification stage to condition the solver on high-level principles before attempting algebraic manipulation.
This paper introduces IMO-Bench, a challenging benchmark suite designed to evaluate mathematical reasoning at the International Mathematical Olympiad (IMO) level, comprising tasks for short-answer verification (IMO-AnswerBench), proof writing (IMO-ProofBench), and proof grading (IMO-GradingBench). The authors demonstrate that current frontier models, while improving, still struggle with robustification (reformulated problems) and subtle logical fallacies in proofs, proposing a new gold-standard model (Gemini Deep Think) that achieves significant gains through agentic search and verification.
The researchers curated a dataset of 400 short-answer and 60 proof-based Olympiad problems, applying 'robustification' techniques (e.g., variable renaming, adding distractors) to prevent memorization. They developed 'AnswerAutoGrader' and 'ProofAutoGrader' based on Gemini 2.5 Pro to evaluate model outputs, validating these graders against human expert ratings, and used an agentic framework with self-verification loops to achieve gold-medal level performance.