Can LLMs Generate High-Quality Test Cases for Algorithm Problems?
  TestCase-Eval: A Systematic Evaluation of Fault Coverage and Exposure

Zheyuan Yang; Zexi Kuang; Xue Xia; Yilun Zhao

返回文献探索

Paper · arXiv 2506.12278

Can LLMs Generate High-Quality Test Cases for Algorithm Problems? TestCase-Eval: A Systematic Evaluation of Fault Coverage and Exposure

Zheyuan Yang, Zexi Kuang, Xue Xia, Yilun Zhao

16 upvotesJune 13, 2025arXiv 预印本

AI 摘要

TestCase-Eval is a benchmark for evaluating LLMs in generating comprehensive and targeted test cases for algorithm problems.

test-case generationFault CoverageFault ExposureLLMsalgorithm problemshuman-crafted solutionsCodeforcestest setsfailure modesincorrect code implementation

Abstract

We introduce TestCase-Eval, a new benchmark for systematic evaluation of LLMs in test-case generation. TestCase-Eval includes 500 algorithm problems and 100,000 human-crafted solutions from the Codeforces platform. It focuses on two pivotal tasks: (1) Fault Coverage, which measures how well LLM-generated test sets probe diverse input scenarios and cover a wide range of potential failure modes. (2) Fault Exposure, which evaluates whether LLMs can craft a tailored test input that reveals a specific incorrect code implementation. We provide a comprehensive assessment of 19 state-of-the-art open-source and proprietary LLMs on TestCase-Eval, offering insights into their strengths and limitations in generating effective test cases for algorithm problems.