PanGu-Coder2: Boosting Large Language Models for Code with Ranking
  Feedback

Bo Shen; Jiaxin Zhang; Taihong Chen; Daoguang Zan; Bing Geng; An Fu; Muhan Zeng; Ailun Yu; Jichuan Ji; Jingyang Zhao; Yuenan Guo; Qianxiang Wang

Paper · arXiv 2307.14936

PanGu-Coder2: Boosting Large Language Models for Code with Ranking Feedback

Bo Shen, Jiaxin Zhang, Taihong Chen, Daoguang Zan, Bing Geng, An Fu, Muhan Zeng, Ailun Yu, Jichuan Ji, Jingyang Zhao, Yuenan Guo, Qianxiang Wang

42 upvotesJuly 27, 2023arXiv 预印本

AI 摘要

PanGu-Coder2, a model fine-tuned with RRTF framework, demonstrates superior code generation performance, outperforming existing Code LLMs on multiple benchmarks.

RRTFpre-trained large language modelscode generationOpenAI HumanEvalCoderEvalLeetCode benchmarks

Abstract

Large Language Models for Code (Code LLM) are flourishing. New and powerful models are released on a weekly basis, demonstrating remarkable performance on the code generation task. Various approaches have been proposed to boost the code generation performance of pre-trained Code LLMs, such as supervised fine-tuning, instruction tuning, reinforcement learning, etc. In this paper, we propose a novel RRTF (Rank Responses to align Test&Teacher Feedback) framework, which can effectively and efficiently boost pre-trained large language models for code generation. Under this framework, we present PanGu-Coder2, which achieves 62.20% pass@1 on the OpenAI HumanEval benchmark. Furthermore, through an extensive evaluation on CoderEval and LeetCode benchmarks, we show that PanGu-Coder2 consistently outperforms all previous Code LLMs.