IOPO: Empowering LLMs with Complex Instruction Following via
  Input-Output Preference Optimization

Xinghua Zhang; Haiyang Yu; Cheng Fu; Fei Huang; Yongbin Li

返回文献探索

Paper · arXiv 2411.06208

IOPO: Empowering LLMs with Complex Instruction Following via Input-Output Preference Optimization

Xinghua Zhang, Haiyang Yu, Cheng Fu, Fei Huang, Yongbin Li

21 upvotesNovember 9, 2024arXiv 预印本

AI 摘要

TRACE is a benchmark for improving complex instruction-following in LLMs, using IOPO to enhance input-output alignment, demonstrating superior performance over SFT and DPO.

large language models (LLMs)TRACEbenchmarkcomplex instruction-followingIOPO (Input-Output Preference Optimization)in-domainout-of-domainSFTDPO

Abstract

In the realm of large language models (LLMs), the ability of models to accurately follow instructions is paramount as more agents and applications leverage LLMs for construction, where the complexity of instructions are rapidly increasing. However, on the one hand, there is only a certain amount of complex instruction evaluation data; on the other hand, there are no dedicated algorithms to improve the ability to follow complex instructions. To this end, this paper introduces TRACE, a benchmark for improving and evaluating the complex instructionfollowing ability, which consists of 120K training data and 1K evaluation data. Furthermore, we propose IOPO (Input-Output Preference Optimization) alignment method which takes both input and output preference pairs into consideration, where LLMs not only rapidly align with response preferences but also meticulously explore the instruction preferences. Extensive experiments on both in-domain and outof-domain datasets confirm the effectiveness of IOPO, showing 8.15%, 2.18% improvements on in-domain data and 6.29%, 3.13% on outof-domain data compared to SFT and DPO respectively.