Open-Source Large Language Models Outperform Crowd Workers and Approach
  ChatGPT in Text-Annotation Tasks

Meysam Alizadeh; Maël Kubli; Zeynab Samei; Shirin Dehghani; Juan Diego Bermeo; Maria Korobeynikova; Fabrizio Gilardi

返回文献探索

Paper · arXiv 2307.02179

Open-Source Large Language Models Outperform Crowd Workers and Approach ChatGPT in Text-Annotation Tasks

Meysam Alizadeh, Maël Kubli, Zeynab Samei, Shirin Dehghani, Juan Diego Bermeo, Maria Korobeynikova, Fabrizio Gilardi

7 upvotesJuly 5, 2023arXiv 预印本

AI 摘要

Open-source LLMs demonstrate competitive performance in text annotation tasks against proprietary models and human-based services.

Large Language ModelsLLMsHugginChatFLANzero-shotfew-shottemperature parameterstext annotation tasks

Abstract

This study examines the performance of open-source Large Language Models (LLMs) in text annotation tasks and compares it with proprietary models like ChatGPT and human-based services such as MTurk. While prior research demonstrated the high performance of ChatGPT across numerous NLP tasks, open-source LLMs like HugginChat and FLAN are gaining attention for their cost-effectiveness, transparency, reproducibility, and superior data protection. We assess these models using both zero-shot and few-shot approaches and different temperature parameters across a range of text annotation tasks. Our findings show that while ChatGPT achieves the best performance in most tasks, open-source LLMs not only outperform MTurk but also demonstrate competitive potential against ChatGPT in specific tasks.