Arxiv paper - The BiGGen Bench: A Principled Benchmark for Fine-grained Evaluation of Language Models with Language Models

In this episode, we discuss The BiGGen Bench: A Principled Benchmark for Fine-grained Evaluation of Language Models with Language Models by Seungone Kim, Juyoung Suk, Ji Yong Cho, Shayne Longpre, Chaeeun Kim, Dongkeun Yoon, Guijin Son, Yejin Cho, Sheikh Shafayat, Jinheon Baek, Sue Hyun Park, Hyeonbin Hwang, Jinkyung Jo, Hyowon Cho, Haebin Shin, Seongyun Lee, Hanseok Oh, Noah Lee, Namgyu Ho, Se June Joo, Miyoung Ko, Yoonjoo Lee, Hyungjoo Chae, Jamin Shin, Joel Jang, Seonghyeon Ye, Bill Yuchen Lin, Sean Welleck, Graham Neubig, Moontae Lee, Kyungjae Lee, Minjoon Seo. The paper introduces BIGGEN BENCH, a comprehensive benchmark designed to evaluate nine distinct language model capabilities across 77 diverse tasks with instance-specific criteria that better reflect human judgment. It addresses limitations of existing benchmarks, such as abstract evaluation metrics and coverage bias. The authors apply BIGGEN BENCH to assess 103 advanced language models using five evaluator models, making all resources publicly accessible.