|国家预印本平台
首页|Metritocracy: Representative Metrics for Lite Benchmarks

Metritocracy: Representative Metrics for Lite Benchmarks

Metritocracy: Representative Metrics for Lite Benchmarks

来源:Arxiv_logoArxiv
英文摘要

A common problem in LLM evaluation is how to choose a subset of metrics from a full suite of possible metrics. Subset selection is usually done for efficiency or interpretability reasons, and the goal is often to select a ``representative'' subset of metrics. However, ``representative'' is rarely clearly defined. In this work, we use ideas from social choice theory to formalize two notions of representation for the selection of a subset of evaluation metrics. We first introduce positional representation, which guarantees every alternative is sufficiently represented at every position cutoff. We then introduce positional proportionality, which guarantees no alternative is proportionally over- or under-represented by more than a small error at any position. We prove upper and lower bounds on the smallest number of metrics needed to guarantee either of these properties in the worst case. We also study a generalized form of each property that allows for additional input on groups of metrics that must be represented. Finally, we tie theory to practice through real-world case studies on both LLM evaluation and hospital quality evaluation.

Ariel Procaccia、Shirley Zhang、Benjamin Schiffer、Serena Wang

计算技术、计算机技术

Ariel Procaccia,Shirley Zhang,Benjamin Schiffer,Serena Wang.Metritocracy: Representative Metrics for Lite Benchmarks[EB/OL].(2025-06-11)[2025-07-25].https://arxiv.org/abs/2506.09813.点此复制

评论