Measuring what Matters: a review
1. The paper#
Bean et al. review 445 LLM benchmarks for construct validity, with 29 expert reviewers. Construct validity is the degree to which a benchmark score is evidence for the claim being made about the phenomenon. They break a benchmark into phenomenon, task, metric and claim.
They got to 445 from 46,114 papers at ICML, ICLR, NeurIPS and ACL venues, via a keyword filter, a GPT-4o-mini screen and expert review. Each benchmark was coded on a 30-field codebook.
They also measured their own reviewers. On a 46 paper double-review the mean agreement was BPK 0.524, and on the most judgment heavy field, task ecology, it was 0.146.
2. The checklist#
They turn the review into 8 recommendations and a 28 item checklist, Appendix A. The eight, quoted:
- Define the phenomenon
- Measure the phenomenon and only the phenomenon
- Construct a representative dataset for the task
- Acknowledge limitations of reusing datasets
- Prepare for contamination
- Use statistical methods to compare models
- Conduct an error analysis
- Justify construct validity
3. Findings#
- 47.8% of benchmarks measure a phenomenon that is contested or undefined.
- 61.2% measure a composite phenomenon, and most of those still report a single number.
- 27.0% used convenience sampling at least partly, 12.3% exclusively. 32.3% use any real world tasks.
- Exact match is the metric in 81.3%. LLM as judge 17.1%. Human ratings 13.0%.
- 16.0% do any statistical testing.
- 53.4% present any evidence for construct validity. 32.4% include a human baseline.
Clearly, existing benchmarks are flawed in terms of construct validity, and this serves as evidence for a call for greater rigor in evaluations.
4. My thoughts#
I ran the full 28 item checklist on GDPval, which postdates their dataset. GDPval came out at 11 pass, 9 partial, 4 fail, 4 not applicable. It sits near the top of their dataset on most columns.
However, the checklist missed a key criticism of GDPval in that the rubrics were flawed when executed. So I think it can be adapted on a case by case basis for auditing benchmarks like GDPval, the structure holds but the items need extending for the benchmark in front of you.
Some items in their codebook are heavily contested between reviewers. Face validity has a kappa of 0.927, but phenomenon defined is 0.073, sampling method 0.122 and task ecology 0.146. Two experts reading the same paper disagree on whether the phenomenon is defined at all. Maybe benchmark construction is inherently ambiguous for non-trivial work, and the checklist should be read with that in mind, a pass on a 0.07 item is one person's opinion.
The validity audit is good, but validity is orthogonal to other dimensions such as reliability and reproducibility, which are also necessary for quality evaluations. Exact match is maximally reliable and contributes nothing to validity, and a benchmark can be the other way round.
Saturation is often treated as a failure. In terms of validity, reproducibility and contamination it is not, assuming the benchmark and the experiment are of high quality. It is only a loss of discriminative power. Lots of people discard the results of saturated benchmarks and only care about the ones that still separate models, which IMO is wrong, a saturated high quality benchmark is telling you the capability is there.
I also wonder how it holds up on industry benchmarks compared to academic ones. The dataset is conference papers to 2024, and only 33 of 453 are industry authored. One data point, but it suggests the failure modes might be different, and the checklist was built from the academic ones.
Worth reading, and worth using to audit benchmarks.