ECBD: a review
1. The paper#
ECBD is a framework for designing and analysing benchmarks. The authors take a framework from educational testing, Evidence-Centered Design (Mislevy et al. 2003), and map it to benchmark design.
2. The framework#
The framework splits benchmark construction into intended use, which comes first, and then five modules.
Their roles, quoted from Figure 2:
- Intended use (precedes the modules): “What are the intended objects of evaluation? Who are the intended users of the benchmark? How should the users interpret and use the benchmark results?”
- Capability: “Connection between the benchmark and its intended use.”
- Content: “Each item elicits capability evidence about the capabilities it targets.”
- Adaptation: “Adaptation methods are well-suited for all obj. of eval.”
- Assembly: “Selected set elicits sufficient evidence to measure the capabilities.”
- Evidence: “Extracted evidence captures the capabilities targeted by the item”, then “Accumulated evidence captures the capabilities of interest.”
For each module you have to describe, justify and support.
3. The worksheet#
They convert the framework into a worksheet of 20 questions, available at github.com/isle-dev/ECBD.
They caveat it:
- The questions are not exhaustive, and they expect practitioners will identify additional questions relevant to their own benchmark.
- Validity is not the only thing that matters. They name provenance, privacy and reliability as things a benchmark should be assessed on that ECBD does not cover.
- They analysed three benchmarks, and only from the papers introducing them, not the websites or code repositories.
4. Their findings#
- Intended use is vaguely specified.
- Complex capabilities get decomposed into subcapabilities with little elaboration.
- Abstractions are not separated from the metrics used to measure them.
- Re-purposed data is not connected / adapted properly to the capability.
- Evidence is often unjustified, or justified by precedent (however even new metrics go unjustified)
The overarching finding is the one that matters: benchmarks describe their design choices, justify them less often, and rarely gather validity evidence at all.
5. My thoughts#
I used this framework to formalise a benchmark on professional work. It helped with rigor in construction, and with the interpretability of the benchmark. Justifying my capability choices against the intended use exposed that I had never argued why those capabilities and not others, which is a real gap and I would not have found it by reading my own draft.
However ECBD is pre-agent and general. It is from June 2024 and the object of evaluation it imagines is a language model that takes text in and produces text out. It was not robust, specifically in my experience, to harness-related adaptation items.
So I think ECBD should be seen as a reference framework which needs tweaking and adapting for each use case, rather than something you apply as-is. The authors say this themselves in their limitations, that the questions are not exhaustive and that practitioners will find additional ones specific to their own benchmark.
This is a part of benchmarking that industry currently underappreciates, formal frameworks pre-construction. The gap is not documentation, HELM documents heavily. The gap is justification and validity evidence, and ECBD is the best starting point I've read for closing it.
Worth reading.