Posts
-
September 9, 2026
A review of SCHEME, a benchmark for coordinated sabotage by teams of AI coding agents, and what I would test first.
-
September 7, 2026
A review of DeCE, an evaluation framework that has an LLM judge report precision and recall from criteria extracted from an expert's gold standard answer. The Required and Helpful split is a great piece of rubric design and the recall half tracks experts at 0.80, but the gold standard answer cannot be exhausted and the 0.78 headline is agreement with experts asked to think in DeCE's terms.
-
September 7, 2026
A review of Scales++, an item-centric method for picking a tiny benchmark subset that predicts the full score. It subsets benchmarks that model-centric methods cannot touch, at 18x lower setup cost, and works best exactly where random sampling is worst. The open questions are the annotator's borrowed validation and whether 16 general dimensions resolve inside a single domain.
-
September 6, 2026
A review of Bean et al.'s construct validity checklist, built from 445 LLM benchmarks. I ran all 28 items on GDPval and it came out 11 pass, 9 partial, 4 fail.
-
September 5, 2026
A review of ECBD, a framework for constructing benchmarks taken from educational testing. I used it to formalise a benchmark on professional work, which aids rigor and interpretability in benchmark construction, however it is pre-agentic and domain agnostic therefore it needs to be adapted case-by-case.
-
September 4, 2026
A review of OpenAI's GDPval. The task creation is strong, but the grader is flawed and the analysis could be extended. Tracing the gold taskset on HuggingFace turned up 4 bugs in 15 tasks.
-
September 4, 2026
An RL environment which trains a model to write test suites for functions which operate on structured data. 20 steps of GRPO took a 4B from 0.41 to 0.88.