Scales++: a review
1. The paper#
Scales++ is a method for picking a small subset of a benchmark that predicts the full score. Bean, Seedat, Chen and Schwarz at Thomson Reuters Foundational Research, May 2026.
The existing methods for this are model-centric. Anchor Points, tinyBenchmarks and metaBench all pick items by looking at how past models did on them, which means you need data on model performance before you can start, you cannot subset a new benchmark at all, and you are assuming future models will fail where past models failed. The paper's proposal is to pick items on their intrinsic properties instead, which they call item-centric.
2. The method#
Every item's text is annotated by GPT-4o on 16 cognitive demand dimensions (logical reasoning, specific knowledge areas and so on), each scored 0 to 5. They drop the dimensions with no variance, reduce the 16 to 3 with UMAP, run k-means with k equal to the budget, and keep the item nearest each centre.
The target model is evaluated on those k items only. The full score is then predicted twice. The first estimator is a cluster-size weighted average of the k scores. The second fits 16 logistic regressions of score against demand, one per dimension, and predicts every unselected item from those. The two are blended with a weight from Song and Schmeiser that is optimal given the bias of the second and the variance of the first.
Scales++ Lite replaces the GPT-4o annotator with a GNN on Qwen2.5-7B embeddings, trained on 8,000 Tulu3 items. It annotates the whole Open LLM Leaderboard in under 20 minutes.
3. Results#
On the Open LLM Leaderboard a 0.25% subset predicts the full score with 3.2 points of mean absolute error. Random selection is 1.0 worse and gp-IRT is 0.3 worse, at 18x the setup cost. On Humanity's Last Exam, where no item-level history exists so IRT cannot run at all, they evaluated six frontier models themselves and a 2% sample gets 2.9 MAE against 3.2 for random.
Per benchmark it is more mixed. Their words: similar (within 2% MAE) or better than all other methods in roughly 70% of cases. TruthfulQA is bad for it. On GSM8K only 52.3% of items have a unique embedding, against 84% or more on the other benchmarks and 96.6% on TruthfulQA, and they suggest embedding density as a meta-property for predicting where the method will work.
4. My thoughts#
The item-centric argument is right, and I am surprised no one had taken it before. Model-centric subsets are being actively invalidated. Post-training teams target the failures of previous models, so the items chosen because past models failed them are exactly the items the next round of training fixes first, which breaks the assumption model-centric methods depend on.
My read of the results is that it works at very small sample sizes and random catches up. So the real value is not cheap leaderboard numbers, it is steering training. Running every benchmark at every checkpoint is too expensive and risks overfitting to the eval suite, so you validate on a subset during training and hold out unseen benchmarks for the finished model. This is what Thomson 1.0 does with CapTrack. A caveat from the authors, which I take, is that the crossover with random is an empirical finding and not a law, and it will move with the benchmark set.
The annotations are GPT-4o judgments, and the validation for them is borrowed. It comes from the General Scales paper, where a Delphi consensus of five human raters agreed with GPT-4o at 0.75 to 0.94 across dimensions. That validation holds for one annotator on one set of benchmarks. If you change the annotator, it no longer applies, which the authors agree with and would fix with a new human study. If you change the domain, it no longer applies either, even with the same GPT-4o. The paper reports that only 52.3% of GSM8K items have a unique embedding, against 84% or more on the other benchmarks. The authors read that as embedding density rather than domain, and Scales++ still beats IRT there. My read is that 16 general dimensions cannot tell maths problems apart from each other, and any single specialised domain will look the same to them. In conversation with the authors, the answer to that was that for a new domain they would redesign the 16 around the atomic skills you actually care about, and that a per-domain, per-dimension agreement study would show exactly where the scales have validity, especially the levels. I think that is the right future work and it is what I would want before using the scales on a specialised domain.
Lite is elegant, but it has a cost. It distils the annotator, so any annotator noise is trained into the GNN, and the rationale GPT-4o was required to give is traded for a bare class. In exchange a frozen annotator is deterministic and versionable, which an API model is not.
Worth reading. It is a good use of data-centric ML pointed at evals, and the authors were generous with their time when I asked about it.