GDPval: a review
1. The benchmark#
GDPval measures AI's capability on real world economically valuable tasks. It covers the top 9 sectors contributing to U.S. GDP, with at least 30 tasks per occupation in the full set. They use human experts to author the tasks, and their primary metric of evaluation is head-to-head human expert comparison. They open-source a 220-task subset of the benchmark, and host an experimental automated grader alongside it.
2. Taskset#
They take every sector (9) contributing over 5% of US GDP, measured as value
added, within those they take the highest earning knowledge work occupations, and
keep an occupation only if more than 60% of its O*NET tasks, weighted
by relevance and importance and frequency, are classified as digital. That
classification is done by GPT-4o. Experts are then brought in to author realistic
tasks against the surviving list.
Using experts to author the tasks is great, and so is building the list off documented occupation and industry data rather than inventing a taxonomy.
However there are construct validity issues. Value added counts capital income + imputed rents, not income earned, therefore the weighting is sensitive and this analytical choice is not justified (Real Estate comes out top at 13.8% of GDP while its five occupations earn about $62B. Professional, Scientific and Technical Services sits at 8.1% and earns $742B).
The occupation boundary is drawn by a GPT-4o classifier whose item level accuracy is never reported, at a 0.60 threshold that is never justified.
3. Grader#
Grading is blinded expert pairwise comparison against the human deliverable, with three completions per model and three graders per task. They also host an experimental automated grader, which they do not treat as a replacement.
Open sourcing the gold taskset is great, and expert pairwise comparison is high quality grading. It is slow and expensive and they paid for it.
Tracing the taskset on HuggingFace I found 4 bugs across 15 tasks: a rubric whose total contradicts its own arithmetic, a rubric checking different spreadsheet columns than the prompt specifies so the expert deliverable fails its own rubric, a prompt and rubric anonymising the same party differently, and an expert chart plotting different figures than the rubric asks for. Every quality check they run is done by reading, none by running the rubric.
The reward design is inconsistent, most criteria are worth 1 point each whatever they are, and a judge only +5 for overall formatting and style appears in 9 of the first 10 tasks I read and is usually the largest single item. Some authors did encode value in the weights, +5 for conclusions and +4 for intermediate steps, so it can be done. Also, ~90% of the rubric could have been executed in code.
4. Analysis#
Their analysis attempts to explain what the model performance means economically. Three scenarios: a naive ratio of model cost against human cost, and two that account for a professional reviewing the output and redoing it themselves when it falls below their bar. For GPT-5 the naive cost improvement is 474x. Once you include review and redo it is 1.18x.
| Model | Win rate | Naive | Try 1x | Try nx |
|---|---|---|---|---|
| gpt-4o | 12.5% | 5172x | 0.90x | 0.53x |
| o4-mini | 29.1% | 1265x | 1.06x | 1.22x |
| o3 | 35.2% | 480x | 1.13x | 1.47x |
| gpt-5 | 39.0% | 474x | 1.18x | 1.63x |
Naive is model cost against human cost. Try 1x and
Try nx add the professional's time reviewing the output and redoing it
when it falls below their bar. For gpt-4o the realistic figures are below 1, meaning
using the model leaves you worse off than not using it.This is brilliant, the point of a benchmark like this is the economic implication of model performance, and this is the section that actually gets at it.
IMO if they wanted the full picture of value they should have factored in the cost of failure. They say themselves that the analysis does not capture the cost of catastrophic mistakes. GPT-5 fails 61% of the time and roughly 3% of those failures are rated catastrophic, so the saving only holds if nothing expensive goes wrong. For a lawyer or an auditor that is not a safe assumption. The cost analysis also only covers OpenAI models.
5. Conclusions#
GDPval is a good paper + benchmark in an area of evals that has not had enough attention. I think the expert authored taskset is brilliant, and pricing model performance economically is the right idea. However both the grader + the analysis can be improved in terms of efficiency, validity and rigor.
I would definitely recommend reading it.