DeCE: a review
1. The paper#
DeCE is an evaluation framework for an LLM judge. Instead of one pointwise score per answer, the judge reports precision and recall, both scored against an expert's gold standard answer. Yu, Seedat, Herrmannova, Schilder and Schwarz at Thomson Reuters, EMNLP 2025 Industry Track.
The setting is legal QA: long-form answers that have to cite the statutes, regulations and cases that apply. DeCE generates the checklist for each question from the gold standard answer.
2. The method#
Each task is a question, a model answer, and a gold standard answer written by an expert in two sections: Required Information, the content that governs the question, and Helpful Information, supporting authorities.
Recall is computed from the Required Information only. The judge turns it into a checklist of criteria, marks each 1 or 0 against the model answer, and recall is the fraction satisfied. Coverage of what must be there.
Precision is computed against the whole gold standard answer. The judge breaks the model answer into claims, marks each 1 or 0 on whether the gold standard answer supports it, and precision is the fraction supported. How much of what the model said the expert would stand behind.
Claude 3.5 Sonnet does every step. The headline metric is F2, recall weighted twice as much as precision. They give two reasons: DeCE's recall correlated with the experts better than its precision did, and gold standard answers are not exhaustive so precision is noisy.
3. Results#
They test DeCE on 224 legal questions, each with an expert's gold standard answer, and five models answer every question: GPT-4o, Gemini 2.5 Pro, DeepSeek-R1, Llama 3.1 405B and a legal fine-tune of Llama 3.1 70B trained in-house.
Four legal experts then scored model answers themselves, one expert per answer, in two ways. For 100 answers (20 questions, all five models) they scored precision and recall the way DeCE does. For all 224 GPT-4o answers they gave one overall score out of 4, the way a pointwise judge does (the paper's Table 3 lists that count as 244). Each automatic metric is then correlated with the matching human score. The comparison metrics are a pointwise LLM judge, two multidimensional judges (GPTScore and G-Eval) and RAGChecker, a claim-checking method.
| Metric | vs human precision | vs human recall | vs human F2 | vs human overall |
|---|---|---|---|---|
| Pointwise judge | 0.35 | 0.59 | ||
| GPTScore | 0.69 | 0.56 | 0.48 | |
| G-Eval | 0.66 | 0.53 | 0.48 | |
| RAGChecker | 0.62 | 0.39 | 0.38 | |
| DeCE | 0.69 | 0.80 | 0.78 | 0.46 |
Blank cells were not measured. The F2 and overall columns come from two different studies. The paper gives G-Eval F2 as 0.42 in its Table 2 and 0.48 in its Table 3; the table above uses Table 3. On F2, DeCE correlates with the experts at 0.78, against 0.35 to 0.48 for everything else.
On recall DeCE gets 0.80 and the others get 0.39 to 0.56, so criteria extracted from the Required Information track what experts count as complete far better than generic criteria do. On precision DeCE gets 0.69 and so does GPTScore, so the claim-checking half is no better than an existing judge. All of the gain is recall.
However, when experts give one overall score instead of scoring in DeCE's terms, the pointwise judge agrees with them better than DeCE does, 0.59 to 0.46, though that column is 224 GPT-4o answers and the F2 column is 100 answers across five models. DeCE matches experts best when the experts are asked to think like DeCE.
The criteria the judge writes are good. Experts reviewed all 979 of them and changed 11.95%, dropped 0.7% and added 2.0%, and 54.5% of questions needed no edits at all.
4. My thoughts#
The Required and Helpful split is a great contribution to rubric design. It encodes an inductive bias, and in the right setting, a domain where that distinction is real, it improves grader correctness. The results show it: recall against the Required criteria tracks experts at 0.80 where the generic frameworks get 0.53 to 0.56.
Precision rewards saying less, recall rewards saying more. Each caps the other's exploit.
However the gold standard answer is not exhaustive. "Supported" means the expert wrote it down, not that it is true, so a valid authority the expert did not list scores 0. The paper proposes two fixes. Flexible authority matching moves the legal judgement into the verifier, and human-in-the-loop promotion lets model outputs feed the reference over time. Both are reasonable, and both trade away some of what made the reference-based design clean.
DeCE beats the pointwise judge when the experts score precision and recall, the pointwise judge beats DeCE when the experts give one overall score, and the paper never tests which of those an expert does unprompted.
So this is a great framework where the rubric design fits: expert domains with required content and a hierarchy of supporting material, law, medicine, compliance. There it encodes the right inductive bias and it correlates better with experts.
Worth reading if you build graders for expert work.