14 分钟读完llm / evaluation / agents / measurement

Bad Judges, or Missing Information?

Routing, escalation triggers, best-of-n selection — every system built around a conversational agent consumes the same upstream estimate: did this turn satisfy the user? The industry treats that as a judging problem and answers with stronger judges. We treated it as an information problem, and measured identifiability across nested information sets. The information axis moved far more than the capability axis — but the convenient reading of that result turned out to be wrong, in a more interesting way.

本文

Every router, every escalation trigger, every best-of-n selector sitting on top of a conversational agent is consuming the same estimate: did this turn satisfy the user? It is the most load-bearing number in the system and the one nobody can check.

The standard move is to treat this as a judging problem. Write a finer rubric, calibrate against humans, use a bigger base model, ensemble a few of them. All of that presumes the residual is judge error. Three things made us doubt it. Human annotators disagree on subjective judgments in ways that don't shrink with more careful guidelines, and the disagreement is itself informative. LLM judges are unstable across runs on identical inputs and sit well below the ceiling — GPT-4 agrees with humans only 63.2% of the time on Auto-J. And satisfaction is an unobservable construct, so every operationalization of it smuggles in an untested assumption.

So we asked a different question. Not can we build a better judge, but: at the moment the score is actually consumed, how much of satisfaction is identifiable from the information available?

That question has a different shape. It is a measurement question, and it has a measurement answer.

Why search solved this and agents broke it

Estimating user satisfaction is one of the oldest measurement problems on the web. Twenty years of search research — click-based implicit feedback, good abandonment, dwell time, frustration prediction — got somewhere not because someone built a great model, but because of a structural property: the user's next action happened inside the system, and it was instrumented. Satisfaction was latent, but its consequences (did they click, how long did they stay, did they rewrite the query, did they leave) were on record. Prediction was hard; identification was not.

Agents take that property away. After the agent hands over a patch, an email draft, a refactor, the satisfaction event happens outside the system. Whether the code ran, whether the email was sent — the system doesn't see it. What's left instrumented is one narrow channel: the user's next message. And most turns don't have one. Only 57.2% of WildChat sessions are multi-turn; LMSYS-Chat-1M is 50.4%. Even when the user keeps talking, only 24–30% of turns carry an explicit reaction.

This distinction — when the score is consumed — is not rhetorical. It determines what you're allowed to measure.

Some consumers are post-hoc: monitoring dashboards, offline analysis, quarterly reviews. They can legitimately read the user's reaction, because by the time they look, it exists. Other consumers are decision-time: a router picking a model for turn t, an escalation trigger, a selector choosing among candidates. At the instant they act, the reaction does not exist. For the large fraction of turns that never get a reaction at all, it never will. We study the second kind. That is why every result below is reported per information set rather than as one aggregate number — an aggregate would average over conditions that never coexist in production.

What we actually varied

A turn carries satisfaction-relevant information through four channels:

Four channels, and the moment the score gets consumed

The four channels, laid out on the timeline of a single turn. Everything to the right of the dashed line is invisible to a router, because it hasn't happened yet.

  • Q — conversation history plus the current request. Exists before the agent does anything.
  • A — the agent's full execution trace: reasoning, tool calls, tool returns, edits, reply. Exists by the time the agent finishes.
  • F — the user's subsequent reaction, within the same topical segment. At routing time, this does not exist.
  • E — external consequence: whether the produced code survived, whether it was executed. Later still, and only in some corpora.

The decision-time evidence set is C = Q ∪ A. This is not a modeling preference. When a router scores turn t, F and E are not being withheld from it; they have not happened.

We then defined identifiability the only way that makes the comparison meaningful. Let y be latent satisfaction and R the reference we score against. For an information set S:

Â(S) = sup over judges J: S → ŷ of agree(J(S), R)

— the best agreement rate achievable by any judge allowed to see only S.

Because the information sets we compare are nested, monotonicity of  is a construction, not an empirical finding: any judge that works on a small set is also a legal judge on a larger one, by pretending the extra input isn't there. No additivity assumption, no independence assumption, no variance decomposition. In Blackwell's terms, nesting is sufficiency, so the ordering holds for any reasonable scoring rule, not just the one we report.

One honesty clause, and it matters for everything downstream: we do not measure Â(S). We measure a lower bound on it — the score of the best judge we could actually build. A single increment is therefore not a bound in either direction. "This channel adds little" does not imply "this channel carries little information." That inference only goes through if you separately rule out the explanation the judge can't read it. A good part of this project exists to test that explanation, and it survived better than we expected.

The measurement grid crossed two axes:

AxisWhat it holds
InformationQ ⊂ Q+A ⊂ Q+A+F ⊂ Q+A+F+E, plus a control condition Q+Ã — the request paired with another session's trace, length- and format-matched
CapabilityDeepSeek-V4-Flash → Qwen3.5-122B-A10B → DeepSeek-V4-Pro → GPT-5.4, roughly a 4× span on external benchmarks. A 5-model ensemble is reported separately, not as a fifth tier — ensembling reduces variance, which is a different quantity from capability
Non-prompt baselinesB2: logistic regression on 17 hand-built behavioral features (turn count, tool-error rate, edit churn, retries, latency, self-correction rate). B3: linear probe on frozen Qwen3-Embedding
MetricPrimary TPR@FPR=0.10 (the routing operating point), plus PR-AUC and ECE. Splits and bootstraps (1,000 resamples) are by session, never by turn

B2 and B3 fail differently from prompt judges. Feature engineering and gradient training don't suffer from instruction-following failures or long-context degradation. Running them up the same information ladder gives us a second ladder. Where the two ladders disagree on the same turns, we can separate "not enough information" from "the instrument can't read it."

The references are their own axis

Agreement is always relative to R, so R has to be audited too. We used three references from different sources, and we call none of them ground truth.

R_vote — Arena pairwise votes. The only signal where the judge is the person involved. But it measures preference between two answers, not satisfaction with one, and preference data rewards surface properties (length, formatting); we report length correlation as a covariate.

R_human — human annotation, three annotators, Fleiss κ reported, 30% of items re-labeled a week later. Annotators see the user's reaction, which has to be stated explicitly: a reference that can't see the reaction can't adjudicate the reaction channel's value. Collection is still running, so every reaction-channel number here is an upper bound.

R_model — LLM pseudo-labels, kept only at k=5 self-consistency ≥ 4/5. It covers the whole corpus, but it demonstrates stability, not correctness. Stably wrong labels pass that filter.

Two anti-circularity rules, applied without exception. No reference may read the agent's trace, otherwise trace increments are contaminated at the source. And no cell containing F is ever scored against R_model, because R_model is generated from F. That second rule is why the reaction channel can only be reported as an upper bound.

Finally, "how small an increment counts as useless" can't be vibes. We derived a bound from routing utility: one detected dissatisfied turn is worth 1, a false alarm costs 0.1, the dissatisfied base rate is 0.2, and instrumenting the trace costs 0.005 per turn. Instrumenting pays iff π·ΔTPR > ε, i.e. δ = ε/π = 0.025. Only increments inside ±0.025 have earned the right to be called "doesn't matter."

Corpora

We chose corpora by which channels exist, not by application domain.

CorpusTraceReactionExplicit labelsOutcomeScale
Arena human preference—weak✓ pairwise votes—135,634 battles
WildChat—✓ multi-turn——826k sessions / 1.96M turns
LMSYS-Chat-1M—✓ multi-turn——1M sessions / 2.016M turns
SWE-chat✓ full agentic✓ sparse—✓ code survival5,795 sessions / 62,544 turns

The two chat corpora are not agentic, and that is their job: they hold trace richness at zero, which isolates what a full execution trace contributes when compared against SWE-chat. SWE-chat is the load-bearing column — real open-source developers working with a coding agent, with line-level human/agent code attribution. Traces cover 69.2% of turns; attribution covers 5,569 sessions, enough to carry the grid.

Explicit reaction anchors fall into five classes: thanks, cooperative correction, negative sentiment, paraphrase, topic switch with a closing move. Abandonment is not an anchor — walking away with the answer and giving up are indistinguishable in the logs. Anchor rates: 29.7% (SWE-chat), 23.7% (WildChat), 29.2% (LMSYS). Judged turns: 14,086 across 3,129 SWE-chat sessions at a 0.500 dissatisfied base rate; 998 WildChat turns (0.893); 1,246 LMSYS turns (0.806). Base rates that far apart make raw cross-corpus comparison meaningless, so we don't do it. Every cross-corpus statement below is after base-rate matching, and even then only as corroboration.

SWE-chat's selection bias deserves to be stated plainly. It keeps only sessions that ended in a commit; sessions that were abandoned outright were lost at collection. What survives is mild dissatisfaction — the developer was unhappy but still shipped. That biases SWE-chat's identifiability downward, which is one more reason every key increment is computed within the same turns, where truncation cancels.

Results

The information axis dwarfs the capability axis

Here is the full grid on SWE-chat, at the routing operating point.

Satisfaction detection by information set and judge strength

The full grid: four judges × five information conditions, on SWE-chat. Within a column the judge changes; within a row the information does.

Read it in both directions. At Q+A — the decision-time set a router can actually use — the four judges span only 0.029. Give the same judge the trace and it gains up to +0.095. Let it see the user's reaction and the number jumps to about 0.85, though that last one is an upper bound, for reasons covered below. Put the three quantities on the same turns, same metric:

  • Δ_cap (spread across four judges at Q+A) = 0.029, CI 0.026, 0.049
  • Δ_trace (Q → Q+A) = +0.032 … +0.095 depending on tier; strongest tier CI +0.074, +0.133
  • Δ_react (Q+A → Q+A+F) = +0.428 … +0.510, an upper bound for reasons that get their own section

The intervals don't overlap: 0.029 for a 4× capability span, against +0.074 to +0.133 for handing the strongest judge the trace. And the reaction channel — the one that does not exist at decision time — moves the number several times further, before we discount it.

That last clause is doing real work. Δ_react is an upper bound, and once decontaminated it shrinks to about +0.07. But even at +0.07 the reaction channel's true increment is the same order of magnitude as the trace increment, and it is permanently unavailable to the consumers we care about. The ranking of the two axes is the finding; the size of the second number is not.

The trace increment is small, but it is not zero

+0.095 sounds unimpressive until you have a ruler. The pragmatic equivalence bound derived above is ±0.025: an increment inside that band means instrumenting the trace doesn't pay back. The strongest tier's 90% CI is +0.074, +0.133 — the entire interval sits outside the band, and even the weakest tier's lower bound is above zero. By the standard of "is this worth instrumenting," the trace channel is not noise. It carries information.

Worth stating separately because it killed our own prior. We went in expecting the trace increment to land inside the equivalence band — the hypothesis being "the judges are fine, what's missing is the reaction." The data said otherwise.

The judges really are reading the trace

There was an alternative explanation for a small increment: maybe the judge ignores the trace entirely and scores from the request alone. The mismatch control rules that out. We paired each request with another session's trace, matched on length, format, and token count, so only the correspondence between request and trace is broken.

Scores fell to 0.107–0.120 — 0.14 to 0.21 below the request-only condition. Irrelevant long context actively hurts the judge. It didn't ignore the trace; it read it hard, and read it wrong. Which also means the trace increment measured above is a lower bound on the true increment, not an upper one.

Two ladders: at least half the bottleneck is the instrument

Now the experiment the whole design was built for. Same turns, same reference, two instruments with different failure modes, each climbing the same ladder.

InstrumentQQ+AQ+A+FΔ_traceΔ_react
B2 · feature regression (supervised)0.1020.2690.329+0.167 0.139, 0.200+0.060 0.045, 0.080
B3 · embedding probe (supervised)0.2120.2470.537+0.035 0.019, 0.052+0.290 0.248, 0.328
V4-Pro · strongest prompt judge0.2560.3510.849+0.095 0.074, 0.133+0.498 0.431, 0.547

Two instruments on one information ladder

The same turns, climbed by three instruments. Left: what the trace is worth. Right: what the reaction appears to be worth — an upper bound, since the judge's reference was generated from the very channel being added.

The ladders disagree, and each disagreement has a reading.

Trace channel: B2 extracts 76% more than the judge (1.8×), CIs disjoint. So "the trace doesn't carry much" is false. The information is there; the prompt judge can't pull it out. B2's features are hand-designed behavioral quantities, and gradient training doesn't degrade over a long context. B3 gets only +0.035, which is its own useful negative: extraction method matters, and not just any supervised approach will do.

Reaction channel: the judge gets +0.498, B2 gets +0.060 — the ranking flips. The reason is contamination, and it's structural. The judge's reference, R_model, is generated from F. So the judge reads the user's reaction and predicts a label that was manufactured from the user's reaction. On the 1,543 turns where both reference types exist, we can estimate the contamination directly: +0.193 against the same-source reference, +0.072 against the independent one. About 63% of the apparent increment is shared input. B2's +0.060 lines up with the decontaminated estimate, which is exactly what you'd expect from an instrument that never sees F.

Calibration: judge ECE 0.276, B2 ECE 0.045 — a 6× gap. The prompt judge's score cannot be used as a probability. B2's can.

So the picture is more interesting than the one we started with. In decision-time evidence, trace information genuinely exists and current prompt judges under-extract it; reaction information genuinely exists and is permanently unavailable at decision time. Both have direct system consequences.

Two holes we plugged in B2 before believing it. First, B2 trains on R_model and scores against R_model — isn't it just imitating the annotator? No: R_model is generated from Q+F and has never seen a trace, so trace features cannot imitate a perspective the annotator didn't have. The remaining leak path is time metadata — the gap until the user's next message is itself a property of F. Ablation: dropping the time features changes nothing (0.267 vs 0.268), and using only time features gives 0.103, i.e. chance. Second, which features do the work: the single most informative block is "did the agent produce a reply at all" (0.299 alone; removing it drops 0.268 to 0.147). Execution features — tool errors, edit churn — contribute 0.157 on their own.

The outcome channel moves the wrong way

Adding code survival (E) lowered scores at all four tiers: Δ_E ∈ −0.065, −0.031, every CI upper bound below zero.

Resist the tempting reading. Code survival is closer to correctness than to satisfaction. Adding E doesn't just add information; it shifts the construct being measured. "The code survived" and "the user was satisfied" are two different things, and conflating them actively degrades identification of the second.

Undecidability has structure

This part needs no ground truth at all. We asked five judges from five different model families to score the same input C_t and used their disagreement as a decidability signal (n = 13,863).

  • Disagreement correlates with judge error (Spearman ρ = +0.23). Low-disagreement half: TPR 0.382. High-disagreement half: 0.260. Which turns are unjudgeable is itself predictable from decision-time evidence.
  • But disagreement has no predictive power over B2's errors (ρ = −0.01; 0.258 vs 0.267 across halves). "Unjudgeable" is currently a property of the judge, not of the interaction. That also explains why feeding disagreement into B2 as an abstention score buys little (AURC 0.766 vs 0.737 for random abstention).
  • A 5-model ensemble reaches TPR 0.365 / AP 0.711, only slightly above the strongest single judge (0.352 / 0.671). Another data point on the thin returns to stacking judges.

Cross-corpus corroboration

In the two chat corpora, "trace" is just reply text — no execution records — and the pattern is similar. Q+A over Q: +0.155 +0.074, +0.299 for V4-Pro on WildChat, +0.182 +0.129, +0.254 for GPT-5.4 on LMSYS; the reaction channel again pushes scores to 0.71–0.82. These are ~1,000-turn samples with wide intervals, so they corroborate and nothing more. We don't report cross-corpus rankings before base-rate matching.

What this changes for a system

Stop optimizing only the judge. On decision-time evidence, a 4× capability span is worth 0.029. The money should go to instrumenting a feedback channel — something lightweight and low-friction that makes satisfaction observable inside the product. That is the prescription twenty years of search research actually supports, and it is the axis with the largest increment in our data.

Never report judge accuracy without the information set and the reference. 0.85 on full conversation and 0.35 on decision-time evidence are answers to two different questions. And of an 0.85 measured against LLM pseudo-labels, a large share may come from reference and input sharing a source. One sentence of disclosure prevents most misreadings.

For long traces, a prompt judge is the wrong instrument. A logistic regression on 17 hand-built features extracts 76% more from the trace, calibrates 6× better, and costs a rounding error. If a production router needs a satisfaction signal today, that route is more practical than another round of prompt tuning.

Turns you can't judge should be abstained on, not guessed. Undecidability is structured and predictable. The evaluation problem becomes a triage problem: the system's job is to know which turns it cannot call, not to call all of them.

Traps we walked into

These are about measurement itself, and they apply to anyone evaluating LLM judges.

When reference and input share a source, scores inflate by nearly 2×. A judge that reads the user's reaction, scored against a label generated from the user's reaction, shows a +0.193 reaction increment. Against an independent reference: +0.072. Without removing the shared source you are reporting recitation, not information gain.

Judge parse failures are not random. We initially dropped failed parses. They turned out to concentrate on long traces and dissatisfied labels — up to 7.4% in the worst cell — so the dropped items were exactly the ones worth keeping. After fixing the parser and re-running, every cell fell below 1.6%. Dropping failures silently biases judges upward.

Judge comparisons are extremely sensitive to tier-selection rules. On the same data, picking the "strongest judge" by sample size yields a 5.2× instrument gap; picking by trace increment yields 1.8×. Fix the selection rule before you look at the numbers, or the conclusion is a choice you made.

"Measuring satisfaction" and "measuring dissatisfaction" are two different scores. TPR@FPR and PR-AUC are asymmetric under class inversion. We once flipped label polarity and a baseline improved for free. Always state which class is positive.

What we still don't know

Human references are still being collected. Until they land, every reaction-channel number here is an upper bound, and the 63% contamination estimate is a single-point correction rather than a distribution.

"Unjudgeable" is currently a property of judges rather than of interactions. To show that undecidability is structural — a property of the turn itself — we need human labels and judge disagreement measured on the same turns.

And the cross-corpus evidence is about a thousand turns per corpus with wide intervals. Whether what we see in chat corpora transfers to agentic settings is, for now, unanswered.

If you take one thing away: before you buy a better judge, check what your judge is allowed to see, and whether it can actually read it. Those two questions were worth more than four capability tiers.