A judge is evidence, not truth

Release review cites the LLM grader: "Plan quality 0.91, ship it." Nobody asks what the score means, whether it agrees with human-labeled busy weeks, or whether yesterday's provider update shifted the rubric. The number looked like truth. It was evidence: uncalibrated, unscoped, treated as oracle. The gap between those two words, evidence and truth, determines whether a release gate protects users or merely reassures the team.

Graders are instruments in an evaluation stack, not replacements for domain judgment, household acceptance, or exact checks. Evaluation as engineering defines the stack and its two lanes. This page defines what a grader can and cannot claim, and why the distinction matters for every team that uses scored assessment to gate releases.

Helpful context: Confidence theatre is what happens when numbers lack discipline. Why you cannot read a cognitive unit applies the same humility to leaf artifacts. Reliability is assembled from multiple instruments, not one score.

Three grader types, three epistemic statuses

Not all graders share the same relationship to truth. Collapsing them into one dashboard number is how teams ship while safety predicates remain unexamined. The table below separates the three types, but the important point is that each type demands different handling in a release gate.

Grader	Strength	Epistemic status
Deterministic	Exact properties	Truth for defined predicates
Model-based	Nuance, rubric fit	Evidence, requires calibration
Human	Domain, gold labels	Anchor for disputed criteria

A deterministic grader checks whether cashew is absent from the plan. This is boolean truth on a defined predicate once the allergy rule is endorsed. Disagreement here means the check or the data is wrong, not that judgment differs. The epistemic status is clean: if the predicate is correctly specified, the result is ground truth for that property.

A model-based grader scores whether a plan is "practical for busy Thursday." This is evidence against a rubric, not a binary fact. The score must agree with human-labeled busy weeks on the regression set to carry weight. A 0.91 does not mean the household will accept the plan. It means the plan met declared practicality criteria on declared cases as assessed by a model whose calibration may drift with the next provider update.

Human calibration anchors the other two. When a model grader and a deterministic check conflict, or when two model graders disagree, human-labeled cases provide the reference point. Gold cases exist to teach graders and bound their claims, not to annoy the release pipeline. When two competent labelers disagree on busy-week fit, the terrain may be genuinely open. The architecture should lower authority or represent disagreement rather than force a single score.

What model graders can measure

A model-based judge on JudgeWeekdayPracticality can measure fit against a written rubric: active effort within budget, ingredient prep compatible with calendar gaps, leftovers policy respected. It cannot measure whether the household's unstated preference for warm lunches changed since last month unless that preference is in working state or endorsed knowledge. The boundary of the rubric is the boundary of the claim.

Rubric scope documentation is what prevents operators from treating 0.91 as safety on allergy predicates that require exact lane pass. Every grader rubric should document in one paragraph what the score measures, what it ignores, and which exact checks must pass regardless of score. "Practicality grader green" is not "household will love this week." It is "this plan meets declared practicality criteria on declared cases." The narrower the stated scope, the more honest the evidence.

Calibration cases anchor the judge to human judgment on disputed terrain. Without them, scores drift when the provider updates weights, when prompt templates change, or when the regression set no longer matches production shape. The same score at two different points in time may describe different behaviour after a provider swap. See silent model upgrades for how invisible changes erode calibration.

A judge informs promotion. It does not replace the domain or the exact lane beside it.

Introduction to Thoughtware . Ch. 27

Pairing lanes honestly

Every model grader on consequential paths sits beside exact checks where predicates exist. Practicality may stay graded. Allergy must not. The pairing is what makes the release gate honest, because each instrument covers what the other cannot.

In the Meal Companion, a release candidate clears exact checks first: allergies, schema, and portion arithmetic all pass in deterministic code. Then JudgeWeekdayPracticality meets its floor on the human-labeled busy-week set. Regression cases from last quarter still pass. Calibration disagreements between labelers are triaged with a named owner rather than averaged into a mean score. The grader does not replace household acceptance. It measures a defined rubric on declared cases so operators can detect drift before users do.

This matters because the alternative, a single grader gating everything, hides which criterion failed and whether the failure is an exact predicate or a judgment call. An allergy violation buried inside a composite practicality score is invisible until a household member has a reaction. Separate instruments make separate failures visible, and visible failures are fixable.

Determinacy ceilings and open terrain

Open decisions have a determinacy ceiling: no cognitive unit exceeds the human agreement rate for its class. Graders measure progress toward that ceiling on covered cases. They do not abolish it. When labelers persistently disagree, the terrain is genuinely open, and the architecture should abstain, defer, or represent the disagreement rather than treat the grader as a tie-breaker oracle.

The same principle applies to monitoring. A rising mean grader score with widening variance on busy-week cases is a warning even if the floor still passes. The variance signals that the grader is confident on easy cases and unstable on hard ones. Tracking distribution shift alongside mean score catches this pattern before it reaches users. See determinacy ceiling and reliability for how evidence composes across dimensions.

Human anchors and drift

Human-labeled cases are expensive. They are also the only honest anchor for rubrics where competent people disagree. A regression set of forty busy-week plans costs reviewer time once. Reusing that set across releases turns grader scores into comparable evidence. Without anchors, 0.91 today and 0.91 next month may describe different behaviour after a provider swap, and no one notices until households reject plans that the grader passed.

Schedule recalibration when drift alerts fire: grader and human labels diverge beyond threshold on the same cases. Recalibration maintains instrumentation. Skipping it converts graders into oracle theatre, where the number persists but the meaning evaporates.

Downstream, household acceptance remains evidence of a different kind. A grader-green plan that households reject tells you the rubric or calibration set is wrong, not that households are noisy. Feed those rejects back into calibration rather than dismissing them as out of scope. That feedback loop, from production outcomes through calibration sets and back into grader tuning, is how evaluation matures.

What this looks like in practice

Team prepares JudgeWeekdayPracticality v3 for release. The calibration set contains forty plans labeled by two household-domain reviewers on busy-week shapes. Disagreement on six plans goes to escalation: three close as rubric gaps that need rewording, three are documented as open terrain with lower autonomy. The regression set holds twelve production failures from the last quarter, including the Week-14 duplicate mains bug and the Thanksgiving leftover assumption error. The exact lane runs before grader scores are considered: allergy and schema checks pass first, and a failure in either blocks regardless of grader output.

The promotion gate blocks if any regression fails, even if mean grader score rose from 0.86 to 0.91. Provider swap triggers a full rerun scheduled in the release checklist. Operators see grader score, calibration age, and exact-check status on one dashboard. Household acceptance remains a separate signal tracked downstream, feeding back into calibration when accepted plans diverge from grader predictions.

Common failures

The most frequent failure is one grader ruling everything. Practicality, safety, tone, and assumption discipline need different instruments and different calibration sets. A composite score that merges them hides which dimension failed and whether the failure matters for safety or for preference. An uncalibrated model judge is a close second: scores move with provider updates while release thresholds stay fixed, producing false greens and false reds that erode trust in the gate itself.

Treating human disagreement as noise is subtler but equally damaging. When two competent labelers disagree on busy-week fit, the terrain may be genuinely open. Forcing consensus through averaging produces a score that neither labeler endorses. Skipping the exact lane is perhaps the most dangerous variant: the grader says "looks safe" while allergy enforcement was never coded. Fluency and rubric fit are not substitutes for predicates. And oracle marketing, where customer-facing copy cites grader scores as accuracy percentages, is confidence theatre rather than evaluation.

When any of these appear in release policy, the right response is to treat them as eval debt and fix the gate before scaling autonomy. Grader disagreement surfaced in monitoring, with widening disagreement bands visible to operators, prevents each of these failures from compounding silently.

What to do next

The discipline described here, treating graders as scoped evidence rather than oracle truth, has a direct consequence for how teams operate in production. The next concept addresses the cultural and product shift that follows: replacing pretend certainty with monitored evidence, structured abstention, and continuous evaluation as an operating norm rather than a launch ceremony.

Read evaluation as engineering, evaluation replaces certainty, and why you cannot read a cognitive unit.

Read next: Evaluation replaces certainty.