Evaluation

Schemas validate. Arithmetic is exact. Allergy checks pass. The household still rejects the week: Tuesday takes too long, two dishes feel identical, a material assumption stayed hidden. Verification asks whether an exact property holds. Evaluation asks whether cognitive work meets the standard required for its purpose. The distinction matters because confusing the two produces either false precision, where teams grade what should be exact, or false safety, where exact checks stand in for judgments that require nuance.

Without evaluation, cognition cannot become reliable material or close decisions. A team that ships on demo fluency alone discovers failures in production with no record of what changed, no suite to rerun, and no way to tell whether a fix helped. This page is the canonical eval primer for the Evaluation track. Every cognitive unit and Skill note that mentions eval should link here rather than re-teach the foundations.

Helpful context: Evaluation is part of architecture states the white-paper principle. Reliability is earned conduct over time. Fluency is not evidence warns against impression-based release. Why you cannot read a cognitive unit explains why static inspection is insufficient.

Two lanes: exact checks and graders

Thoughtware evaluation runs on two lanes that answer different questions. Exact checks ask whether a defined predicate held: cashew absent from the plan, schema valid, portions equal four, unchanged days preserved after a local patch. These are binary at the boundary, cheap to run, and cheap to defend. A failing allergy check leaves no room for interpretation, because it functions as a deterministic gate, and it belongs in deterministic code.

Graders ask whether judgment met a rubric. Did the plan handle a busy weekday with appropriate active-effort budgeting? Does the variety across the week avoid repetitive textures? Is the assumption discipline visible in the plan rationale? These questions admit legitimate disagreement among competent reviewers, which is precisely why they require calibration cases and human anchors rather than regex.

The cheapest honest instrument wins. Allergy enforcement is not a vibe. Practicality scoring is not a regex. When a property can be written as a predicate with no legitimate disagreement, it belongs in the exact lane. When competent people may disagree on nuance, it belongs in a grader with a rubric and calibration set. Both lanes run in the same stack, and neither replaces the other.

Lane	Answers	Meal Companion examples
Exact checks	Did an exact property hold?	Cashew absent, plan schema valid, portions equal four, unchanged days after local patch
Graders	Does judgment meet rubric?	Weekday practicality, variety, household fit, assumption discipline

1. 01Exact checksDeterministic gates
2. 02cognitive unit gradersLeaf judgment rubrics
3. 03Skill / trajectoryProcedure and path
4. 04BehaviouralPosture and conduct
5. 05DownstreamAfter acceptance

Evaluate at several levels. Plausibility alone is never proof.

Cognitive judges as instruments

Graders score work against standards. They are fallible instrumentation: evidence for promotion and monitoring, not oracle truth. A model-based judge on weekday practicality may score 0.91 on a plan that a human-labeled busy week set would reject. The score is useful when scoped, calibrated, and paired with exact checks. It becomes dangerous when treated as ground truth, because the team stops asking what the number means and whether yesterday's provider swap shifted the rubric. Teams new to eval often collapse everything into one LLM judge. That judge becomes a black box that gates release while hiding which criterion failed and whether calibration has drifted. A judge is evidence, not truth addresses the epistemic discipline required to keep judges honest.

Evaluation restores decision locality to quality. Different failures require different repairs.

Stack depth for the Meal Companion

Evaluation depth matches where failure would hurt and where the architecture places responsibility. The exact lane covers allergy enforcement against endorsed household knowledge, schema validation on plan JSON, and shopping-list consistency after local patch. These run on every path that could surface a plan. No plan reaches the household without passing them.

The cognitive unit suite tests leaf judgments in isolation. JudgeWeekdayPracticality is evaluated on human-labeled busy weeks: Tuesday with a late meeting, Thursday with school pickup duty. Regression cases from last quarter stay green. Calibration disagreements between reviewers get triaged, not averaged away. When labelers persistently disagree on a case, the architecture responds by lowering autonomy on that class or by representing the disagreement in the output rather than forcing consensus.

Agent scenarios test full trajectories from originating sentence to accepted plan under stated cost and latency budgets. Did the agent ask targeted questions only when the gap was material? Did it abstain when authority or sufficiency failed? Behavioural checks complement these by verifying posture: targeted questions fire only on material gaps, no plan promotes chat preference into endorsed knowledge without explicit household action, and deferral copy names what was withheld. Downstream evaluation asks what happens after household acceptance. Did shopping integration and portion math remain consistent? Evaluation does not stop at the model boundary.

Cost and latency are first-class criteria in every suite, not afterthoughts bolted on after accuracy targets pass. A plan that meets every rubric but arrives after dinner fails its purpose.

Release as a gate

Nothing promotes to live use without clearing its suite. Later versions are not accepted because average score rose. They must retain calibration against human anchors and avoid known failures from production-shaped cases. Promotion gates should be written down: which exact checks block, which grader floors apply, which regression IDs must pass, what happens on provider change. Scheduled evaluation on model swap is part of architecture, not a nice-to-have.

User thumbs-up and internal "looks good" reviews belong outside the gate. They can inform which cases to add to suites. They cannot substitute for repeatable measurement against declared criteria. The distinction matters in practice: a release candidate for the Meal Companion weekly planner runs through allergy exact checks on ten endorsed restriction profiles (including trace-amount edge cases deferred to human), the JudgeWeekdayPracticality grader on forty human-labeled busy-week plans with documented rubric scope, full agent scenarios on five originating sentences including one that triggers abstention on medical diet interpretation, a behavioural check that no plan promoted chat preference without explicit household action, and a latency budget confirming the plan is ready within four minutes on reference hardware. Release blocks if the Week-14 regression (duplicate mains on consecutive Tuesdays) returns, even if mean grader score improved. Operators see eval dashboard fields: calibration age, abstention rate by class, exact-check status. There is no "100% confident" banner.

Datasets, thresholds, and ownership

Evaluation without datasets is slogans. A rubric that says "practical for busy evenings" must attach to labeled examples: what busy means in this household, what failure looks like, what abstention should produce when inputs are insufficient. The rubric gains meaning through cases, and cases gain authority through reviewers who understand the domain.

Thresholds encode consequence. A variety judge floor may differ from an allergy exact check: the latter has no floor below pass. Owners should be named for suite health, calibration age, and regression debt, the same way security patches have owners. When evaluation debt accumulates, teams ship on fluency again by accident. Treating eval debt like security debt keeps it visible, scheduled, and blocking promotion when critical regressions return.

Ownership has cadence. Name an eval owner for each production cognitive path the same way you name on-call for services. The owner maintains suite health, schedules reruns after provider or library change, and triages calibration drift. Weekly regression on critical paths, monthly full rerun on provider swap, quarterly calibration refresh on open-terrain graders. Without cadence, eval becomes launch theatre performed once and then abandoned.

Trajectory and Skill suites

When cognitive units compose into agent loops or Skills, trajectory suites test order, stop conditions, and seam behaviour. Leaf cognitive unit greens do not guarantee composed behaviour under loop pressure. A judge that passes individually may fail in a trajectory where its output feeds a downstream cognitive unit with different assumptions. Meal Companion agent scenarios fail release when the critique loop multiplies without material improvement or when targeted questions fire on non-material gaps. These trajectory-level failures are invisible to leaf suites, which is why evaluation at multiple levels matters.

Common failures

The most frequent failure is a suite built on cherry-picked prompts only. Lab fluency hides busy-week failures that production sees every month. A grader-only green on safety is a close second: the practicality score passes while the allergy exact check was never coded. One number gating everything, where a single LLM score replaces rubric, calibration, and exact lane, is a variant of the same mistake. Evaluating once at launch and never again is perhaps the most damaging, because provider drift, retrieval changes, and library substitution all require ongoing suites. Each of these failures is predictable, and each is avoidable once the architecture treats evaluation as continuous engineering rather than a one-time ceremony.

Connect eval dashboards to product metrics. Low accept rate after eval pass may signal contract mismatch with household language, not model weakness alone. That feedback loop, from production outcomes back into calibration sets, is how evaluation matures alongside the product.

What to do next

The foundations described here become concrete when applied to individual instruments. The next step is understanding what graders can and cannot claim, because treating a scored assessment as ground truth is one of the most common ways evaluation fails in practice. Exact checks, grader evidence, and human calibration each carry different epistemic status, and release policy that collapses them into one dashboard number ships while safety predicates remain unexamined.

Read why you cannot read a cognitive unit, trajectory evaluation, and downstream consequence.

Read next: A judge is evidence, not truth.