Evaluation is part of architecture

A team ships AssessMealPracticality with a polished template and a confident demo. Tuesday looks fine in the meeting. Production traffic includes weeks the demo never exercised: three busy evenings, a leftover preference, a temporary pasta avoidance that must not become durable knowledge. Households reject plans. Support reads the template, sees reasonable instructions, and concludes users are picky.

Reading the template did not answer the operational question. Behaviour on labeled cases would have. White Paper Principle 6 insists evaluation is part of architecture: designed alongside the responsibility, bound to cognitive unit identity, not bolted on after launch in a spreadsheet labeled QA.

This page bridges the cognitive unit track to the Evaluation section. Evaluation as engineering is the Evaluation-track primer. Reliability is not a confidence score explains what evaluation produces. What is a cognitive unit shows where hooks attach.

Why cognitive contracts need designed measurement

Deterministic code invites ordinary tests: given inputs, expect outputs. Cognitive contracts give up that certainty by design. More than one defensible answer may exist. Behaviour drifts when models change even if templates stay fixed.

Without designed evaluation, teams revert to vibes: demo applause, anecdotal support tickets, leaderboard metrics that reward fluency over fit. Vibes-based quality assessment feels efficient until a regression ships that no metric detected, and the team discovers that anecdotes lack the stratification needed to isolate which case class degraded. Architecture cannot be maintained on those signals. A capability is architecturally incomplete if there is no defined way to determine whether it performs well on the case classes that matter. For AssessMealPracticality, that means labeled busy evenings, agreed rubrics, stratified figures, and suite records tied to template version and model pairing. "We will monitor complaints" is not a suite.

Hooks belong in the contract

Evaluation is not a separate repository that vaguely references feature names. It lives in the contract through several concrete elements.

Suite pointer. Which labeled cases and graders apply to this identity version. Promotion criteria. What figures must hold before a library entry advances. Case class stratification. Busy evening, allergy edge, leftover request, and similar buckets drawn from judgment terrain. Pairing discipline. Template version plus model pairing plus date. Substrate shifts under stable templates are identity events (evidence dies when the template changes).

Identity changes rebuild baselines. Policy overrides, such as a cheaper screening model, remeasure when stakes are high. Callers depend on published figures at the interface, not on tribal knowledge in chat.

What each Meal Companion judge needs

Judges and critics need calibration, not single-number vibes.

cognitive unit	Evaluation question
JudgeLocalRepairQuality	Did the patch fix the weakness without breaking accepted meals
CritiquePlan	Does critique correlate with human rejection reasons
AskTargetedQuestion	Was the question the smallest useful acquisition
JudgeWeeklyVariety	Does accepted variety match household intent on labeled weeks
AssessMealPracticality	Busy evening accuracy versus labeled expectations

Without JudgeLocalRepairQuality, loop compression is blind. The agent patches until something looks fine. Loop compression without judges trades cost for silent quality loss.

Exact checks still belong in deterministic code: allergies, portions, schema validity. Evaluation spans both exact and graded layers. Architecture specifies both.

Evaluation is part of architecture.

Thoughtware White Paper · Pr. 6

Evaluation connects to encapsulation and libraries

Encapsulation makes evaluation economical. Suites attach to library identity. When AssessMealPracticality v2 promotes, figures travel with the entry. Call sites do not rebuild ad hoc graders.

Library semantic versioning without evaluation is packaging theater. Promotion gates exist to prevent silent behaviour change across products sharing a domain library. Invoice intake parallels the meal example in enterprise material: extraction and classification cognitive units carry suite pointers before orchestration code depends on them. Financial consequence raises the cost of missing hooks.

What this looks like in practice

Before launch, the team prepares labeled weeks drawn from grounding scenarios, including busy patterns and spinach deadlines. Rubrics for practicality and variety align with endorsed household knowledge. Baseline runs on v1 identity produce published figures per class. Blocking rules prevent promotion when busy-Tuesday accuracy regresses beyond tolerance.

After a model vendor update with no template diff, scheduled remeasurement runs automatically. Figures shift. The team treats the event as substrate change, not as "users became picky."

Review still matters for clarity, hole design, and policy language in templates. Review does not replace measurement. Why you cannot read a cognitive unit explains the epistemic gap reading cannot close.

Graders versus exact checks

Graders score open judgments with rubrics and labeled cases. Exact checks validate schemas, arithmetic, and permission gates. Architecture specs list both layers per feature. A plan patch might pass schema validation while failing JudgeLocalRepairQuality. Both results matter.

Teams new to evaluation often build exact tests first because they feel familiar. That is necessary and insufficient. Busy-evening fit requires graders because correctness is contextual. Principle 6 asks for both layers in the same contract review, not a phased QA roadmap that never reaches graders.

Relationship to human judgment

Human nodes participate in evaluation too. Measure approval screen effectiveness. Measure time-to-accepted-plan after targeted questions. Principle 6 applies to the whole architecture, including human nodes and deterministic gates, alongside model calls.

Human judgment remains (human judgment remains part of the system) where authority requires it. Evaluation verifies that returns of leadership happen on the right terrain, not that humans were eliminated.

Trajectory and promotion evidence

Evaluation attaches to trajectories in addition to isolated calls. A week that accepted after one critique differs from a week that required three patches. Trajectory graders ask whether the loop behaved acceptably for the case class. Promotion decisions use trajectory evidence plus per-edge figures.

For JudgeLocalRepairQuality, store cases where patches failed despite fluent wording. Those cases become regression anchors during library promotion. Without them, teams promote on average scores that hide brittle repairs. Exact checks remain in the deterministic shell and still require tests. Evaluation architecture spans graded and exact layers. Skipping either layer produces systems that sound measured while remaining blind on the dimension that failed households.

Common failures

QA phase thinking. Evaluation appears as a milestone after feature complete. Hooks are missing from contracts. Launch proceeds with demos.

One aggregate score. A single number hides class regressions. Busy evenings improve while allergy edges fail.

Unowned graders. Each team writes slightly different rubrics for the same library entry. Figures are incomparable.

Evaluating the demo path only. Suites cover happy weeks while production traffic lives in edge combinations.

Minimum viable evaluation

Minimum viable evaluation for a new cognitive edge includes at least ten labeled cases per critical class, a published rubric, and a baseline run before any library promotion. Ten is a practical floor that forces stratification instead of one happy path.

Below that floor, mark the capability experimental in architecture records. Experimental edges may ship behind feature flags with explicit risk acceptance. They do not ship as silent production dependencies without suite pointers in contracts.

Connect evaluation design to graders not truth. Graders score declared standards. They do not replace domain ownership or household consent. Architecture uses graders to observe behaviour, not to settle values. Promotion meetings read figures aloud per case class. Silence usually means figures were not stratified.

Principle 6 turns judgment from a demo trick into material you can ship with promotion rules.

What to do next

Add a suite pointer when naming a cognitive unit, in the same pull request as the contract. Stratify suites by case class drawn from terrain, including edge combinations demos skip. Block library promotion without published figures for the identity version. Read the Evaluation section when designing judges and trajectories.

When someone says "we trust the model," translate the claim into suites and figures or mark the capability experimental. Architecture speaks in promotion criteria, not in confidence adjectives.

Read next: Architecture outlives implementation vocabulary.