Fluency is not evidence

“Wednesday: spinach and chickpea curry—a gentle midweek meal that uses the spinach before it turns.” The sentence sounds attentive. It can also be wrong: the household asked for spinach to be used before Wednesday.

Nothing in the sentence changes when the constraint fails. Fluency is produced independently of the grounds for a claim, so it cannot serve as evidence that the claim is correct. The habit of treating fluent language as proof makes sense in ordinary conversation, where coherence often signals care. Generated prose breaks that link.

Helpful context: Meaning, knowledge, and authority separates the different grounds a system may be missing. Evaluation as engineering develops the checks that must sit beside generated output, and abstention as a result explains why a careful system sometimes declines to answer.

The heuristic worked before this medium

In human communication, clarity often takes effort: someone who has understood a situation usually can explain it better than someone who has not. That correlation taught us to read well-formed language as a sign of thought.

Language models can produce the surface of that thought without the corresponding evidence. They can explain an incorrect plan as smoothly as a correct one, which means the reader's usual shortcut now accepts exactly the artifact that needs independent checking.

Meaning is not enough either

The system may understand what the household asked and still lack the facts needed to answer. It may have an endorsed allergy record and still lack permission to order groceries. Interpretation, knowledge, and authority are distinct conditions, yet fluent prose integrates them into one reassuring voice.

That voice must not become the place where the product decides how much to trust itself. The architecture must retain differences in evidence: confirmed allergy, observed preference, uncertain inference, and missing information should lead to different behavior—not merely different adjectives.

Build evidence beside the prose

Closed constraints need deterministic checks, so the spinach deadline should fail independently of the meal description. Open judgments, however, need named units and evaluation cases. The busy-Tuesday decision should be reviewed for the quality of its trade-off, not for whether the explanation feels reasonable.

Conduct needs its own checks as well: whether the system exposes assumptions, cites endorsed facts, asks a question when grounds are insufficient, and refuses when authority is missing. Asking the model to “be careful” verifies nothing because it uses prose to judge prose.

Fluent claim	Evidence that should exist outside the prose
This uses the spinach in time	A deterministic date and ingredient check
This dinner fits a busy evening	AssessMealPracticality cases for effort, timing, and household tolerance
This reflects your preference	Provenance showing whether the preference was endorsed, inferred, or temporary
I can place the order	A permission boundary and approval record
No question is needed	A conduct check that missing information was not material

The table shows why one evaluator cannot carry the whole burden. Some failures are exact and should be caught before prose is generated. Some are judgments and need cases. Some are authority failures and should stop the action no matter how strong the recommendation looks. Fluency smooths these differences away; architecture restores them.

Test the surface by breaking it

For any visible answer, list what would establish its correctness without reading the answer. Then deliberately violate one condition and verify that the product catches it. If the only test is whether a reviewer likes the response, fluency remains the system's real evidence.

Fluency also distorts the team that evaluates it

The problem does not end at the product surface. A reviewer reading a polished answer is subject to the same shortcut as a household, so offline evaluation can reward a response for sounding considered even when it missed a material constraint. Agreement between reviewers is not proof that the answer was grounded if each reviewer saw only the prose.

That is why evaluation must reach behind the surface. A plan review needs access to the interpreted request, endorsed knowledge used, closed checks that passed or failed, and the named judgment that made the trade-off. The final explanation can help a reviewer understand the result, but it cannot be the only evidence for the result.

Different errors require different instruments. An ingredient deadline needs a deterministic assertion. A recommendation needs cases in which reviewers assess whether its trade-off is defensible. A claim about household preference needs provenance. An action needs an authority gate. A trajectory that skips a necessary question needs behavioral review. These instruments complement one another because they correspond to different reasons an answer can look good while being wrong.

Make evidence visible in the interaction

The strongest products do not merely keep these checks internally. They expose the grounds at the point where a user needs to rely on them. The Companion can distinguish a confirmed allergy from a guessed preference, show which constraint shaped a recommendation, and ask for confirmation when a conclusion depends on an uncertain interpretation.

This is not an argument for exposing chain-of-thought. The product need only reveal the evidence category, the relevant source, and the boundary of its confidence. “This uses the allergy record you confirmed” is useful. “I inferred that this is a standing preference from one sentence last month” invites correction. “I can prepare the cart but need approval to submit it” preserves authority.

Those distinctions restore the information that fluent prose normally smooths away. They let a household challenge the right thing before an incorrect plan becomes a consequential action.

Read next: Why the words we have fail examines the parallel mistake in the vocabulary teams use to describe cognitive systems.