False familiarity

The week looks like every other busy week. The interface shows the familiar fast plan badge. The household skims and approves. Only at dinner does someone notice: the plan ignored a new dietary constraint mentioned once, buried in a long message. The system matched surface similarity. Busy evenings, spinach, leftovers. The hidden axis, an unfamiliar medical restriction, made this week frontier terrain, not established pattern.

That misfire has a name: false familiarity. It is among the most trust-damaging failures because it feels like competence right up until consequence arrives.

Helpful context: Applicability conditions are the primary defense. Deliberation and automaticity explains why compression exists. Least trustworthy input wins helps diagnose contaminated context.

Why it happens

Compression exists because full deliberation is expensive and slow. Guards make compression safe when guards are complete. False familiarity happens when guards check visible features like busy flags but miss latent ones like medical mentions, when retrieval mixes endorsed knowledge with stale chat, when temporary context resembles durable pattern, or when evaluation suites cover happy paths but not near-miss weeks.

The system did not get lazy in a human sense. It executed a strategy licensed by incomplete conditions. Diagnosis targets guards and capture, not moral language about the model. The fix is always architectural: tighten conditions, add regression, make fallback visible.

Surface similarity is not applicability

Human cognition pattern-matches constantly. Software expertise amplifies the habit at machine speed. A week that looks busy may differ on medical, guest, equipment, or budget axes. A request that repeats phrasing from last month may change one material word.

Applicability conditions exist to slow that match until inspectable state confirms fit. When they are incomplete, false familiarity is the predictable result. The gap between "looks similar" and "meets conditions" is where most compression failures originate.

Surface similarity is not applicability. Familiar shape is not familiar terrain.

Introduction to Thoughtware · Ch. 26

A vivid misfire, then the fix

The misfire: Busy Week Pattern v0.2 runs on a week that mentions low-FODMAP for the first time. Pattern anchors busy evenings and spinach correctly. Meal candidates violate an unexplored medical constraint. Fluency and speed mask frontier terrain.

The fix: Guard adds condition three, "no unfamiliar medical restriction present." Failure routes to deferral, not guess. Regression adds low-FODMAP mention cases. Experience logs guard failure rate for monitoring.

One story beats ten warnings. The design response is always tighten guards, add regression, make fallback visible. That response works because the failure is structural, not accidental.

Relationship to abstention and trust

False familiarity pushes the system toward false completion. A plan appears when abstention or deferral was correct. Users lose trust faster from confident wrong compression than from honest slow deliberation.

Posture matters here. A system that says "this week matches your usual busy pattern" without showing guard results performs confidence theatre. The honest version says "busy week pattern applies, medical restrictions unchanged from endorsed knowledge." The alternative says "pattern withheld, unfamiliar dietary mention detected."

Trust repair after false familiarity requires visible guard changes alongside apology copy. Users need evidence the fast path will refuse next time. Words alone are not that evidence. A version bump with a named new condition is.

Monitoring guard failure rates

Production tracks guard failure rate per expertise artifact. Rising failures signal environment shift or stale conditions, not model degradation. v0.3 may need tighter checks even when accuracy on old suites stays flat. The combination of stable old-suite accuracy and rising guard failures is a clear signal that the world changed, not that the model forgot.

Pairing telemetry with qualitative review strengthens the signal. Support tickets that say "it usually works" are early indicators of guard rot. Near-miss logs are better indicators still.

Three surface signals and hidden axes

A busy week layout may hide a new medical term in a long message. Correct behaviour is guard fail leading to defer. A repeat ingredient list may hide an expired allergy endorsement. Correct behaviour is knowledge check fail leading to ask. A "same as last time" user phrase may hide changed context like guests or a broken oven. Correct behaviour is context refresh leading to slow path.

Training teams to ask "what could look the same while materially differing?" generates guard conditions more reliably than brainstorming model improvements. That question is the single best tool for guard authoring because it forces attention onto the axes that surface similarity obscures.

Diagnosis after an incident

When reviewing an incident, five questions clarify whether false familiarity occurred. Did an expertise artifact or shortcut run? Which guard conditions were checked? Which material axis differed undetected? Would a near-miss regression case catch this next week? Did the interface show compression honestly?

Affirmative answers to the first two questions with failure on the third indicate false familiarity. The response is fixing guards before tuning prompts. Prompt tuning treats the symptom. Guard revision treats the cause.

Context contamination

Least trustworthy input wins applies when stale chat, ambiguous retrieval, or mixed memory forms contaminate working state. False familiarity often combines weak guards with contaminated context. Fixing guards alone may not suffice if selection policy treats every mention as equally endorsed.

Auditing what entered working memory for the run reveals whether false familiarity began before compression, at selection time. The failure may originate upstream of the guard, in which case the guard passed honestly but the inputs lied. Both layers need inspection.

Prevention culture

Teams rewarded only for speed build fragile compression. Teams rewarded for guarded speed build libraries. Promotion meetings ask "what near-miss cases exist?" alongside average latency. Incident reviews version guards alongside model retraining.

False familiarity is a design failure with a design response. Treating it as random model error guarantees recurrence. Several organizational habits raise risk: shipping compression to meet a latency OKR before guards exist, letting each squad maintain private prompt shortcuts without library placement, skipping near-miss regression because happy path accuracy looks fine, and treating support tickets as one-offs instead of guard version inputs. Each habit is fixable without new models. Governance and library discipline address most recurrence.

Communicating misfires without blame

Postmortems name the artifact version, the guard condition missing or untested, and the regression case added. Blaming the model teaches the team to tune prompts instead of tightening guards. Blaming the user teaches the team to hide deferral. Blaming luck wastes the incident.

Households deserve plain language after a misfire. "We applied a fast path that did not fit this week because a new dietary mention needed full review." That sentence rebuilds trust faster than "we updated our model." The first is specific, testable, and implies a structural fix. The second is vague and offers no evidence of prevention.

Building a near-miss library

Maintaining a near-miss library alongside happy path suites turns incidents into assets. Each incident becomes a case: surface description, hidden axis, expected guard outcome, link to artifact version. Over time the library becomes the best training material for new architects. It also speeds regression authoring when similar incidents recur.

Near-miss libraries work best when linked to experience logs. Replaying the week, showing the guard gap, and showing the fix in v0.3 combines narrative with evidence in a way policy slides cannot match. Inviting support to contribute near-miss cases catches surface similarity failures before engineering sees them.

Running a quarterly false familiarity review even when incidents are zero catches drift before users do. Reviewing near-miss logs is guard maintenance, not model tuning. Treating those reviews as a reliability ritual gives them the same standing as other periodic infrastructure checks.

The Mirror section ends here. Trust begins with how the system shows up when guards fail or defer. Cognitive posture is the next concept in the curriculum.

What to do next

Reviewing the last three production incidents where the system "should have known better" reveals how many were false familiarity: compression or retrieval on surface match when a hidden axis differed. Adding one guard condition and one regression case per incident class converts each incident from anecdote into structural improvement.

Applicability conditions covers guard design. Abstention as a result covers when the correct output is no plan at all. Least trustworthy input wins addresses the upstream contamination that often accompanies weak guards.

Read next: Cognitive posture.