Human Factors: Over-Trust And Delusion Risk From Conversational Software

Issue 67 Edition 2026-03-08 4 min read

Not accepted General

Sources: 1 • Confidence: Medium • Updated: 2026-04-12 10:23

Key takeaways

Very short exposure to a relatively simple computer program can induce powerful delusional thinking in otherwise normal people.
The observation that brief exposure to a simple computer program can induce delusional thinking is attributed to Joseph Weizenbaum and dated to 1976.

Very short exposure to a relatively simple computer program can induce powerful delusional thinking in otherwise normal people.
The observation that brief exposure to a simple computer program can induce delusional thinking is attributed to Joseph Weizenbaum and dated to 1976.

Under what specific conditions (instructions, disclosure, interface design, user population) does brief exposure to a simple program induce delusional thinking, and how is "delusional thinking" operationalized?
How prevalent and severe is the effect described (e.g., frequency, duration, downstream harms) in controlled studies or replicated settings?
What is the primary source/context for the 1976 attribution (exact quote location, surrounding argument), and does it describe a specific observed incident or a general warning?
What mitigations, if any, are implied or supported within the source material (e.g., disclosures, interaction limits, auditing), and do they reduce the effect?
Is there any direct decision-readthrough (operator, product, investor) supported by this corpus beyond general awareness of human-factors risk?

Conversational software deployments may face human factors risk where users over trust outputs, increasing reputational, legal, and support costs for providers and integrators.
Demand may rise for mitigation features in conversational interfaces such as disclosures, usage limits, and audit tooling, creating product differentiation based on safety and user outcomes.
Regulators and enterprise buyers may treat conversational software as needing stronger risk controls because susceptibility is framed as long known, not solely driven by modern model capability.

Replicated controlled studies showing brief exposure to conversational programs can induce measurable delusional thinking, with defined operational metrics and observed frequency and duration.
Documented incidents where users develop persistent false beliefs from conversational interactions, leading to downstream harms and driving customer complaints, policy responses, or litigation.
Evidence that specific interface disclosures or interaction constraints reduce over trust or delusion like outcomes, and that buyers require these mitigations in procurement.

High quality studies fail to replicate the effect or find it limited to narrow settings, with delusional thinking measures not exceeding baseline for most users.
Clear boundary conditions show the phenomenon requires atypical instructions, deceptive framing, or unusual user populations, reducing relevance to mainstream conversational products.
Source review shows the 1976 attribution is rhetorical or anecdotal without empirical basis, weakening its value as an operational risk signal.