The Confidence trap
AI is useful, but by default it is not a source of truth. The danger is not only hallucination - it is confidence that should have been challenged.
At Sugra Systems, Inc., we started working with AI long before the release and mass excitement around ChatGPT. Back then there were closed models, available only via API calls, for testing and integrations, and we had one constant question: can this be trusted in a real business process?
That is why the first “wow effect” passed for me a long time ago. When you use AI not as a toy, but as part of a product or business process, you quickly develop resistance to the hype. Practice shows that AI is useful, but by default it is not a source of truth.
Today we often hear: “AI told me”, “I checked it with ChatGPT”, “AI confirmed my idea”. An AI answer is not verification. It is generated text that itself needs to be verified. The model providers say this directly: ChatGPT and Claude, and others, can produce incorrect or convincing-sounding answers, fabricate quotes and sources, and in complex matters they should not be used as the only source of truth.
There is another risk. AI often does not challenge. It agrees, strengthens a weak idea, and removes the feeling of risk. OpenAI even rolled back a GPT-4o update because the model had become too flattering and too agreeable.
In legal and legislative matters, this has already been measured: general-purpose LLMs hallucinated in 58-88% of cases on verifiable questions about court cases. This is a systemic risk, not random errors.
In real life, this has already led to consequences. Air Canada was found responsible for incorrect information provided by its chatbot. Lawyers in Mata v. Avianca were sanctioned for fake cases generated by ChatGPT. NEDA suspended its Tessa chatbot after harmful recommendations. And in the Character.AI case, the court did not dismiss the lawsuit at an early stage after allegations linking the chatbot to a teenager’s emotional dependency.
AI is one of the strongest tools introduced in recent years. But exactly because of that, we cannot work with it at the level of admiration. AI accelerates both thinking and error. It helps test an idea, but it can also beautifully confirm something that should have been challenged, and give confidence that turns out to be only an imitation.
For brainstorming, this risk may be acceptable. For law, medicine, finance, cybersecurity, infrastructure, and serious decisions, it is not. In these areas, AI should not be a pleasant companion, but a disciplined critic that checks, doubts, and shows risks.
That is why we should approach AI not as an assistant that confirms everything, but as a tool that must be deliberately configured: give it the role of a critic, add fact-checking, and leave the final decision to a human. The main question is no longer whether we use AI. It is whether we have configured it to protect the quality of decisions, instead of simply confirming our assumptions.
Sources
-
OpenAI: https://help.openai.com/en/articles/8313428-does-chatgpt-tell-the-truth
-
OpenAI GPT-4o rollback: https://openai.com/index/sycophancy-in-gpt-4o/
-
Large Legal Fictions: https://academic.oup.com/jla/article/16/1/64/7699227
-
Air Canada: https://www.canlii.org/en/bc/bccrt/doc/2024/2024bccrt149/2024bccrt149.html
-
Mata v. Avianca: https://law.justia.com/cases/federal/district-courts/new-york/nysdce/1:2022cv01461/575368/54/
-
Character.AI