Should We Trust AI? Three Approaches to AI Fallibility

By Kevin Townsend for SecurityWeek
Wednesday, July 23, 2025

• 559
• More

The problem with LLMs is that they mostly work but sometimes don’t – and we cannot easily tell which it is doing. We don’t know when or why it is right or wrong. The danger in agentic AI is that a wrong response can become an autonomous, unsupervised and potentially damaging action. Yet agentic AI is blossoming everywhere because we assume it works correctly and, anyway, it is saving us so much money.

The result, according to Ilia Kolochenko, CEO at ImmuniWeb and adjunct professor of cyber law & cybersecurity, is an over-heating market with vast amounts being spent on faith rather than sound logic – and he sees this as an AI Bubble mirroring and likely to follow the same path as the Great Dot-Com Bubble that burst in March 2000.

Unaware of the longer term danger, and intent on maximizing the short term benefit, our efforts are focused on remediating the symptoms of the weakness rather than abandoning the technology. For agentic AI, this largely revolves around applying human oversight and intervention to a system designed to be automatic – which is a contradiction in terms. It is almost certainly doomed to fail. We struggle to ensure security by design in software development, and we cannot prevent logic flaws in code. A primary cause is pressure from business leadership to complete tasks as fast and as cheaply as possible – that pressure will be repeated in human oversight of, and intervention in, agentic AI implementations; we’ll take shortcuts.

Kolochenko accepts that RAG provides a slight improvement, but says, “I don’t think it will be the ultimate solution. When you do augmentation, you still need data; and you will never have perfect data. So, it may bring improvement in terms of quality, and it may reduce some problems – but I don’t think it will prevent hallucinations, discrimination, bias, and whatever else we already have in AI.”

Kolochenko places his faith in the future: the bubble bursting and the passage of time will provide the solution. It will not make AI more trustworthy, but it will teach us how, where and when we can use it safely and securely. AI will be designed to help real users rather than chase elusive, and expensive dreams.

“I think we are observing the second episode of the dot-com bubble. People believe in miracles. They need magic in their lives because otherwise life is boring. When they believe they’ve found this magic, they think life is great and everyone can be a billionaire. They’ll blindly follow the arrows that are laid down for them, instinctively, because this is how our brains work. So now we have everybody, including C-level executives of the largest companies, over excited and thinking, ‘Goodness, with AI, we’ll make huge profits; we’ll do this and make that.’ But very few of them understand how AI works.”

The dot-com bubble bursting did not stop the internet, it refocused it more sustainably. We’ve had huge and beneficial developments through responsible investment, including search engines, e-commerce, cloud computing, social media, mobile computing, web2 and web3 coming after the dot-com bubble. There’s still much wrong with the internet, but society is better off with it than without it.

Kolochenko believes that AI will follow the same pattern. “I believe that once this hype around AI has disappeared, and I think it will probably happen soon, we will again have some interesting tools. For example, journalists will be able to use faster spell checkers. Don’t dismiss it. Current, or should I say native, spell checkers are somewhat simplistic or primitive. An AI spell checker will likely detect the wrong word even if correctly spelled and subtle semantic errors. That will save time and improve the output of authors who don’t trust current gen-AI to create their output.”

If you look at optimistic elements of trust in AI from both Polyakov and Kolochenko, there is one major common factor: the trusted AI apps are all self-contained, have a single purpose, and work with the user (and therefore have human oversight) rather than working instead of the user.

This is vastly different from the currently emerging crop of agentic AI apps, which are expected to autonomously complete complex rather than singular tasks with complicated and diverse actions and reactions, without human intervention. It is here that Kolochenko completely loses trust.

“To successfully manage something, you need to be at least as smart as what you are managing,” he comments. “You can give a chimpanzee a transmission electron microscope designed for scientific research but that doesn’t mean the chimpanzee will be able to do scientific research. A microscope is an advanced tool, but if you don’t know how to use it, it is worthless.”

Kolochenko isn’t comparing human users to chimpanzees but pointing out the mismatch between the complexity of AI tools being offered, and the relatively simple requirements of most users. He believes the current AI bubble will burst, and many companies will suffer – but it will teach and force us to realign AI with users’ needs rather than some complex, flashy, cool but unmanageable operation. Read Full Article