Artificial Intelligence can sometimes be wrong in a very convincing way: clean code, coherent explanations, answers delivered without apparent hesitation. The real problem, however, lies in the trust we often place in it before we even check.
We discussed this with Michele Ridi, Chief Presale Officer at Bitrock and Fortitude Group, in a recent episode of our podcast Bitrock Tech Radio. The reflections that follow stem from that conversation: which pitfalls are triggered precisely when we are convinced we are working better, and which methods, habits and tools can help companies reduce them.
Why we trust AI even when it’s wrong
We often trust AI because a well-structured explanation, with the right terminology and a confident tone, conveys expertise. With modern AI tools, that impression frequently arrives before any verification does.
The same happens with software code: consistent variable names, clear comments and an elegant structure make it look like a solid starting point. Yet a vulnerability, or a function that doesn’t actually exist, can easily be hidden inside.
This pattern recurs across business settings, well beyond software development. An AI-generated meeting summary almost always looks excellent at first glance, yet a proposal can quietly turn into a decision (for example, “we could deliver in November” becomes “we will deliver in November”).
In 2023, a group of researchers at Stanford observed exactly this: in the security tasks examined, participants using an AI assistant wrote less secure code and were more inclined to consider it secure. The case is specific and tied to the tools available at the time, but the mechanism still holds: error comes with a false sense of security.
The first line of defense therefore starts with a simple question: if this answer were wrong, would we notice? And with what consequences? This is what trust should be calibrated on: the impact of a potential error – not the elegance of the text.
The perceived productivity trap
Productivity becomes a trap when it is measured by how it feels. A first draft that appears in a matter of seconds is impressive and can give the impression that half the work is done: but then the data needs to be checked, inaccuracies corrected and the output adapted to its context. These tasks are part of the work and must be accounted for.
In 2025, METR studied 16 experienced developers working on projects they knew well. Using the AI tools available at the time, they took 19% longer, yet continued to believe they had saved time. The result naturally needs context: it applies to those tasks, those people and those tools. METR later reported that, with newer tools, the picture was changing, although it was harder to measure. The key point remains, however, the gap between feeling more productive and actually being more productive.
For a company, this means looking at the end-to-end process: how long does it take to reach an approved document, a working change or a resolved issue? If one person generates ten documents in an hour and a colleague then spends two days reviewing them, individual work has certainly been accelerated, but the overall process may well have become heavier.
Then there is the question of practice: those who systematically delegate the initial analysis risk losing familiarity with the very steps needed to verify the result. To avoid this risk, it is better to use AI to compare alternatives and surface new hypotheses, while leaving room to understand why a solution works and under what conditions it stops working.
Practical habits and a culture of challeng
Protecting yourself doesn’t mean turning every answer into an endless review. A few habits, applied at the right points, are enough:
- Distinguish facts from inferences. If an answer contains an important piece of data, you need to be able to trace it back to its source and check that the source actually says so. If it contains a conclusion, you need to understand which assumptions it rests on. Asking the model to state them helps, but its explanation must be assessed too.
- Don’t let the first answer define the problem. Asking “how can we introduce a chatbot to reduce support requests?” means the solution has already been chosen. It is better to first ask which requests come in, why they come in, and which could be avoided by improving the service. AI is efficient at developing a direction: that is exactly why you need to check it is the right one. Asking it to argue an alternative position or to raise objections broadens the reasoning, but it does not replace independent verification.
- Decide up front how to recognize an improvement. When piloting an AI assistant, write down in advance what you expect: less time to complete a task, fewer errors, fewer manual steps. The corrections required must be measured as well. Otherwise, you risk judging the project by the two answers that worked best during the demo.
Last but not least, there is another habit to cultivate, perhaps the most important one, and it is cultural: being able to challenge. If questioning an answer comes at too high a cost, people stop doing it; this applies to AI just as it does to a decision made by the most senior colleague in the room.
For a company, all this means creating the conditions for anyone to say “I’m not convinced by this answer” without friction. Ultimately, real control remains in the hands of people: they are the ones who set the criteria, recognize when a result doesn’t add up, and decide how far to rely on the model.
AI governance in the enterprise
When AI becomes part of business applications, technology must make the controls described so far workable in practice. A person using an assistant can pause, read and verify. When multiple applications call different models, shared rules are needed: who can access what, which services can be used, which data can flow through, and how much is being spent.
This is where Radicalbit AI Gateway comes in. Part of the Fortitude Group Product Portfolio, it acts as a controlled checkpoint between applications and models, making it possible to:
- centralize access and enforce shared policies;
- collect information on requests, responses and usage;
- monitor costs and performance;
- protect data, for example by masking personal information.
Consider a company where multiple teams are building AI-powered features. Without shared management, each team tends to configure access, limits and monitoring according to its own criteria. Radicalbit AI Gateway makes it possible to bring these controls back to a shared, consistent level.
One clarification is needed, however: being able to trace an answer does not prove it is correct. A gateway alone cannot guarantee that a model won’t generate inaccurate information. Assessing quality also requires source checks, testing on the specific use case and verification proportionate to the consequences of a potential error. What the gateway contributes is giving the company visibility and governance tools: in the event of anomalies, it makes it possible to reconstruct the interaction and intervene, and to apply rules systematically rather than relying on any one individual’s memory.
These aspects become even more important with Agentic AI, i.e. systems capable of taking actions autonomously. In this scenario, it is essential to define clearly which operations AI can perform on its own, under what circumstances it must stop, and who is authorized to step in. This is where method and architecture meet: people define the criteria, and technology ensures they are applied continuously and consistently.
Conclusion: defining and preserving the criteria for judgment
The pitfalls of AI do not depend solely on the limitations of the models. They arise from the encounter between highly convincing answers and the way we evaluate them: we trust the form, we measure productivity by the feeling of speed, and we let the first answer steer the entire line of reasoning.
This is why control must operate on two levels. The first concerns the way people work: checking sources, distinguishing facts from inferences, defining in advance what “improving” means, and fostering a culture in which challenging is normal. The second concerns the systems companies build: where shared rules, traceability and governance tools are needed that do not depend on any one individual’s memory.
In both cases, control must be concrete, backed by useful information and real ability to intervene. Only then does the value of AI show up in results, and not just in demos. With agentic systems becoming increasingly autonomous, that margin for intervention will matter even more.
Defining these criteria and translating them into solid architectures is where method and technology must move forward together. It is on this balance that Bitrock supports companies, bringing consulting and technology expertise to guide them from defining governance rules to delivering secure, observable, production-ready AI solutions.
Contact us for a tailored consultation.
FAQ – Frequently Asked Questions
Overconfidence in answers that are well written but wrong, and perceived productivity, which does not always match the actual productivity measured across the entire process.
By distinguishing facts from inferences, not accepting the first framing of the problem, defining success metrics before experimenting, and fostering a culture in which challenging is possible.
By looking at the end-to-end process, not just the speed of the first draft: for example, how long it takes to reach an approved document, a working change or a resolved issue, including all the necessary corrections and revisions.
It is an environment in which questioning an AI answer comes at no real cost. If challenging costs too much, people stop doing it and errors go unnoticed.