Written by: Nolen Akerman & Fraser Lockhart
Earlier this year, friends were conducting research on weather stations for their backyard garden. Evaluating the set of features for all the models was a tiresome process, so they sought assistance from a friendly online AI chatbot. The response was impressively formatted with tables and links, highlighting the best features of each model. The only problem: several of the models were completely made up, and there was no record of them existing from any manufacturer.
You have most likely run into this same situation in either personal or business use of AI tools. Scenarios like this may leave you wary of the use of AI assistants or perhaps even turn you off to the use of AI at all. If you detect a blatant factual issue, it shakes your confidence in all the other information the tool has provided.
Reconsidering the Level of Expertise of AI Systems
Many early career experiences start with being an intern, especially in Information Services. As an intern, you will undoubtedly make many mistakes and become very frustrated. Effective managers understand this learning curve and provide additional training resources, documentation, and points of contact for questions.
Comparing a new AI-enabled solution to an intern is an often-used analogy, but for good reason. These AI agents are touted as smart and capable with undergraduate equivalent reasoning skills, but they need domain-specific training documents and feedback to grow into their potential. Users assume the AI system is the expert, but even AI vendors market their solutions as ‘assistants’, ‘co-pilots’, or ‘helpers.’
Why Implementing AI-Enabled Software Requires a Paradigm Shift
Most deterministic computer programs are constrained to logic paths and strict input and output rules. IT departments and business users have many decades of experience conducting testing processes on these systems and expect that, without coding changes, the output of the system will remain unchanged over time. With AI solutions, the benefits of expanded logic paths and more flexible input options come with a cost of testing repeatability. Random hallucinations confound UAT testing scripts, and updated LLMs or changes to underlying training documents make regression testing a moving target.
To address the complexities of this new paradigm of software testing, start with these questions for your AI-enabled solution:
- What level of accuracy and repeatability is required in this solution to provide value to the company?
- What is the accuracy of the current human-enabled or traditional software-enabled solution we are looking to replace?
- What are the consequences of a ‘wrong’ answer by the solution? How much is at risk?
- What level of control will human oversight have in this process? Will all answers be reviewed by a human for accuracy?
- What is the roadmap for improvement? What new documents or instructions can we create for the agent?
- Can we combine ‘legacy AI’ methods (expert systems) with an LLM front-end to balance usability and repeatability?
The Opportunity: Turning Discomfort into Advantage
Once an AI application is deemed fit for the intended use, there will be an organizational learning curve and the need for managed organizational change. This may cause discomfort in the organization, not only from the change in ways of working, but also in the output of the tool.
Answers that may seem counterintuitive or simply wrong may actually be picking up on unexpected or non-obvious patterns within the data. These results may challenge “how things have always been done” and some long-held, but unfounded assumptions. AI is able to parse large datasets and find hidden patterns and trends; just because an answer is unexpected does not mean it is incorrect.
Being able to detect hallucinations and blatantly wrong answers from valuable insights is a skill developed through extended use of AI tools, a rigor for fact-checking, and a balanced skepticism of the responses. The good news, however, is that most people already have some experience with this type of rigor from their use of the Internet. The same judicious review of online content applies to the response from an AI tool.
Revisiting the Intern Analogy: Building Trust Over Time
When you recast the image of AI as a human assistant working with less than perfect data instead of an infallible expert, you keep your diligence in review of their work. Just like with an intern, you may give AI a task that it performs flawlessly, and that performance builds trust for assigning that same task in the future. That trust is earned through results, not blind acceptance. These tasks can start small with pilot programs or limited tests and grow with positive results.
The insights and efficiencies from these pilots can be tracked, and those outcomes, once tied to business value, provide the credibility for future expansion. These results should also be shared internally as success stories to build excitement and confidence in other areas of the business. This virtuous cycle is enabled when the balance between progress and perfection is properly aligned to the business task.
III. The Human Reaction to Change (Behavioral Perspective – Fraser)
When an AI recommendation challenges what someone believes to be true, the reaction is rarely neutral.
It’s emotional, first, logical second.
That doesn’t make people resistant; it makes them human.
In high-stakes environments, where decisions impact revenue, performance, or reputation, people are wired to protect what’s working (or what they think is working). So, when AI suggests something different, it can feel less like insight and more like risk.
A few natural tendencies are at play:
- Confirmation Bias
We look for information that supports what we already believe. If the AI says something different, it is easy to dismiss it instead of examining it.
- Loss Aversion
The fear of making the wrong move is stronger than the excitement of making a better one. “What if this doesn’t work?” often outweighs “What if this improves results?”
- The Trust Gap
It’s difficult to trust a recommendation when you don’t fully understand how it was generated. Without that clarity, skepticism fills the gap.
- Experience Bias
“This has always worked for us” becomes the benchmark even when conditions have changed.
Here’s the key shift:
Skepticism is not a problem. Immediate dismissal is.
The most effective leaders don’t blindly accept AI recommendations, but they also don’t reject their instincts. They pause long enough to explore what might be true before deciding what to do.
Because in many cases, discomfort isn’t a sign that something is wrong.
It’s a signal that something different is being revealed.
And that’s often where the real opportunity starts.
Ready to Turn AI Potential Into Real Business Impact?
At Demand Chain AI, we help organizations move beyond AI hype and build practical, trusted solutions that drive smarter decisions, operational efficiency, and measurable growth.
If your team is ready to balance innovation with confidence and turn AI insights into action, let’s connect.
