AI agents are here
6 days ago
6 min read
Updated: 5 days ago
but have we learned how to work with them?
AI agents are entering into procurement processes. From supplier onboarding and risk assessment to sourcing and contract management, more activities can be supported by agents. Adoption is accelerating, and so is the correction that follows it: over 40% of agentic AI projects will be canceled by the end of 2027, mainly because of unclear business value and organizations overestimating what the technology can do on its own.[1]
But there is one part of the implementation that gets surprisingly little attention: what happens when people actually have to work with them?
We often see the same pattern. An agent is introduced, users are trained, and then people start finding ways around it: a manual check here, an Excel file there, an internet search to validate a recommendation, or an email sent outside the process because it feels safer.
This pattern shows up across the industry, not just at individual clients. A recent study found that only about 5% of custom enterprise AI pilots reach production with measurable financial impact, while general-purpose tools succeed far more often. The same study found that employees at more than 90% of the surveyed companies use personal AI tools for work on a regular basis, even though only 40% of those companies had an official subscription in place.[2]
This doesn't necessarily mean the agent is bad, or that users are resistant to change. It can simply mean that people haven't yet learned how to work with it.
Why is an AI agent different from traditional software?
With traditional software, users learn the workflow: what to enter, where to click and what happens next. An agent is different: it interprets information, brings together different sources, and produces results or recommendations that may not always be what the user expected, which changes what users actually need to learn.
It is no longer enough to understand the functionality. They also need to develop judgement around the output: What does the agent need from me? Which sources does it use? When can I rely on its recommendation? When should I question it? And what can I do if the result does not seem right?
Researchers call this skill appropriate reliance: trusting an AI system exactly as much as its actual reliability warrants. It's a documented, teachable capability, which is why it needs to be built into an implementation rather than left to chance.[3] This is especially important in procurement, where a recommendation can influence decisions about suppliers, risks and ultimately the business.
What happens when users don't understand the recommendation?
We recently saw this during an AI agent implementation at a Healthcare software provider. The agent was introduced to support risk assessments during supplier onboarding, analyzing available risk information, identifying potential supplier risks, and providing recommendations for Procurement and Risk Management. The user group consisted of approximately 15 buyers and Risk Management professionals.
The technology worked, but an interesting challenge emerged: users had to learn how to evaluate the recommendations themselves. They wanted to understand why the agent had identified a given risk, which information and sources contributed to the assessment, and whether they should accept the recommendation or investigate further.
And when users were uncertain, they did what experienced procurement professionals naturally do: they verified the information themselves. Some information was transferred into Excel for additional checks, external sources were searched manually, and additional internet research was performed to validate what the agent had identified.
From the user's perspective, this was reasonable. From a process perspective, however, something important was happening: automation was being introduced, but parts of the work were still being performed manually in parallel. The agent hadn't failed technically, but the new way of working hadn't yet fully replaced the old one.
This gap between a technically working system and an actually trusted one shows up well beyond procurement, too. Tracking employee trust between May and July 2025 alone, one workforce index found that trust in company-provided generative AI fell 35%, while frontline workers' trust in agentic AI systems specifically fell 89%.[4] Trust in agentic systems is fragile, and it is built or lost in exactly the moments described above.
The workaround is information
This is why workarounds shouldn't simply be treated as non-compliance: they are valuable feedback. If buyers repeatedly conduct an additional internet search after receiving a risk recommendation, the first question should not be "Why aren't they following the new process?" A much more useful question is "What is missing from the agent's output that makes them feel they still need this check?"
Maybe the source isn't sufficiently transparent, maybe users don't understand how the recommendation was created, maybe they need the option to provide additional information, or maybe the agent needs to consider another source that experienced buyers know is relevant. Every workaround can therefore tell us something about the gap between the technical process and the actual way people work.
What helped in practice?
In our example, the answer was not simply another round of standard system training. The focus shifted toward helping users understand how to work with the agent and its underlying LLM: how recommendations were generated, how to evaluate the information and sources behind them, and how they could further improve a result when necessary. This included understanding which sources should be considered, providing additional context, and learning how to interact with the agent when the initial result wasn't sufficient.
In practice, this meant helping users build judgement: knowing when to trust the agent, when to challenge it, and how to improve its output.
Over time, this changed the way the agent was used: manual activities could be reduced, more of the risk assessment process could be automated, and process lead times were shortened. What started with a limited group of Procurement and Risk Management users developed into a company-wide standard for the process.
The payoff of getting this right is measurable at the portfolio level, too: procurement organizations investing deliberately in both technology and people (what one industry survey calls "digital leaders") hit their cost savings targets far more often than the rest of the field (96% vs. 80%), and were more than twice as likely to say their function enables innovation (56% vs. 24%).[5]
How do we build this into an implementation?
At MULTIPLAI, we focus on four areas:
Process Design → Testing → Communication & Enablement → Hypercare & Change
Process Design: Define what the agent does, what the buyer does and where human judgement is required. This includes defining the handover between human and agent, not only the technical workflow.
Testing: Test real procurement scenarios and exceptions with future users, not only the standard process. Users need to see what happens when information is incomplete, a source creates an unexpected result, or the recommendation does not match their initial expectation.
Communication & Enablement: Go beyond explaining buttons and functionality. Help users understand how recommendations are created, which sources are considered, where the limitations are and what they can do to improve a result.
Hypercare & Change: Look for manual checks, Excel files, additional internet searches and process deviations. Instead of treating them only as adoption issues, use them as input to understand where the agent, process or user guidance needs to improve.
So, when is an AI agent really implemented?
Going live is one milestone, but the more important question comes a few weeks later: are people working with the agent, or have they learned how to work around it?
A technically functioning agent is not enough if users constantly double-check its recommendations, reproduce their work manually or bypass it when they are uncertain. More than 4 in 10 agentic AI projects are on track to be scrapped for exactly these reasons, and that is the difference between a pilot and a company-wide standard.
Successful implementation requires users to understand the agent's recommendations, challenge them when necessary, and learn how to make it better through their interaction with it. Implementing the technology is only part of the work; the other part is helping people learn how to work with it.
-----------------------------------------------------------------------------------------------------------------------------
Sources Cited
[1]Gartner. (2025, June 25). Gartner predicts over 40% of agentic AI projects will be canceled by end of 2027 [Press release]. https://www.gartner.com/en/newsroom/press-releases/2025-06-25-gartner-predicts-over-40-percent-of-agentic-ai-projects-will-be-canceled-by-end-of-2027
[2]MIT NANDA. (2025). The GenAI divide: State of AI in business 2025. MIT Media Lab. https://nanda.media.mit.edu/ai_report_2025.pdf
[3]Mehrotra, S., Degachi, C., Vereschak, O., Jonker, C. M., & Tielman, M. L. (2024). A systematic review on fostering appropriate trust in human-AI interaction: Trends, opportunities and challenges. ACM Journal on Responsible Computing. https://doi.org/10.1145/3696449
[4]Deloitte. (2025). Workforce trust with AI: Q3 2025 Deloitte TrustID workforce index. https://d1lzrgdbvkolkd.cloudfront.net/4749_Deloitte_Trust_ID_Workforce_AI_Report_Q3_2025_3aa42f916c.pdf
[5]Deloitte. (2025). 2025 global chief procurement officer survey.
Comments