The one-size-fits-all formula for achieving a high return on AI investment doesn’t exist. What impressed us the most when analyzing different surveys is the staggering difference in the number of companies that achieve the expected ROI with AI and those that don’t.
Menlo Ventures’ survey found that 30% of enterprises consider easily quantifiable ROI as the primary criterion for selecting generative AI tools. But then 46% cite disappointment in their AI ROI.
IBM surveyed CEOs to discover that only 25% of their AI initiatives delivered ROI, and 16% of them scaled enterprise-wide. And if we consider the famous MIT study, which found that 95% of companies investing in AI fail to achieve ROI, the pattern gets even clearer.
Ensuring predictable and stable AI ROI is challenging, and enterprises often feel frustrated when trying to determine whether their AI initiatives prove worthy of the time, money, and effort invested.
This article will explain:
- Specifics of AI ROI compared to other digital solutions
- Six reasons why enterprise AI projects fail to deliver ROI and how to avoid them
- Lessons from AI leaders on maximizing ROI
We backed up our research with hands-on experience, as Xenoss consultants help companies organize their AI budgets and build a customized roadmap to measure the ROI of each specific AI project.
How measuring ROI on AI investments differs from other software solutions
Businesses often apply identical ROI formulas and expectations to AI as they do for traditional software. But with the latter, companies primarily pursue clear functional goals, creating unrealistic measurement frameworks. Traditional software investments pursue clear functional goals through a linear process: problem – digital solution – implementation – result.
For instance, you implement a SaaS HR system for efficient people management. Monthly costs are transparent, usage metrics are trackable, and you get a clear ROI of increased efficiency of the HR department. A classic ROI formula looks like this:
To give a clear financial example, implementing an ERP system with a TCO of $200,000, which allows the company to earn $100,000 in net profit, would mean a positive ROI of 50%.
By contrast, AI investments follow a fundamentally different decision-making process: hypothesis – experimentation – adoption – evolving outcomes. It’s more complex and requires patience.
For instance, you adopt an AI sales assistant to help your sales team quickly close deals. With the help of the assistant, sellers close some deals faster, others not at all, and in some instances, they may even need to double-check or override the AI’s suggestions.
The ROI is no longer a simple equation of hours saved. It depends on the accuracy of the recommendations and adoption rates across the team.
In other words, traditional ROI is deterministic. The savings and efficiencies map neatly to business outcomes. AI ROI is probabilistic. It emerges only when models perform reliably, employees trust and adopt them, and the organization adapts processes to capture the new value.
Given this complexity, companies need to set the right lens for measuring the value of AI. Instead of relying on a single ROI formula, they should frame AI outcomes across multiple goal-oriented dimensions.
Approach AI projects with a goal-driven mindset
According to Gartner, depending on the goal you want to achieve with AI, the focus may be on different business outcomes, such as classic ROI, return on employee (ROE), or return on the future (ROF).
If the goal is increased employee productivity, then your go-to business outcome is ROE, which shows the impact of AI on employee productivity. This business outcome is measured by employee engagement and well-being, as well as the time saved and increased task velocity.
Workflow efficiency projects utilizing LLMs, agents, assistants, and copilots warrant traditional ROI evaluation. These initiatives focus on quantifiable financial gains through cost reduction and revenue generation.
For ambitious AI projects that pursue competitiveness at the core, the ROF would be a suitable measure of success. It means you invest in a few experimental AI projects (e.g., five different R&D initiatives) at scale, presuming that if at least one project is successful, it’ll pay for the previous failures.
Gartner suggests balancing all three business outcomes to get the most comprehensive assessment of AI benefits for your business. Financial gains aren’t the only thing you can achieve with AI, and you shouldn’t limit yourself to it.
In theory, enterprises may understand that AI ROI isn’t a straightforward path. However, in practice, they often make hasty decisions that prevent them from realizing tangible AI ROI.
Six reasons enterprise AI projects miss ROI expectations
As an AI and data engineering company, we provide enterprises with AI investment consulting services. From this experience, we have identified six common reasons why AI ROI expectations and actual ROI often differ.
#1. Hype AI adoption with never-ending experiments
Big tech companies heavily invest in AI to win a fierce competition. The byproduct of these tech games is increased AI hype and the FOMO effect among smaller companies, which they attempt to counter by hastily investing in AI without a clear ROI strategy in place or by running many chaotic AI experiments.
A Harvard Business Review article warns companies against the “AI experimentation trap”, as never-ending AI experiments can burn resources, overwhelm teams, and never scale into production.
Instead of running several hyped AI experiments without a clear goal, SMBs and large enterprises alike should focus on solving pressing business and customer problems and defining use cases where AI could bring the most value.
#2. High expectations without measurable KPIs
Businesses set high hopes for AI, giving it almost magic wand powers. Gartner revealed that 74% of CEOs expect AI to be the most transformative technology of all for their businesses. But AI won’t work by itself. It needs solid infrastructure, active cross-company adoption, and clear ROI metrics by which you define its success.
And here’s the trick. A McKinsey study finds that only 18% of large organizations have well-defined KPIs to track the efficiency of gen AI solutions. If the goal and expected outcomes help you choose the right direction, KPIs serve as your map, helping you stay on track.
Gen AI KPIs can span different areas:
- Reliability and responsiveness metrics, including model latency, error rate, drift, and uptime, are used to evaluate the overall performance of gen AI.
- Model quality metrics, including coherence of the output, instruction following, text quality, and verbosity, help fine-tune the model’s accuracy to ensure it generates high-quality content.
- Business function metrics, such as customer churn, average handle time for customer service, click-through rate, time on site, or revenue per visit for product, marketing, and service use cases.
- Adoption metrics, including adoption rate, frequency of use, and session length, help evaluate the usability and accessibility of the AI solution.
- Business value metrics, including cost savings, revenue generated, and customer experience, are used to evaluate the outcome of the AI project.
Choose specific KPIs for each AI use case. For instance, if your sales or marketing team uses an AI chatbot for content generation, such as writing emails, sales decks, or marketing reports, then metrics that help evaluate content quality as well as model reliability would be necessary.
Brainstorm and identify the key metrics that are most important to your team. You can then redistribute the gen AI efficiency measurement among different team members.
For example, entrust the IT or R&D departments with tracking technical metrics using tools like Grafana and OpenTelemetry, while delegating the measurement of business metrics to internal or external business analysts via business intelligence (BI) tools like Tableau and Looker.
#3. Limited data infrastructure readiness
AI implementation requires preparation. Organizations can’t expect AI solutions to provide valuable results when data is siloed, processes aren’t documented, and employees switch between several systems that aren’t interconnected. Although it’s possible to integrate AI with legacy systems, these integrations still require thorough preparation of the data infrastructure.
Building data pipelines that fetch relevant and high-quality data from centralized data storage, including both structured and unstructured datasets, is the first step in implementing AI. Because without this foundation, your project will inevitably fail at the production stage.
The best way to achieve a high level of AI system accuracy is to build enterprise knowledge bases based on retrieval-augmented generation (RAG) with real-time access to all internal documentation and feed AI solutions with continuous company data.
When real-time enterprise data becomes the lifeblood of your AI system, it produces reliable outputs that bring tangible value to your business, including faster decision-making, reduced operational costs, improved customer experiences, and new revenue opportunities.
#4. Lack of in-house capacity to maintain AI systems
A shortage of AI engineers who can implement, maintain, and fine-tune gen AI solutions can lead to stalled projects, slower adoption across business units, and ultimately, failure to realize the promised ROI.
When facing AI skills shortages, 49% of enterprises are investing in upskilling or reskilling their in-house employees, while 46% of companies are cooperating with external IT integrators and consultants to bridge the gaps. The choice depends on the budget and time-to-market requirements.
Cultivating internal AI skills can yield better long-term results if you’re planning on more AI projects in the future. However, partnering with expert AI and data engineers can also prove effective, as you pay for the AI project during development and then shift to an on-demand payment for system maintenance and support. You get to tap into vast AI knowledge and expertise without any extra expenses on maintaining an internal AI department.
#5. Ineffective change management practices or their absence
Without strategic change management and AI adoption strategies (e.g., clear communication, phased rollouts, executive buy-in, employee training, and feedback loops), AI experimentation as well as enterprise-wide AI adoption can be catastrophic.
Prioritize solving specific problems for users or employees and introduce AI as a solution and enabler. Comprehensive training programs and security guidelines build user trust, accelerate adoption rates, and encourage consistent usage patterns that deliver faster business benefits.
Business unit leaders, HR, and learning and development specialists can support your mission by managing and facilitating the adoption of AI.
#6. Complex TCO of AI projects
Similar to ROI difference, traditional IT costs (maintenance and service fees) are mostly predictable, whereas gen AI costs are unpredictable and volatile. That’s why initial investment during experimentation can differ from the costs necessary to launch AI in production and maintain it.
Gen AI models continuously learn and can drift over time if not adequately monitored and managed. Thus, maintenance costs for AI software can vary depending on the level of fine-tuning efforts.
AI volatility can also increase AI infrastructure costs, as different computational, training, and inference tasks put varying pressures on hardware and software AI components. In this respect, the decision to run AI software in the cloud or on-premises is crucial. While cloud deployment allows you to benefit from cloud FinOps for efficient cost tracking, an on-premises AI rollout provides more control over your infrastructure.
To optimize performance, ensure flexibility, and reduce GPU usage costs, Toyota has adopted a hybrid approach to launch their AI platform. They reduced the number of on-premises servers to one and use it for normal operations, while scaling to the cloud environment for peak demand. With the hybrid approach, the Toyota team reduces the current TCO while future-proofing software for scaled demand.
To implement AI for ROI and drive transformative enterprise value, ensure you:
- Have a clear business goal with a focus on real business or customer problems (rather than hype or the FOMO effect)
- Continuously measure adoption and implementation results with business-specific KPIs
- Feed your AI system with high-quality proprietary data in real time
- Onboard skilled specialists and foster AI adoption with clear-cut change management strategies
- Compose a well-planned AI budget to avoid over- or underspending and ensure the successful launch of your AI project in production, as well as its gradual scaling
These steps can bring you closer to ROI-positive AI projects, but to truly understand what works in practice, it’s worth looking at how leading enterprises succeed with AI.
Breaking the missed-ROI pattern: Lessons from gen AI leaders
The Google Cloud survey on gen AI ROI discovered that companies leading in AI initiatives have four or more AI projects in production and have invested more than 15% of their operating expenses in AI. These strategic investments generate higher and faster ROI across multiple use cases compared to organizations with less strategic AI use.
But high investments and scaled AI use are already the characteristics of them as leaders, and here are three decisions that helped them become those leaders:
- Clear vision for future growth. Among business goals, they prioritize AI adoption for improved customer experience and the development of new products and services, rather than optimizing only current operational needs.
- Aligned technology and business objectives. They have a clear understanding of how the technological benefits of AI tie to their business strategy.
- Dedicated gen AI teams. Leaders in gen AI projects prioritize building specialized AI teams that drive technological improvements but also foster cross-company adoption.
In line with our conclusion in the section on reasons for missed AI ROI, Google’s report confirms that core drivers of AI success are teams with a clear vision of AI benefits, not only today but also in the future.
The McKinsey survey yields similar findings on what differentiates leaders in AI initiatives from those who are still figuring out how to derive value from this breakthrough technology. Here’s what Bryce Hall, Associate Partner at McKinsey, said on the matter:
We’re now far enough into the gen AI era to see patterns among companies that are capturing value. One significant difference is that these companies focus as much on driving adoption and scaling as they do on the up-front technology development.
This is not just hand-waving. Instead, they are following specific management practices that enable them to be successful—such as developing a clear roadmap for scaling, establishing and tracking KPIs, and driving change management by ensuring senior leaders are actively engaged in driving gen AI adoption.
How do real-life enterprises adopt AI and ensure ROI with it?
Walmart invested in gen AI training before it got mainstream
When generative AI emerged, the Walmart AI/ML team began training open-source large language models (LLMs) to match their business specifics and those of the retail industry in general. This decision enabled them to experiment with AI sooner than most of their competitors and implement it across the entire company.
But AI experimentation wasn’t random. They set five objectives: improve customer experience, developer productivity, operations, and generate content. To measure their results and correct the direction if necessary, Walmart has established specific checkpoints for measuring AI efficiency. They focus on model quality evaluation, A/B tests, and human feedback to keep AI experimentation and production-ready models under control.
When their product catalog became much bigger as more and more retailers were getting on the platform, offering an online shopping experience, the company implemented a gen AI solution with multiple LLMs to create, clean, and improve 850 million data elements. To do the same task manually, Walmart would’ve required nearly 100 times their current workforce.
With an improved catalog, Walmart gathered valuable insights into customer shopping habits. By introducing AI-powered search and sales assistants, the company also saw an increase in sales, as customers could quickly find what they needed. As a payoff, in Q2 2024, they achieved 4.8% revenue growth and 21% growth in the e-commerce function. Such results they mainly attributed to generative AI initiatives.
The example of Walmart demonstrates that to succeed with AI and achieve a high ROI, you should have a specific goal for your AI initiatives, measure its impact at set milestones, and train generative AI solutions with custom data to match your business’s specific needs.
Sentara Health sees 4 times ROI from the pilot gen AI program
Sentara Health has adopted gen AI technology to facilitate quick and efficient chart reviews in the electronic health record (EHR) system, providing a draft assessment of the patient and saving clinicians’ time while increasing documentation accuracy.
What takes clinicians hours of manual search, an AI system performs in seconds, and most importantly, it retrieves the most comprehensive information on the patient, which clinicians could overlook after repeating this process for multiple patients in a day. Such accuracy and attention to detail are what particularly convinced clinicians to use AI after they tried it during the pilot program.
However, to ensure active adoption and use, Sentara Health has also identified AI champions among physicians to serve as informal leaders who can demonstrate to their colleagues the efficacy of AI. They also established an AI oversight program to validate AI solutions, check for drift, and ensure their security and proper integration into the hospital workflow.
Already at the pilot stage, the company could secure 2-4 times ROI per clinician. Such a success of AI implementation at the administrative level in one hospital prompted scaling AI use to all 12 hospitals.
Here is how the Chief Health Information Officer at Sentara Health, Joe Evans, explains their success:
So, from the view of our CFO and hospital operations leaders, the pitch to them is to be able to show the hard ROI and the benefit of capturing the CCS and MCCs [Complications or Comorbidities and Major Complications or Comorbidities] to help with DRG [Diagnosis-Related Groups] upgrades, which helps with hospital reimbursement.
And it’s easy to map out to them, and that’s what we did after the pilot. We could say this is what we spent on this solution, and this was our hard return on investment. And those results are what helped us be able to spread it through all 12 hospitals.
Sentara Health’s success hinges on a clear problem, effective adoption strategies, and a pilot program designed to measure its impact, even on a small scale. As a result, what started with simple experiments and AI implementation in one hospital, yielding a clear ROI, now extends to more facilities and medical departments, promising even higher ROI, as more clinicians use AI in their workflows.
Bottom line
When investing in AI solutions, you’re investing in the future. AI implementation requires significant preparation, including setting up infrastructure, establishing data pipelines, and enabling the team to work effectively.
For these efforts to pay off, you need time and enough resources to support the volatile nature of AI initiatives. But once you pass these initial stages of AI experimentation, prototyping, A/B testing, and feedback loops, you’ll gain confidence to invest more lavishly into your AI projects and scale them across business functions to become a gen AI leader in your industry.
Xenoss can be by your side the whole time, from AI feasibility study and data infrastructure assessment to team training and comprehensive AI ROI measurements.
