Stop Measuring AI Products Like It's SaaS
This piece draws on findings from Gartner's June 2025 report, "Prove the Value of Your AI Product by Moving Beyond Usage Metrics," by Vuk Janosevic, David Yockelson, and 1 more.
For a decade, product leaders have judged software success by the same handful of signals: monthly active users, Net Promoter Score, time-to-value. These metrics made sense for a world where software was a tool people opened, used, and closed. More usage meant more value. The two moved together.
That relationship has broken. AI products don't wait for a human to open a dashboard — they act, decide, and execute on their own. The value they create often shows up nowhere near a login screen: an error that never happened, a decision that got made three days faster, a risk that got flagged before it became a write-off. None of that registers on a usage graph.
Gartner Research puts a number on the resulting confusion: 60% of tech providers say they can't isolate generative AI's impact from everything else happening in the business, and 54% say there's no standard ROI framework to even attempt it. Product leaders who can't answer "what did the AI actually do" are stuck defending price increases with adoption charts. That's a losing argument, and it's only getting more common.
The fix isn't a better dashboard. It's a different definition of what's worth measuring in the first place.
Where legacy metrics break down
Three properties of AI products explain why traditional SaaS measurement doesn't transfer.
The first is probabilistic outcomes. A traditional feature does the same thing every time it's clicked — its value shows up as a click. An AI agent's value shows up as a decision it made differently than a human would have: an error avoided, a better option surfaced, a task completed end-to-end instead of half-finished. There's no click to count.
The second is latent value. Efficiency gains and risk reduction from an AI system often surface weeks after deployment, once the system has processed enough real-world cases to compound its advantage. Standard weekly or monthly dashboards are built to catch immediate activity, not value that accrues quietly and shows up later.
The third is outcome dilution. Most AI deployments today sit inside human-plus-AI workflows, not fully autonomous ones. When a person and an agent jointly produce an outcome, that outcome is easy to misattribute — credited entirely to the human, or written off as "the tool helped a bit." Without a deliberate attribution method, AI's actual contribution gets buried inside a process metric that was never designed to separate the two.
Put together, these three properties mean a product leader who only tracks usage is structurally blind to most of what their AI product is doing. The system can be delivering serious value and the dashboard will show nothing unusual.
The AI product value framework
Gartner's response to this gap is a three-layer framework that connects technical performance to business outcomes, giving product leaders one consistent language to use with engineering, customer success, and the C-suite.
Layer one is technical health metrics — the foundation. These track whether the system is trustworthy and operating within acceptable bounds: model accuracy, latency, infrastructure cost per inference, drift detection, confidence thresholds, and decision auditability. This layer answers a narrow question — is the system behaving reliably — and nothing more. It's necessary, but on its own it says nothing about business impact.
Layer two is operational proxies — the "heartbeat" metrics that line managers actually check week to week. Time saved per task, workflows completed without human intervention, hand-offs eliminated, opportunities surfaced, decisions accelerated. These are the earliest visible signal that an AI system is changing how work gets done, well before that change shows up in a quarterly business review.
Layer three is outcome indexes — the layer executives care about. Cost avoided, revenue influenced, errors or risk events prevented, shifts in customer or employee satisfaction. This is where AI activity finally translates into the language a CFO or board will act on.
Before any number gets promoted to one of these layers, Gartner applies three filters worth adopting as a discipline: Is the outcome observable through system data or user feedback? Is it credible — does it align with what stakeholders already care about? And is it attributable — can it be reasonably traced back to the AI's intervention rather than manual effort running in parallel? An outcome that fails any of these three filters isn't a metric yet. It's a claim.
What this looks like in practice
The framework is only useful if it's built into the product itself, not bolted on as a reporting exercise after the fact. Gartner's research points to a few instrumentation patterns worth studying.
OpenEnvoy, an AI-native AP/AR automation platform, anchors its value tracking in exactly this layered structure. At the technical layer, the team monitors automation rates and exception handling to confirm the model is performing consistently in live workflows. At the operational layer, it tracks billing discrepancies, duplicate detection, and vendor compliance patterns — signals of process effectiveness, not just system health. At the outcome layer, it quantifies the financial exposure identified and prevented, often surfacing material overpayments within a customer's first month. Reviewing all three layers together, rather than any one in isolation, is what makes the resulting ROI story hold up in front of a buyer.
Ema, an enterprise agentic AI platform, making AI Employees for HR, IT, and Finance, takes a similar approach but starts even earlier — during onboarding, before a single agent is deployed. Teams work with each client to define the specific business outcome they're targeting, whether that's faster response times or higher resolution rates, and capture a baseline before deployment so that any later improvement can be attributed with confidence. From there, telemetry tracks technical performance, operational metrics quantify how much work agents absorb from human teams, and the results are contextualized against a client's own CRM or support-platform data — connecting agent activity directly to a metric the client was already tracking.
Different products, same underlying discipline: value tracking has to be designed into the system from day one, not reconstructed from usage logs after a renewal conversation goes sideways.
The window is closing
Usage metrics were never wrong — they were just built for a category of software that no longer describes what AI products do. An agent that resolves a ticket end-to-end, prevents an error before it happens, or surfaces a decision three days earlier than a human would have doesn't show up as a session. It shows up as a business outcome, and only a framework built to trace technical performance through operational change to business result will catch it.
Product leaders who build this measurement discipline now get to define what "AI value" means for their category — and set the pricing and positioning conversation on their own terms. Those who keep leaning on adoption charts will keep losing the argument to whoever in the room can point to a number the CFO already trusts.
The shift from counting usage to proving outcomes isn't optional anymore. It's the difference between a product that's merely being used and one whose value is being recognized, trusted, and paid for.