Stop Measuring AI Products Like It's SaaS
This piece reflects Ema's own view. Three passages are quoted verbatim from Gartner, "Prove the Value of Your AI Product by Moving Beyond Usage Metrics," by Vuk Janosevic, David Yockelson, and Radu Miclaus, 23 June 2025. All other analysis and commentary are Ema's own. View the full report on Gartner.
For a decade, product leaders have judged software success by the same handful of signals: monthly active users, Net Promoter Score, time-to-value. These metrics made sense for a world where software was a tool people opened, used, and closed. More usage meant more value. The two moved together.
That relationship has broken. AI products don't wait for a human to open a dashboard. They act, decide, and execute on their own. The value they create often shows up nowhere near a login screen: an error that never happened, a decision that got made three days faster, a risk that got flagged before it became a write-off. None of that registers on a usage graph.
The resulting confusion is now measurable.
Sixty percent of tech providers cite difficulty isolating generative AI's (GenAI's) impact from other factors, and 54% point to the lack of standard ROI frameworks.
Product leaders who can't answer "what did the AI actually do" are stuck defending price increases with adoption charts. That's a losing argument, and it's only getting more common.
The fix isn't a better dashboard. It's a different definition of what's worth measuring in the first place.
Where legacy metrics break down
Three properties of AI products explain why traditional SaaS measurement doesn't transfer.
- 01
Probabilistic outcomes. Gains emerge not as clicks or sessions, but as fewer errors, better choices or new decisions altogether.
- 02
Latent value. Efficiency, risk reduction and performance lift may materialize weeks after deployment, invisible to standard dashboards.
- 03
Outcome indexes measure the strategic business impact, how AI contributes to cost reduction, revenue growth or risk mitigation.
Each one breaks a different assumption. A traditional feature does the same thing every time it's clicked, and its value shows up as a click. An AI agent's value shows up as a decision it made differently than a human would have, and there's no click to count. Standard weekly or monthly dashboards are built to catch immediate activity, not value that accrues quietly and shows up later, once the system has processed enough real-world cases to compound its advantage. And when a person and an agent jointly produce an outcome, that outcome is easy to misattribute: credited entirely to the human, or written off as "the tool helped a bit."
Put together, these three properties mean a product leader who only tracks usage is structurally blind to most of what their AI product is doing. The system can be delivering serious value and the dashboard will show nothing unusual.
The AI product value framework
The alternative is a three-layer framework that connects technical performance to business outcomes, giving product leaders one consistent language to use with engineering, customer success, and the C-suite.
- 01
Technical health metrics ensure that performance, reliability and responsible operation stay within acceptable thresholds.
- 02
Operational proxies reflect how AI reshapes day-to-day execution, signaling early value through workflow improvements and time savings.
- 03
Outcome indexes measure the strategic business impact, how AI contributes to cost reduction, revenue growth or risk mitigation.
What this looks like in practice
We built our product around a version of this same instinct, long before we saw it laid out this way. When a customer's AI Employees go live, the question we care about from day one isn't how many tickets got touched. It's whether the outcome the customer actually cares about (a faster resolution, a cleaner handoff, fewer escalations) moved, and whether we can show our fingerprints on that movement rather than just claim credit for it.
That means baseline numbers get set before an agent goes live, not after. Every improvement gets checked against what the customer's own systems already say happened, not against a dashboard we control. It's slower to set up than shipping a usage report. It's also the only version of the story a CFO actually believes.
The window is closing
Usage metrics were never wrong. They were just built for a category of software that no longer describes what AI products do. An agent that resolves a ticket end-to-end, prevents an error before it happens, or surfaces a decision three days earlier than a human would have doesn't show up as a session. It shows up as a business outcome, and only a framework built to trace technical performance through operational change to business result will catch it.
Product leaders who build this measurement discipline now get to define what "AI value" means for their category, and set the pricing and positioning conversation on their own terms. Those who keep leaning on adoption charts will keep losing the argument to whoever in the room can point to a number the CFO already trusts.
The shift from counting usage to proving outcomes isn't optional anymore. It's the difference between a product that's merely being used and one whose value is being recognized, trusted, and paid for.
This piece reflects Ema's own view. Three passages are quoted verbatim from Gartner, "Prove the Value of Your AI Product by Moving Beyond Usage Metrics," by Vuk Janosevic, David Yockelson, and Radu Miclaus, 23 June 2025. All other analysis and commentary are Ema's own. Gartner subscribers can view the full report on Gartner.
GARTNER is a trademark of Gartner, Inc. and/or its affiliates.