How to Measure AI ROI: A KPI-First Framework

Why AI ROI conversations go sideways

The same scene plays out across mid-market companies from Bogotá to Austin. An engineering team ships an AI system. Six months later, a CFO asks what it returned. Someone pulls up a dashboard. The dashboard shows model accuracy and p95 latency. The CFO asks again, slower: what did it return?

The conversation goes sideways because the team measured the model and the business measured the outcome. Both were correct inside their own frame. Neither answered the actual question.

We see the same pattern in almost every engagement where we come in to recover a stalled initiative. The technical work was solid. The instrumentation was missing. Without a baseline captured before go-live, without a clean attribution story, and without a shared metric, the team ends up defending the model instead of reporting the value. That is the failure mode this post fixes.

If this pattern feels familiar, the adjacent post on why most AI pilots fail covers the upstream causes — scoping and sponsorship issues that produce the same symptoms.

The three ROI questions every executive asks

Strip away the ceremony and executives ask the same three questions about any AI investment. Answer them cleanly and you will earn the next round of funding. Answer them vaguely and the program dies at budget review.

1. Did it move the number we care about? Not accuracy. Not F1. The primary business KPI — loss rate, average handle time, inventory turns, gross margin on a segment.
2. How much of that movement is because of us? The attribution question. A 12 percent lift means nothing if the market moved 11 percent for reasons you did not cause.
3. Can we trust the number? The measurement integrity question. If the baseline is shaky or the window is cherry-picked, sophisticated executives will sniff it out immediately.

Every piece of the framework below is engineered to answer one of these three questions. When you design instrumentation with this as the brief, the rest becomes mechanical.

The KPI-first framework

The framework has five steps. In a well-scoped engagement, steps one through three happen before a single line of model code is written. Steps four and five run continuously once the system is live.

1. Baseline. Measure the current-state KPI with the same definition, data source, and time window you will use post-launch. If the baseline cannot be measured, the use case is not ready.
2. Target. Set a target KPI delta that is ambitious but defensible. Anchor it to comparable benchmarks and to the sensitivity analysis on your baseline.
3. Instrumentation. Wire the telemetry before you ship. Model outputs, business outcomes, and the link between them. Instrument the counterfactual path too — what happens when the AI is not invoked.
4. Attribution. Choose the cleanest causal design your context allows: randomized A/B, holdout cohort, geo split, or pre-post with matched controls. Document the limits.
5. Reporting. Build one dashboard for the technical team and one for the business. Keep the business view to three numbers maximum. Refresh weekly.

The biggest single predictor of whether an AI system will defend its ROI is whether the baseline was captured before launch, with the same definition used at measurement time. Everything else is downstream of that one decision.

The instrumentation step is where most teams underinvest. They build beautiful training pipelines and wire up Prometheus for latency, then realize three months in that they have no clean way to join a model prediction to the business outcome it influenced. Budget two to three weeks of data engineering for joinable, auditable event logs before you think about production traffic. Our post on data engineering foundations for ML teams goes deeper on the event schemas that make this work.

Direct vs indirect ROI — and why both matter

Direct ROI is the kind finance understands without translation. A fraud model avoids 180 thousand dollars of chargebacks in a quarter. A forecasting model reduces safety stock and frees working capital on a specific SKU family. A churn model retains a cohort worth a measurable amount of annual recurring revenue. These are dollars on a P&L line with a defensible attribution story.

Indirect ROI is equally real, but finance needs help to see it. Analyst capacity freed when classification is automated. Time-to-insight cut from three days to four hours. Regulatory risk reduced because a control that was sampled is now comprehensive. These show up as second-order effects: faster hiring pauses, lower contractor spend, avoided fines.

The mistake is either pretending indirect ROI is not real or blending it with direct ROI in the same line item. Report them side by side but in separate rows. Label indirect ROI as "capacity-equivalent" or "risk-avoided" so no one in finance double-counts.

The table is a starting point, not a rulebook. The exact primary signal depends on your accounting and the granularity at which finance already reports. Always reconcile your AI ROI definition with how finance books the relevant line today.

Attribution: proving causality without running a 6-month RCT

A randomized A/B test is the cleanest way to prove causality, and when it is feasible you should run one. In many real contexts it is not. You cannot randomly withhold fraud detection from half your customers. You cannot give a subset of the warehouse the wrong inventory plan. You cannot let agents see personalized recommendations for some users and random ones for others if the legal team has ruled that out.

When a full RCT is off the table, these designs, in rough order of rigor, cover almost every situation:

• Holdout cohort. A permanent or rotating sample where the AI is not invoked. This is usually tractable even in sensitive domains if you negotiate the size carefully with compliance.
• Geo or segment split. Launch in one region or one customer segment first, keep another as control. Practical for rollouts that are operationally staggered anyway.
• Interrupted time series with matched controls. Compare the treated entity before and after launch, with a synthetic control built from similar untreated entities. Effective when the launch is a clean discontinuity.
• Difference-in-differences. Similar to above, but with a control group that moves in parallel before the intervention. The shared pre-trend is the key assumption to test.
• Pre-post with documented assumptions. The weakest but often the only option. Works if you name the assumptions explicitly and stress-test them with sensitivity analysis.

Whichever design you pick, write the measurement plan in a one-page document before the system launches. Have the business owner and the finance partner sign it. The single biggest source of ROI disputes later is someone relitigating the measurement definition after the numbers come in.

Building a board-ready ROI dashboard

A board-ready dashboard is not a technical dashboard with an executive skin. It is a different artifact for a different audience. The technical dashboard has model drift charts, confusion matrices, and per-segment performance. The board dashboard has three things and nothing else:

1. The primary KPI trend, with a visible baseline line and a target line.
2. The dollar impact year-to-date, with a clear split between direct and indirect.
3. A confidence indicator — a simple label like "high / medium / low" backed by the attribution design and sample size.

Everything else goes on page two or deeper. Footnotes for methodology. Sensitivity analysis. Breakdown by segment. The first screen exists to let a busy executive answer the three questions from section two in under ten seconds.

On tooling: the medium matters less than the discipline. We have built these dashboards in Looker, Metabase, Hex, and plain Google Sheets with a scheduled update. What matters is that the source of truth is versioned, the definitions are frozen, and one owner signs off on every refresh. For teams that want the operational monitoring side done right, our writeup on MLOps from notebook to production covers the technical observability stack that feeds this dashboard.

Real outcomes: patterns from our engagements

A few anchor points from the engagements we have delivered at sesgo.ai, with the measurement design we used:

At Soflex, we built an AI-assisted triage system for emergency response. The primary KPI was manual workload on operators, measured as share of calls requiring full manual classification. Baseline was a four-week pre-launch sample under normal load. Target was a 30 percent reduction. Actual outcome: 42 percent reduction, with a holdout cohort that confirmed the effect was not driven by seasonality. Indirect ROI — faster triage during surge events — was reported separately and not monetized.

At PSAG, we shipped computer vision for SKU classification on a production line. Primary KPI was classification error rate against a curated ground-truth sample. Baseline was the pre-existing manual process over a matched production volume. Target was 50 percent error reduction. Actual outcome: 85 percent error reduction. Direct ROI was quantified against rework cost per unit and reported to the operations committee monthly.

At Arison, we deployed AI agents for warehouse management. Primary KPI was inventory holding cost as a percentage of revenue, with stockout rate as a guardrail metric. Baseline was the twelve-month prior trend, adjusted for volume. Outcome: 35 percent reduction in inventory cost with stockout rate held flat. Working capital released was tracked separately by finance.

Three patterns repeat across these. First, the primary KPI was locked and instrumented before launch in every case. Second, at least one guardrail metric was defined to catch unintended consequences. Third, the attribution design was documented and signed off, so when the numbers arrived there was nothing to renegotiate. None of this is glamorous engineering work, but it is the difference between a system that earns the next budget cycle and one that does not.