Proving it: ROI and the honest scorecard

The most-cited agent deployment in the world, and deservedly so at launch. Klarna's AI assistant, built on the LangChain stack, published these figures:

• 2.5 million conversations handled - roughly two-thirds of the company's support chats.
• Resolution time from 11 minutes to 2 minutes - the metric customers actually feel.
• Output equivalent to ~700 full-time agents - the line every headline ran with.
• A projected $40M profit improvement - the line every CFO ran with.

Hold these numbers with respect: they are what a successful large-scale agent deployment looks like. And hold your applause: Part 2 shows what the same deployment looked like a year later, and why both views belong in the same board pack.

The less-headlined cases are in some ways more instructive - narrower, internal, and easier to measure honestly:

• Uber: agentic tooling for large-scale code migration - around 21,000 developer hours saved. Internal, countable, no customer emotions in the loop.
• AppFolio: a property-management copilot that roughly doubled response accuracy - a quality win, not a headcount story.
• LinkedIn: an internal SQL Bot that lets analysts self-serve data questions - value measured in unblocked analysts rather than press releases.

The pattern worth stealing: the most defensible ROI stories are internal-facing, have a clear human baseline, and measure a boring number well. If your first agent's business case looks like Uber's rather than Klarna's, that is a feature.

Beyond the marquee names, the survey benchmarks give you calibration for your own plans:

• 41% of deployments report positive payback within 12 months - agents can pay for themselves, and most still do not within a year. Both halves of that sentence matter.
• Median time-to-value is 5.1 months - budget for two quarters of investment before the curve bends.
• Use case changes everything: SDR-style sales agents average 3.4 months to value; finance and operations agents average 8.9 months. If a vendor promises finance-agent value in six weeks, you now know to smile and check the scorecard.

A year after the headlines, Klarna adjusted course. Customer satisfaction had dropped - the agent handled routine cases fine but disappointed on complex and emotional ones. The company rehired human agents for exactly those cases and settled into a deliberate hybrid model.

• Read it precisely: the $40M-class efficiency numbers largely survived. What died was the "replace everyone" narrative - the deployment was oversold, not worthless.
• The lesson is metric selection: every celebrated number was a cost or speed metric. The dissenting signal - CSAT - was a quality metric, and it was not on the headline scorecard. The problem announced itself in the one place nobody had put a spotlight.
• The leadership takeaway: any scorecard composed only of cost and speed metrics will read as triumphant right up until the customers leave. Quality metrics are not decoration; they are the early-warning system.

Zoom out from one company to the population and the skeptics hold strong cards:

• Gartner: forecasts over 40% of agentic AI projects canceled by end-2027 - the named causes are cost, unclear business value, and inadequate risk controls. Notice: all three are leadership failures from sessions a3-a5, not model failures.
• MIT: the GenAI Divide research found 95% of pilots show no measurable P&L impact - carried, as always in this track, with its methodology caveat: definitions of impact vary and the sample skews, so treat it as a warning shot rather than a precise census.
• The scale gap: roughly 78% of organizations report piloting agents while only about 14% have scaled them - the graveyard between pilot and production is where unclear metrics send projects to die.

Put Part 1 and Part 2 together and the resolution is not "agents work" or "agents fail". It is: agents pay back where somebody defined payback honestly in advance. Which is why this session ends with a scorecard and not a verdict.

Most ROI evidence you will ever see arrives via vendor marketing. Three filters before you believe a number:

• Company-published: the metrics were chosen by the party with the strongest incentive to look good. Not fake - curated. Ask what is NOT reported: no CSAT in a support-agent case study is a choice.
• Survivor bias: case studies are written about the deployments that worked. The 40% that get canceled do not get landing pages. Every case study is drawn from the winners' bracket.
• Post-hoc metric selection: when success is measured after the fact, the teller picks whichever number moved. The defense is the same discipline you will draft today: metrics defined before launch, including the ones that could dissent.

Demand these five before funding scale-up. Anything less is a story, not a scorecard.

And the scorecard's spine: every metric is measured against a human baseline captured before launch, and the whole card is defined before the pilot - so nobody gets to choose the flattering number afterwards. That single discipline is what separates your board pack from a vendor landing page.