The KPIs that actually matter for production AI agents

Your AI agent handled 10,000 tasks last month. But how many did it get right — and how would you know? As organizations move from generative AI (chatbots, content creation, information retrieval) to agentic systems that reason, plan, and execute autonomously, the question of measurement becomes urgent. This transition represents a shift from augmenting human thought to automating human labor. One of the most common questions our customers are asking is: 

“How do we measure success and ROI from our investments in agentic AI?”

The evaluation metrics used for large language models (LLMs) — such as perplexity, BLEU scores, or simple thumbs up/down user feedback — do not suffice for assessing autonomous agents. As organizations deploy multi-agent systems, evaluation becomes more nuanced. We’ve previously discussed that gen AI requires developing a new set of AI metrics and approaches. In this post, we present a framework of Key Performance Indicators (KPIs) to measure Agentic AI investments, organized around three pillars:

1. Reliability & operational efficiency: Can the agent handle complex workflows consistently and cost-effectively?

2. Adoption & usage patterns: How well does the agent integrate into existing workflows, and are people using it?

3. Business value: Is the agent increasing productivity or generating net new value?

We developed this framework through our work with the Google Cloud AI documentation team and drew on insights from Google Workspace on user adoption. As product innovation accelerates, our documentation team’s technical writers face a similar challenge: keeping documentation current. To address this, we built a modular collection of specialized AI agents using Google’s Agent Development Kit (ADK), integrated directly into engineering systems and technical writing workflows. These include a resolution agent (RA) that drafts fixes for documentation bugs, subject to human verification, and a quality check agent that scans for factual errors. 

While our example focuses on documentation, similar metrics can apply to other agent deployments: customer service, sales support, IT operations, or internal workflows. The metrics translate; only the context changes.

Pillar 1: Reliability and operational efficiency 

As agents move from single-turn tasks to multi-step workflows, measuring success requires evaluating the trajectory — the sequence of thoughts and actions — not just the final output. These metrics confirm that an agent reached the right answer through sound reasoning, not a lucky guess, which is essential for reliability and scale. 

Agent reliability

To measure reliability at scale, you can use a critic agent: a secondary, specialized model tasked with auditing the primary agent’s execution logs. The critic agent reviews the user's initial prompt alongside the primary agent's trace (the step-by-step log of thoughts and tool calls). It converts subjective behaviors into objective metrics measured against two standards: the plan and organizational policies. Of the metrics below, plan adherence and argument hallucination rate tend to surface issues fastest — start there.

• Tool selection accuracy: Did the agent choose the right tool for the subtask?

• Argument hallucination rate: Did the agent invent parameters for a function call? This happens when an agent calls a function without the required input in context or incorrectly infers the parameter. 

• Plan adherence: Did the agent call Tool A, then Tool B, then Tool C in the correct order? Compare the initial plan against the actual execution log. Significant deviation may indicate reasoning instability.  

• Consistency score: If the agent receives the same question 10 times, how much does the tool usage path vary?

• Defiance rate (misuse detection): Can the agent detect a malicious prompt and refuse to act? Measure how often your guardrails trigger successfully, and test with workflow-specific adversarial scenarios. 

Operational efficiency

Operational efficiency metrics answer a fundamental question: is the agent efficient enough to run at scale? Cost per successful task is the metric that matters most here — it forces you to pair cost with outcomes rather than measuring tokens in isolation.

• Cost per successful task: Traditional cost-per-token metrics can be misleading for agents. If an agent costs $0.10 per run but fails 50% of the time, your actual cost per successful outcome doubles. Pair cost metrics with success rates.

• Planning efficiency: A well designed agent recognizes when to offload work to a tool (for example, running a script rather than asking an LLM to parse a file manually), which reduces token count. This metric evaluates whether the agent reduces unnecessary reasoning by using tools effectively. By mimicking human workflows, minimizing context, and prioritizing tool calls, you can design for high planning efficiency — helping the agent reach a solution through the most direct path. 

• End-to-end latency: In conversational interfaces, time to first token (TTFT) was the standard metric for perceived responsiveness. For agents, end-to-end trace latency — the total time from initiation to final resolution — matters more. While raw speed shouldn’t overshadow outcome quality or cost for asynchronous tasks, this metric remains an important indicator of system health. As agents become more capable, they can fall into analysis paralysis, cycling through reasoning steps without taking action.

Pillar 2: Adoption and usage 

Adoption metrics reveal how much value your agent adds to workflows and provide insight into your organization’s AI fluency. Agent adoption is splitting into two complementary models: reactive (user-invoked) and proactive (background or system-initiated). 

Reactive agents

Reactive agents act only when they receive explicit user input. In Google Workspace, examples include the Gemini side panel (available in Docs, Sheets, Slides, and Drive) and the "help me write" feature.  These metrics help evaluate how well reactive agents are working:

• Active users: Do people try the agent once and abandon it, or does it become part of their daily routine? Monitor daily, weekly, and monthly usage across departments to track habit formation.

• User sentiment: Surveys and focus groups provide qualitative and quantitative data on sentiment and net promoter score (NPS). Combined with usage metrics, sentiment data helps pinpoint friction: is the issue awareness or experience quality?

• Invocation rate: How often do people invoke the agent (for example, opening the Gemini panel in Workspace)? Measure per active user or per session within a given timeframe.

• Session depth: How many follow-up questions does a user ask?

• Retention rate of generated text: If someone keeps 80% of the AI-generated draft, the agent succeeded. If they delete it and start over, the agent failed.

Proactive agents

Proactive agents operate in the background as event-driven partners, managing context and reducing the cognitive load of initiation. Examples include our resolution agent (RA), which activates when a documentation bug is assigned, and Workspace Flows, which automates work across Google Workspace apps. For proactive agents, acceptance rate and implicit rejection rate tell you the most about real-world performance.

• Acceptance rate: How often do people accept the agent's output without significant edits?

• Implicit rejection rate: Explicit feedback (thumbs down) is rare. The real signal is the undo or revert. If an agent commits a fix that a human later reverts, that's a strong indicator of friction.

• Handoff ambiguity and verification latency: We found that ownership drives speed. When a technical writer owned the bug and the RA assisted, verification was fast because the human felt responsible. When the RA fully owned the bug, the team experienced a bystander effect — uncertainty about who should verify the work led to longer cycle times despite the automation. We settled on positioning the agent as a collaborator. A time-to-verify metric measures the gap between agent completion and human approval — if review takes longer than doing the task manually, friction outweighs value.

• Output friction: How often does a human need to step in and take over a task the agent started? High intervention rates signal a trust issue and suggest the agent may work better in a reactive mode. For the resolution agent, we measured intervention rates by tracking the percentage of changelists that were (a) accepted as-is, (b) accepted with edits, or (c) reverted.

The sweet spot for adoption lies in minimizing friction. Smart Compose in Gmail succeeds because it has almost no input friction (it watches you type) and almost no output friction (hit Tab to accept or keep typing to ignore). The cost of rejection is negligible. In contrast, fully autonomous agents that perform complex tasks (like planning a week) continue to evolve their friction profiles.

Pillar 3: Business value

For business stakeholders, the focus is tangible improvement in outcomes compared to traditional methods, not the underlying technology. Time-to-value acceleration is typically the clearest proof point — it connects directly to productivity gains stakeholders can see. For agentic applications, we recommend tracking these metrics:

• Time-to-value acceleration: Average time reduction per agent-assisted workflow. In our documentation team, writers now start with a changelist already drafted by an agent rather than beginning from scratch for simple bugs. Result: A dramatic reduction in triage overhead and in end-to-end resolution cycles resulting in a step-change improvement in velocity, allowing the team to clear backlogs faster.

• OpEx reduction: How many manual steps did the agent remove, and what was the impact on business metrics? If an agent handles a percentage of support tickets without escalation, that's direct, quantifiable cost reduction.

• New capabilities unlocked: This is where agents deliver ROI by enabling workflows that weren't previously possible. For example, we can now run factuality and style checks across our entire generative AI documentation on demand — a task that would have been impossible for a human team to complete manually.

• Revenue acceleration: Agents can shorten time-to-close by automating cross-functional workflows, such as RFP responses or sales support. In our case study, faster documentation fixes from the resolution and quality check agents improved developer experience and made product adoption more efficient.

Conclusion 

For executives and product managers starting this journey, begin with reliability and efficiency metrics to build trust in enterprise workflows. Then, focus on adoption by reducing friction in how people interact with agents. With that foundation, you'll be positioned to measure true business value and ROI.

Organizations that build in measurement from day one see the strongest returns from agentic AI. If you're ready to start building, explore Google's Agent Development Kit to see how these principles translate into practice.

Acknowledgements - In addition to the authors, Hussain Chinoy and Mikhail Chrestkha greatly contributed to this post.