Developers & Practitioners

A methodical approach to agent evaluation: Building a robust quality gate

November 17, 2025

Hugo Selbie

Staff Customer & Partner Solutions Engineer, Google

Try Gemini 2.5

Our most intelligent model is now available on Vertex AI

AI is shifting from single-response models to complex, multi-step agents that can reason, use tools, and complete sophisticated tasks. This increased capability means you need an evolution in how you evaluate these systems. Metrics focused only on the final output are no longer enough for systems that make a sequence of decisions.

A core challenge is that an agent can produce a correct output through an inefficient or incorrect process—what we call a "silent failure". For instance, an agent tasked with reporting inventory might give the correct number but reference last year's report by mistake. The result looks right, but the execution failed. When an agent fails, a simple "wrong" or "right" doesn't provide the diagnostic information you need to determine where the system broke down.

To debug effectively and ensure quality, you must understand multiple aspects of the agent's actions:

The trajectory—the sequence of reasoning and tool calls that led to the result.
The overall agentic interaction - the full conversation between the user and the agent (Assuming a chat agent)
Whether the agent was manipulated into its actions.

This article outlines a structured framework to help you build a robust, tailored agent evaluation strategy so you can trust that your agent can move from a proof-of-concept (POC) to production.

Start with success: Define your agent’s purpose

An effective evaluation strategy is built on a foundation of clear, unambiguous success criteria. You need to start by asking one critical question: What is the definition of success for this specific agent? These success statements must be specific enough to lead directly to measurable metrics.

Vague goal (not useful)	Clear success statement (measurable)
"The agent should be helpful."	RAG agent: Success is providing a factually correct, concise summary that is fully grounded in known documents.
"The agent should successfully book a trip."	Booking agent: Success is correctly booking a multi-leg flight that meets all user constraints (time, cost, airline) with no errors.

By defining success first, you establish a clear benchmark for your agent to meet.

A purpose-driven evaluation framework

A robust evaluation should have success criteria and associated testable metrics that cover three pillars.

Pillar 1: Agent success and quality

This assesses the complete agent interaction, focusing on the final output and user experience. Think of this like an integration test where the agent is tested exactly as it would be used in production.

What it measures: The end result.
Example metrics: Interaction correctness, task completion rate, conversation groundedness, conversation coherence, and conversation relevance.

Pillar 2: Analysis of process and trajectory

This focuses on the agent's internal decision-making process. This is critical for agents that perform complex, dynamic reasoning. Think of this like a series of unit tests for each decision path of your agent.

What it measures: The agent's reasoning process and tool usage.
Key metrics: Tool selection accuracy, reasoning logic, and efficiency.

Pillar 3: Trust and safety assessment

This evaluates the agent's reliability and resilience under non-ideal conditions. This is to prevent adversarial interactions with your agents. The reality is that when your agents are in production, they may be tested in unexpected ways, so it's important to build trust that your agent can handle these situations.

What it measures: Reliability under adverse conditions.
Key metrics: Robustness (error handling), security (resistance to prompt injection), and fairness (mitigation of bias).

Define your tests: Methods for evaluation

With a framework in place, you can define specific tests that should be clearly determined by the metrics you chose. We recommend a multi-layered approach:

Human evaluation

Human evaluation is essential to ground your entire evaluation suite in real-world performance and domain expertise. This process establishes ground truth by identifying the specific failure modes the product is actually exhibiting and where it’s not able to meet your success criteria.

LLM-as-a-judge

Once human experts identify and document specific failure modes, you can build scalable, automated tests using an LLM to score agent performance. LLM-as-a-judge processes are used for complex, subjective failures and activities and can be used as rapid, repeatable tests to determine agent improvement. Before deployment, you should align the LLM judge to the human evaluation by comparing the judge's output against the original manual human output, groundtruthing the results.

Code-based evaluations

These are the most inexpensive and deterministic tests, often identified in Pillar 2 by observing the agent trajectories. They are ideal for failure modes that can be checked with simple Python functions or logic, such as ensuring the output is JSON or meets specific length requirements.

Method	Primary Goal	Evaluation Target	Scalability and Speed
Human evaluation	Establish "ground truth" for subjective quality and nuance.	Pillar 1 (UX, style, safety) AND Pillar 2 (ethical/costly tool use).	Low and slow; expensive and time-consuming.
LLM-as-a-judge	Approximate human judgment for subjective qualities at scale.	Pillar 1 (coherence, helpfulness) AND Pillar 2 (quality of internal reasoning).	Medium-High and fast; requires careful prompt engineering.
Programmatic evaluations	Measure objective correctness against a known reference.	Pillar 1 (factual accuracy, RAG grounding) AND Pillar 2 (tool call accuracy).	High and fast; ideal for automated regression testing.
Adversarial testing	Test agent robustness and safety against unexpected/malicious inputs.	The agent's failure mode (whether the agent fails safely or produces a harmful output).	Medium; requires creative generation of test cases.

Generate high-quality evaluation data

A robust framework is only as good as the data it runs on. Manually writing thousands of test cases creates a bottleneck. The most robust test suites blend multiple techniques to generate diverse, relevant, and realistic data at scale.

Synthesize conversations with "dueling LLMs": You can use a second LLM to role-play as a user, generating diverse, multi-turn conversational data to test your agent at scale. This is great for creating a dataset to be used for Pillar 1 assessments.
Use and anonymize production data: Use anonymized, real-world user interactions to create a "golden dataset" that captures actual use patterns and edge cases.
Human-in-the-loop curation: Developers can save valuable interactive sessions from logs or traces as permanent test cases, continuously enriching the test suite with meaningful examples.

Do I need a golden dataset?

You always need evaluation data, such as logs or traces, to run any evaluation. However, you don't always need a pre-labeled golden dataset to start. While a golden dataset—which provides perfect, known-good outputs—is crucial for advanced validation (like understanding how an agent reaches a known answer in RAG or detecting regressions), it shouldn't be a blocker.

How to start without one

It's possible to get started with just human evaluation and vibes-based evaluation metrics to determine initial quality. These initial, subjective metrics and feedback can then be adapted into LLM-as-a-Judge scoring for example:

Aggregate and convert early human feedback into a set of binary scores (Pass/Fail) for key dimensions like correctness, conciseness, or safety tested by LLM-as-a-Judge. The LLM-as-a-Judge then automatically scores the agent interaction against these binary metrics to determine overall success or failure. The agent's overall quality can then be aggregated and scored with a categorical letter grading system for example ‘A’ - All binary tests pass, ‘B’ - ⅔ of binary tests pass, ‘C’ - ⅓ of binary tests pass etc.

This approach lets you establish a structured quality gate immediately while you continuously build your golden dataset by curating real-world failures and successes.

Operationalize the process

A one-time evaluation is just a snapshot. To drive continuous improvement, you must integrate the evaluation framework into the engineering lifecycle. Operationalizing evaluation changes it into an automated, continuous process.

Integrate evaluation into CI/CD

Automation is the core of operationalization. Your evaluation suite should act as a quality gate that runs automatically with every proposed change to the agent.

Process: The pipeline executes the new agent version against your reference dataset, computes key metrics, and compares the scores against predefined thresholds.
Outcome: If performance scores fall below the threshold, the build fails, which prevents quality regressions from reaching production.

Monitor performance in production

The real world is the ultimate test. You should monitor for:

Operational metrics: Tool call error rates, API latencies, and token consumption per interaction.
Quality and engagement metrics: User feedback (e.g., thumbs up/down), conversation length, and task completion rates.
Drift detection: Monitor for significant changes in the types of user queries or a gradual decrease in performance over time.

Create a virtuous feedback loop

The final step is to feed production data back into your evaluation assets. This makes your evaluation suite a living entity that learns from real-world use.

Review: Periodically review production monitoring data and conversation logs.
Identify: Isolate new or interesting interactions (especially failures or novel requests) that aren't in your current dataset.
Curate and add: Anonymize these selected interactions, annotate them with the "golden" expected outcome, and add them to your reference dataset.

This continuous cycle ensures your agent becomes more effective and reliable with every update. You can track and visualize the results from these cycles by exporting the runs of these tests and leveraging dashboarding tools to see how the quality of your agent is evolving over time.