AI Agents for Developers / How to Evaluate AI Agents

AI agent evaluation is the process of measuring whether an agent reaches the right outcome, reaches it by a sound process, and does so repeatably. It blends automated metrics with human judgment, because agents rarely have one correct output, and the steps that produce an answer matter as much as the answer.

Those steps are the part you never see. You hand an agent a task, and it comes back green. Whether it reasoned its way there or stumbled in after three wasted tool calls is not in the result, and the result is most of what you have. That is what makes evaluating AI agents harder than evaluating one model response. A multistep agent plans, calls tools, retrieves context, and retries failed actions. Each is a place where things can go wrong quietly.

No single number covers that. Useful evaluation separates task success, output quality, tool-use behavior, reliability, and human usefulness. This article covers those metrics and the practices around them, not pre-release testing or production monitoring: what makes evaluation difficult, which metrics answer which question, workflow quality, human review, common challenges, and best practices. The payoff is knowing which number to trust, and when.

What makes AI agent evaluation difficult

Core AI agent evaluation metrics

Evaluating agent workflow quality

Human evaluation for AI agents

Common AI agent evaluation challenges

Best practices for evaluating AI agents

Evaluation as an ongoing quality process

Quiet failure is only half the problem. The other half is that an agent's behavior is not fixed. It varies across runs, tools, context, models, and workflow states, and each of those is a separate reason the number you collected last week may not describe the agent you have today.

Run the same task twice, and you may get two different plans, two different tool selections, and two different final answers, all of them acceptable. Exact-match scoring breaks on the first run, and so does any regression check that diffs today's output against a stored answer.

The variability across runs has two sources. Model generation can vary between runs, depending on decoding settings and runtime behavior. Retrieved context, tool responses, and environment state can also change, even when the task itself stays the same.

Judge the run against criteria that tolerate that variation. Did the agent achieve the intended outcome, did it respect the task's constraints, and did it stay inside acceptable behavior? For an agent searching a codebase, two different retrieval paths are both fine if they surface the code that answers the question. You are testing whether it accomplished what it was asked to accomplish, not whether it produced the string you expected.

An agent that gets the final answer right might have gotten there poorly, skipping a validation step, calling a tool unnecessarily, or guessing at a missing input instead of asking for clarification. Evaluation has to reach the intermediate steps, where most of the failures live.

That means measuring planning quality, retrieval decisions, tool selection, validation behavior, and retry logic alongside the final output. Process evaluation treats those steps as first-class signals rather than diagnostic detail. Correct results reached through chaotic intermediate steps still point to an unreliable agent. Process evaluation can reveal weaknesses that task success rates alone may miss, including skipped validation, unnecessary tool calls, and ineffective retries.

Most agents depend on tools, APIs, databases, or external services. A poor result might come from the agent choosing the wrong tool, passing invalid inputs, misinterpreting a tool's output, hitting a service that returned unexpected data, or a workflow design that never offered the right tool at all. Attribution is genuinely hard.

Agentic development workflows make this concrete. JetBrains Junie, for example, works across JetBrains IDEs, Junie CLI, and compatible ACP clients. It can navigate a codebase, edit files, run tests, and adjust its approach based on the results. When a test fails after Junie applies a patch, evaluation should distinguish between an incorrect code change, misinterpretation of test output, and ineffective recovery. Each points to a different problem to investigate.

Before you pick a metric, define what success looks like for your specific agent. That definition shapes everything downstream.

A support agent succeeds when it resolves issues without escalation. A coding agent succeeds when it generates accepted code changes or completes development tasks end to end. A research agent succeeds when it returns accurate, relevant information that the user would not easily find on their own. Evaluation approaches vary by agent type, and so do the metrics worth collecting.

Establish success criteria before you collect data or compare agent versions. Without them, you will improve a number and learn nothing from it.

AI agent evaluation metrics come in families, and no single family captures agent quality on its own. This section covers the four you can measure from the run itself: task completion rate, reliability, tool-use accuracy, and output quality. Workflow quality and human usefulness matter just as much and get their own sections below.

Collect task completion rate first, since it tells you whether the agent is doing the job at all. Reliability comes next, because a rate that only holds on every third run is a different problem, and only repeated runs expose it. Tool-use accuracy and the workflow signals explain why that rate moved, which is what you need before changing a prompt or swapping a model.

All of them need an evaluation set to run against, built from representative tasks with acceptance criteria rather than single reference answers. The best practices below cover how to assemble and maintain one.

Task completion rate measures the share of tasks the agent completes according to your predefined success criteria. Completion means it resolved the issue, answered the question correctly, generated valid code, or finished the workflow without escalation. Producing an output is not the same as completing a task.

For example, if an agent successfully completes 60 out of 100 evaluated tasks, its task completion rate is 60%. Define completion using task-specific acceptance criteria, not simply whether the agent produced a response.

n agent is reliable when it behaves consistently across similar tasks, users, environments, and repeated runs. Identical wording isn't required, but predictable task behavior is.

Run each representative task multiple times under comparable conditions. Record the success rate for each task and how much it varies between runs, rather than relying only on the aggregate completion rate. A task that clears nearly all of them looks stable; one that clears about half needs investigation, even if some runs looked fine in isolation. Treating consistency as a testable property turns a vague worry into a number you can track over versions.

Watch for variance in outputs across similar prompts, repeated failures on specific task types, unstable tool selection, and inconsistent escalation decisions.

The question for tool use is narrower. Did the agent pick the right tool, pass it valid inputs, read the result correctly, and skip the calls it did not need? You can usually answer it from the agent's action log, a structured trace of each tool call with its inputs, outputs, and timestamps. For agents that write files, query databases, call external APIs, manage tickets, or work inside development environments, invalid API arguments, unnecessary tool calls, or incorrect interpretation of tool results can indicate tool-use problems even when the final output is acceptable.

The final answer, summary, code change, recommendation, or action has to be accurate, useful, and aligned with the task. Criteria include factual accuracy, completeness, relevance, clarity, groundedness in the source material, and format compliance.

For a code agent, quality covers functional correctness, readability, adherence to project standards, security, and performance. For a knowledge agent, it shifts toward accuracy, source use, and completeness.

Reviewing every output by hand does not scale. Teams automate the first pass with an approach often called LLM-as-judge. An LLM-based judge can score outputs against a defined rubric, using the task, expected criteria, and available evidence. Calibrate its scores against human-reviewed examples and recheck that calibration when the judge model or rubric changes. For coding tasks, combine judge scores with executable tests and targeted human review rather than relying on the judge alone.

How the agent got to its output matters as much as what it produced, and a chaotic path usually costs more to operate even when it reaches the right answer.

The three signals below are diagnostic. They rarely tell you whether to ship, and they usually tell you what to change. The deeper taxonomy of failures for agentic workflows, covering context overflow, goal drift, and cascading errors, is explored in Agentic Workflows Explained. The concern here is narrower: what to measure once a workflow is already running.

For complex tasks, an agent should gather enough context to identify the relevant files, dependencies, and constraints before making changes. Evaluation should check whether its approach fits the task, rather than requiring an explicit plan for every edit. A test-triage agent that reruns a failing test before reading the failure output makes the same category of mistake. It might get lucky this time, but the process won't generalize to the next failure.

Planning quality is whether the agent breaks a task into useful, bounded, and relevant steps before acting. Weak planning shows up as unnecessary steps, skipped prerequisites, vague intermediate goals, or overly broad opening actions that need correction later.

Intermediate decisions carry their own weight: when to retrieve information versus rely on prior context, when to call a tool versus infer an answer, when to ask for clarification versus proceed on assumptions, and when to stop or escalate.

In a coding workflow, an agent may edit files, run tests, inspect the results, and decide whether to revise the change or return it for review. Junie provides one example of this process, and evaluation can examine each action separately, including whether it was necessary and how the agent used its results. An agent that repeats the same failed action without changing its approach is making a decision error, not only a tool error.

Recovery is what happens after something breaks mid-task, whether a tool returns an error, context goes missing, a previous step emits an invalid output, a proposed action is rejected, or the user's request turns out to be ambiguous. Good recovery means recognizing the failure, adjusting the strategy, and escalating when the situation is beyond what the agent can resolve. The pattern to watch for is an agent that retries the same failed action on a loop. That signals shallow error handling, with no mechanism for noticing that the current approach has stopped working.

Some agent outcomes are too contextual, too subjective, or too high-stakes to assess with automated metrics alone. Human review fills that gap, and for those outcomes, it is often the signal that teams weight most heavily.

Subject-matter experts can judge whether the agent's output is correct, useful, safe, and aligned with team expectations in ways automated metrics tend to miss.

Match the reviewer to the domain. Developers handle code agents, judging a change on the axes they would apply in any code review. Support leads and documentation owners do the same work for triage agents and knowledge agents. Reviewers should have access to the agent's intermediate actions when process quality, safety, or recovery behavior is part of the evaluation.

Feedback from real users surfaces problems that controlled evaluation misses. Users show you where the agent helps and where it frustrates, sometimes in the same session.

Useful signals include accepted suggestions, repeated corrections, user ratings, follow-up questions that indicate an incomplete answer, task reopen rates, and manual overrides. At the organizational level, JetBrains Air Governance is designed to provide visibility into AI usage, costs, and their impact on software delivery across JetBrains and third-party agents. These signals can complement agent evaluation, but they should not be treated as direct measures of output quality or task success.

Rubrics make human evaluation consistent. Without them, two reviewers apply different standards to the same output and produce results you can't compare, which makes improvement hard to track over time.

A useful rubric for agent review covers correctness, completeness, relevance, safety, source use, action quality, and usefulness. Score each dimension separately so a drop in one stays visible. Anchor the scale to concrete descriptions. For correctness, 1 means factually wrong or task-breaking, 3 means correct but needing manual cleanup, and 5 means correct and ready to use as is. Clear scoring anchors help reduce disagreement between reviewers and make differences easier to investigate.

Even teams with well-defined metrics run into AI agent evaluation challenges that take time to work through. Most come down to the difficulty of defining what "good" means when success depends on both final output and process quality, and those two dimensions don't always agree.

Plenty of tasks land between clean success and outright failure. The agent finds the right information but writes a weak summary, generates a correct patch but misses test coverage, or hits the primary objective while failing an edge case. Binary pass/fail scoring misses these distinctions entirely.

Score partial success separately. Use task-specific milestones or rubric scores to distinguish partially successful runs from complete failures. For example, a coding agent may implement the requested behavior but fail to add the required tests. Track these outcomes separately rather than treating every incomplete task as an identical failure.

Comparing two versions of the same agent is difficult when outputs vary, but they may both be acceptable. Diffing the outputs surfaces stylistic differences. The substantive ones stay buried.

Compare against stable evaluation dimensions instead: task outcome, constraint adherence, tool-use quality, cost per task, and human usefulness. Run both versions against the same task set and score each on a rubric. Use enough tasks per category, or enough repeated runs per task, that a few percentage points of difference means something. A handful of tasks can't separate a real improvement from ordinary run-to-run variance.

Model and workflow changes can affect quality, cost, and latency in different ways. A larger model or additional validation step may improve results, but the benefit should be measured against its computational and operational cost. Evaluation needs to account for that trade-off explicitly.

If adding a retrieval step increases output quality but can double token usage and latency, whether that's worth it depends on your use case. Track quality, cost, and latency together as linked dimensions. The choice that maximizes quality on your evaluation benchmark may be the wrong choice for your actual deployment context.

These practices won't fix a broken agent, but they will make your evaluation trustworthy enough to act on.

• Define success criteria before selecting metrics or comparing agent versions.
• Build an evaluation set of representative tasks with acceptance criteria, covering high-risk and ambiguous scenarios as well as the cases the agent already handles, and keep it current.
• Evaluate intermediate actions alongside the final output.
• Separate task completion, output quality, tool-use quality, and reliability into distinct metrics.
• Use rubrics with anchored scale descriptions for human review.
• Compare agent versions on the same task set.
• Review failed and partially successful tasks, not only the successes.
• Measure cost and latency alongside quality.
• Rerun the evaluation set after any prompt, tool, or model change.
• Track whether users accept, edit, or reject agent outputs.

Evaluation should be repeated as agents, tools, and tasks change. Track both outcomes and intermediate behavior to identify regressions and decide what to improve.

Revisit the metrics when the agent's tasks change. An evaluation set built for last quarter's workload will keep reporting a pass on work the agent no longer does.