How to Evaluate AI Agents in Production, Before Green Dashboards Lie
A green uptime dashboard can sit above an agent that is confidently wrong. This essay looks at how production agents get evaluated, why their failures stay quiet, and where an evaluation gate belongs in the release process.
In AWS's account of its own agent build, a Customer Insights Agent reported a customer's revenue over three months as “$410.08 Million”, labelled it as sales revenue, produced an ARR estimate that did not appear in the data, and presented its own arithmetic as sourced fact. Monitoring stayed green throughout, because it confirmed that a response had been given, not that the response was right. The question of how to evaluate AI agents in production starts in that gap. In a sense the dashboard was not lying, since it answered a narrower question than the one the business was asking.
Uptime is a good answer to a narrow question. It asks whether the service answered, how long that took and whether it failed loudly. Infrastructure teams learned those questions long ago, and they still matter. An agent adds a second question that the old instruments cannot see, which is whether the answer was true. AWS describes the pattern as soft failure, in which latency stays flat while answer quality erodes, and a single degraded subagent can corrupt every downstream response that depends on it.
I would put silent failure near the top of the list of properties that make agent systems different from other software. I suspect it is underweighted because it does not show up in demonstrations. A demonstration runs once, on a path someone chose in advance. As I read it, the failures that matter tend to appear on the fourth phrasing of a request nobody thought to write down, at an hour when nobody is reading the traces.
When the Dashboard Stays Green
The trouble with a green light is that it is usually true about something. The service did respond. The model did return output in the expected format. None of that says the sentence in the answer was supported by the data that was retrieved. A tool call that quietly returned stale records looks identical to one that returned fresh ones, at the level of a standard uptime check.
The failures I find most interesting are the ones where the system is fluent. A wrong answer delivered with confidence and a neat table is harder to catch than an error page. My suspicion, which I cannot test from where I sit, is that such answers travel further than they deserve, because tidy output tends to be trusted. The Customer Insights case is useful precisely because the wrong figure arrived dressed as analysis.
Why One Passing Run Proves Very Little
Nondeterminism is the first reason single tests mislead. An eval, the shorthand for an evaluation, is a scored test of behaviour rather than a unit check, and the same query can produce different tool selections and reasoning paths. A scenario that passes once therefore says something about one sample rather than typical behaviour. AWS's guidance on its AgentCore Evaluations feature argues for repeated runs per scenario for exactly that reason.
A preprint on arXiv pushes the argument further. Its author, a single independent researcher, proposes seven production failure modes and claims that standard metrics fail to detect four of them entirely, while catching three only after a lag of several evaluation cycles. Two of the modes are cascading errors that look coherent at each step, and tools that quietly fall back to stale data instead of failing. The paper is not peer reviewed, and its production scale claims have not been independently verified, so I treat it as a hypothesis worth testing.
If standard metrics miss whole categories of failure, a passing eval suite tells a team about the cases it covers and little more. Intuition still has a place in my view, since it is often how someone first notices that something feels off, before any metric agrees.
How to Evaluate AI Agents in Production Before Customers See Them
I suspect many teams still picture a release as a code change. An agent has other doors. The underlying model is swapped for a cheaper or newer one, an upstream data source is refreshed, a tool schema changes, or a prompt is tuned by someone in another team. Each can alter behaviour without a single commit in the agent repository, which makes the release boundary harder to draw.
The AWS preproduction test is the number I would put on a slide if I had to choose one. When a data change, a tool change and a prompt update landed apart rather than together, the pass rate fell from 78% to 38%. That figure comes from AWS's own team and describes one system, so I would not generalise it. It still points at something important about the unit of release. The thing that changed is not always the thing that broke, and I would put the gate in the gap between the two.
Where the gate sits is largely a question of ordering. AWS describes CI/CD regression gates using sentinel sets, canary monitoring on evaluation windows lasting 30 minutes, guardrails on all traffic, and online evaluation with A/B rollout. The same post reports cutting mean time to detect from days to under four hours, a figure AWS presents as its own measurement, which I read as a claim rather than a benchmark. As I read the design, the gate that runs before release sits ahead of customer traffic, and the live monitor takes over once the change is out, judged against thresholds set earlier.
Regression testing for agents is the same idea applied to change, and the analogy holds up to a point. A fixed set of sentinel scenarios is rerun whenever something upstream moves, so a cheaper model is judged against the cases the previous one already passed. The analogy strains because agents also drift with no change to test against: the same query can take a different path on a different run, so a clean rerun is a sample rather than a proof.
Defining the threshold before deployment is the part I find hardest to make concrete. AWS's approach translates business outcomes into measurable criteria up front, which sounds obvious until one tries to write the criterion for a reply about a delayed order. In my view a useful AI agent evaluation framework settles what counts as wrong before the first run, not after. Who decides that, and how many wrong answers across repeated runs are tolerable?
Scopes, Metrics and the Tools That Score Them
Arize describes four scopes for judging an agent. Outcome asks whether the final state was reached, path asks whether the process was acceptable and policy compliant, decision asks whether each action was justified by evidence and permissions, and reliability asks whether good behaviour holds across repeated runs. Each one catches a different failure. A correct outcome reached through a forbidden path is still a release problem.
DeepEval splits the testing itself into three levels: end to end tests that treat the agent as a black box, trajectory tests that inspect the full execution trace, and component level tests that look at one step such as tool selection. The distinction matters for diagnosis. An end to end failure says the answer was wrong, while a component failure points to where to look. DeepEval also recommends pulling failing production traces into the dataset, so the next run tests the path that broke. My reading is that much LLM evaluation production practice was built around a single prompt and a single reply, and an agent breaks that assumption at every tool call.
Agent observability and evaluation are easy to confuse, and the confusion has consequences. Observability shows what the agent did, step by step. Evaluation judges whether the outcome was correct, grounded and safe, as one recent playbook puts it. A trace can show that an agent called the order system and then wrote a fluent sentence. It cannot show that the sentence was wrong. Both belong in production, but a trace view on its own does not act as a gate.
What Salesforce Agent Evals Gets Right and What Is Still Missing
Salesforce Agent Evals was announced on 27 October, alongside API Evals and advanced cost controls for Agent Fabric, Salesforce's control plane for managing agents across platforms. Agent Evals is designed to generate tests before deployment and to score agents, LLMs and MCP servers in production against quality thresholds. When results fall short, it triggers predefined responses, including the Agent Kill Switch, with human approval where policy requires it (Salesforce newsroom).
I think the placement is the right instinct. Putting evaluation inside the control plane, beside the kill switch and the spending limits, treats an eval result as something that can change what happens to an agent. A chart someone reviews on Monday cannot do that. The release's illustration is framed as hypothetical rather than a named customer result. A retailer moves its order support agent to a cheaper model, scheduled evaluations catch inaccurate delayed order answers, the evaluation requests approval to quarantine the agent, and is rerun before the agent returns to service.
The gaps are visible in the timeline. Agent Evals and API Evals enter beta in November 2026 and reach general availability in January 2027. WebProNews reports that the manual Agent Kill Switch is generally available now, while the triggered version, which ties shutdowns to evaluation results, is scheduled for January 2027. The step that would close the loop from a failed evaluation to an automatic action is therefore still ahead. Superpower Daily reads the release as asking buyers to base purchase decisions on released features, since automatic shutdown after a failed evaluation is still on the roadmap.
A manual switch waits for someone to notice a problem. A triggered one waits for the evaluation to be right, and that dependence is what would turn an evaluation from a report into a gate. Whether automated triggers would fire in error at scale is not yet known. A false positive that quarantines a working agent is a minor event in a demo and an expensive one during a busy quarter. Salesforce has no independent measurement of Agent Evals yet, so its framing is best read as the start of a conversation. I have written about the governance side of this separately in Agentic AI Governance: From Policies on Slides to Code in Production.
Reading the Survey Without Over Reading It
The LangChain State of Agent Engineering survey offers the wider picture. Its fieldwork ran from 18 November to 2 December 2025 and drew 1,340 responses (LangChain). In the survey, 57.3% of respondents have agents in production, up from 51% a year earlier, and quality is the top barrier to production, cited by 32% (AMLA Labs). Observability is in place for 89%, but only 52.4% run offline evaluations and 37.3% run online ones (Unite.ai).
Of the survey's numbers, the gap between watching and testing is the one I would single out. KDnuggets reads the same data as a “ship and watch” habit, in which teams debug after errors rather than preventing them before deployment (KDnuggets). The phrase is blunt, and I think it fits. My guess is that dashboards win adoption because they feel like progress. That is a guess about motive, and the survey does not test it.
The survey needs its caveats. Technology firms made up 63% of respondents, so it is not a cross section of enterprise buyers at large. Slash Digital calls it an upper bound, because its respondents are engaged enough to answer a survey about agents, and it notes that some secondary sites list a June 2026 publication date that conflicts with the late 2025 fieldwork (Slash Digital). I would treat the direction as more reliable than any single percentage.
Harness's report, published on 10 September 2026 and based on 700 engineering leaders, makes a related point from another angle. It finds confidence in agents running ahead of the testing, security and governance controls in place (PR Newswire). Harness sells in this market, so the finding has a commercial interest behind it. Its Keith Mann argues that controls built for deterministic software miss agent variability, and that a control which passed in testing can still miss things in production.
On the day an evaluation turns red, who is allowed to stop the release? The threshold is easy to write down in a meeting. The harder part is agreeing, before anything is red, how many wrong answers across repeated runs the business can live with.