Evals for AI agents: knowing it works before a customer tells you
What an evaluation set for an AI agent is, how to build one from real cases in your operation, and how to use it to catch a quality drop before a customer complains.
Updated August 5, 2026
Short answer
Automating a narrow process with AI starts at USD 3,000–8,000, and an agent integrated with several systems lands between USD 12,000 and 40,000. On top of that, expect USD 150 to 1,200 per month to operate it: models, infrastructure and maintenance. Integrations drive the price, not the model.
Nearly every answer to this question is useless for the same reason: it gives you a range from USD 500 to USD 80,000 and says “it depends.” It does depend. But it depends on specific, enumerable things, and once you know them you can estimate your own case fairly closely before requesting a single quote.
This is the breakdown we use internally to price work. It isn’t a price list — it’s a map of what moves the number.
In practice, projects fall into three fairly distinct buckets, and the jump between them is not gradual.
| Type | Upfront | Monthly | What it is |
|---|---|---|---|
| Scoped workflow | USD 3,000 – 8,000 | USD 150 – 400 | One process, clear rules |
| Integrated agent | USD 12,000 – 40,000 | USD 400 – 1,200 | Decides, asks approval |
| Multi-process system | USD 40,000+ | USD 1,200+ | Coordinated agents |
The first bucket is an invoice reader that posts to the ERP and flags anything that doesn’t reconcile. The second is an agent that takes orders over WhatsApp, checks stock, builds the order and escalates to a rep when a discount falls outside policy. The third is a platform, and it usually shows up after the second one worked.
If a vendor quotes USD 1,500 for something in the second bucket, they are not quoting what you asked for.
This is the line item that eats half the budget and the one nobody raises in the first meeting. Connecting to HubSpot is half a day. Connecting to an undocumented on-premise ERP reached through a SQL Server database somebody configured in 2014 is two to three weeks of work before a single line of agent logic gets written.
The question that moves your budget most isn’t “which model will you use?” It’s: do your systems have a documented API and a sandbox? If the answer is yes to both, the project is 30–50% cheaper.
A flow that moves data from A to B without deciding anything is cheap and behaves the same every day. An agent that has to interpret an ambiguous message, choose among five possible actions and know when to do nothing is different work: you have to define the boundaries, build the evaluation set, measure, and correct. See why AI agents fail in production for what breaks when that work is skipped.
An agent that drafts replies for a person to review has a near-zero cost of error and can ship fast. One that issues credit notes on its own needs guardrails, an audit trail of every action, amount limits, reversibility and alerting. That’s 20–40% additional hours on the same project.
It isn’t negotiable, but it is a scoping decision: in the first version it is almost always right to leave the irreversible action with a person and automate it later, once you have three months of data showing how often it’s wrong.
Intuition says ten times the volume costs ten times as much. In these systems it rarely does: the difference between processing 500 and 5,000 documents a month is a few dollars of model spend and, eventually, a job queue. Volume starts to matter seriously only above tens of thousands of operations per month.
Build cost is what gets discussed. Running cost is what surprises people.
An agent in production needs, month after month:
That last point separates a project still running two years later from one that quietly went dark in month four. An agent nobody watches doesn’t fail with an error — it degrades, answering slightly worse each week, until somebody in support mentions in passing that “the bot has been saying weird things lately.”
Do this math before requesting quotes. If it doesn’t work out, no vendor is going to fix that.
A realistic shape: 22 hours a week of order entry, USD 12 loaded hourly cost, 70% automatable. That’s roughly USD 8,000 a year saved against a USD 6,500 build and USD 250 monthly. It pays for itself in year one, and from year two the process is essentially free.
After reading a fair number of proposals — ours and competitors’ — these are the ones that most often predicted a project would go badly:
The honest range for a real business process is USD 3,000 to 40,000, and within that range your case is defined almost entirely by the state of your systems, not by how sophisticated the AI is. Before comparing proposals, go find out whether your ERP has an API and a sandbox: that single answer moves the number more than any technical decision you make afterwards.
And if the ten-hour calculation gives you a payback beyond 18 months, the correct conclusion isn’t “AI is expensive.” It’s that this is the wrong process to start with.
Written by
The Vantalogics team
An AI systems agency. We automate processes and build evaluated, monitored agents for companies across Latin America, Spain, the United States and the Gulf.
What an evaluation set for an AI agent is, how to build one from real cases in your operation, and how to use it to catch a quality drop before a customer complains.
The six failures that show up once an AI agent leaves the demo and starts handling real customers, why none of them are visible in testing, and how each one is prevented.