An AI agent at a mid-sized lender starts approving loan applications on its own. For six weeks, it works beautifully. Decisions get faster. Bottlenecks disappear. Customers are happy.
Then a data provider quietly changes a date format. DD/MM/YYYY becomes MM/DD/YYYY.
The agent does not crash. It does not throw an error. It does not page anyone at 3am. It simply starts reading a small group of applicants as ineligible. Then it declines them, politely and confidently, with a well-written explanation attached to every rejection.
Nobody notices for eleven days.
This is the failure that should worry engineering leaders. It is also the one almost nobody tests for. The industry has spent two years learning how to build agentic AI workflows. It has spent far less time learning how to prove those workflows still work on a Tuesday afternoon in month four.
Why agentic AI workflows break differently
Traditional software is predictable. Give it the same input and you get the same output, every time.
That single property is the foundation of how QA works. You write a test. You state the expected result. The test passes or it fails. Regression testing exists to catch the moment a change in one place quietly breaks something in another place. You re-run your checks and watch for red.
Agents do not give you that property.
Large language models produce answers based on probability, not fixed rules. If you have ever asked whether ChatGPT is an LLM and wondered what that means in practice, here is the answer. The same prompt can return a different response on Tuesday than it did on Monday. Both responses can be perfectly reasonable.
Now wrap that model in a system that can call tools, query databases, chain decisions together, and act in the real world. The room for quiet failure gets very large, very fast.
Three things go wrong that classical testing was never built to catch.
Drift. The model shifts underneath you. A provider bumps a version. A retrieval index goes stale. Nothing in your code changed, but your behaviour did.
Compounding error. In multi-agent systems, one agent’s slightly wrong output becomes the next agent’s input. A 5% error rate at step one is not a 5% error rate at step six. Teams that orchestrate multiple agents without a coordination layer learn this the expensive way.
Plausible wrongness. This is the big one. A normal bug announces itself with a stack trace. An agent failure gives you a fluent, confident, completely incorrect answer that looks exactly like a correct one. There is no exception to catch. A hallucination is not a crash. It is a well-formed lie.
Your current test suite will pass all three of these without blinking.
What is actually running in production right now
It would be easy to treat this as hypothetical if agents were still stuck in demos. They are not.
Over the last eighteen months, agents have moved into the workflows where mistakes cost the most. Implementation teams such as BuildNexTech have documented deployments where agentic AI cut manual underwriting effort at a US digital bank by 65%. Another auto-resolved 64% of IT tickets across a 400-location retail chain. A third cut insurance claims triage time by 67%. A fourth returned an estimated 1.2 million hours to a UK financial services firm through document and email automation.
Read that list again with a QA hat on.
Underwriting decisions. Claims triage. IT tickets that grant access. These are not chatbots suggesting blog titles. These are systems making real decisions about money and eligibility, at volume, with very little human review. And the light review is not an oversight. Removing the review was the business case.
The same pattern is showing up inside engineering itself. Agentic coding tools now open pull requests, refactor modules, and write their own tests. Code is entering repositories faster than humans can read it. That means a growing share of production code has never been reviewed closely by anyone.
The deployment curve is steep, and it points straight at high-stakes work. The assurance curve is flat.
The testing gap nobody costed into the business case
Ask a team how they validated their agent before launch. You will usually hear some version of this: we ran it a bunch of times and it seemed fine.
That is not QA. That is a demo with extra steps.
Part of the problem is definitional. Ask five engineers what QA is and you will get five answers, most of them about catching bugs before customers do. But an agent that declines eligible applicants does not have a bug in the usual sense. The code is correct. Every function returns what it should return. The behaviour is wrong.
Most QA engineer job descriptions have no concept of that difference. Neither do most test frameworks or CI pipelines.
Then there is the coverage question. What does 80% coverage mean for a system with a near-infinite input space and no fixed expected output? You cannot list every way a user might phrase a request. You cannot assert on an exact string when the model is free to word its answer differently each time.
There is also an independence problem the industry keeps stepping around. The team that built the agent is usually the team checking it. They know which prompts work. They wrote the happy path. So they end up testing for the behaviour they intended, not probing for the behaviour that actually shows up. Mature engineering teams settled this argument decades ago for every other kind of software.
What good AI QA actually looks like
The practices already exist. They are just not standard yet.
Test behaviour, not exact answers. Stop asserting on strings. Evaluate properties instead. Did the agent stay in scope? Did it cite a real source? Did it refuse when it should have refused? Did it return the right schema? You are grading against a rubric, not matching text. This shift sits at the heart of any working framework for testing non-deterministic AI features in production.
Build a golden dataset and run it continuously. Curate a set of representative cases and nasty edge cases, including the ugly ones from real traffic. Then re-run it constantly, not just before release. Drift does not wait for your sprint boundary. Evaluation belongs in continuous development, not in a launch gate.
Add observability built for probabilistic systems. Traditional monitoring and logging tells you the service replied in 200ms with a 200 status code. It tells you nothing about whether the reply was right. Data observability and AI tracing capture the input, the retrieved context, the tool calls, and the output. That record is what lets a human reconstruct why the agent decided what it decided. It is also what turns an eleven-day silent failure into a same-day alert. Practical guides to AI observability for QA teams go well past uptime dashboards.
Add guardrails and a way back. Assume the agent will be wrong eventually, then design for that day. Spend caps. Rate limits. Approval steps for anything you cannot undo. A tested kill switch with spending limits. None of this signals low confidence. It is the difference between an incident and a disaster.
Keep humans where they earn their cost. Not on every decision. That would defeat the point. Just on the decisions that are expensive to reverse.
None of this is exotic. It is the same instinct that gave us regression testing and staging environments, applied to a system that happens to run on probability. Independent QA partners like Frugal Testing build this kind of validation for AI and ML systems, largely because the in-house version gets deprioritised the moment a launch date wobbles.
Three questions worth asking this week
If you have an agent in production right now, you do not need a full programme to find out where you stand. You need three answers.
How would you know if it broke quietly? Not crashed. Broke. If the agent started giving wrong but reasonable answers this morning, what would tell you, and how long would it take? If the honest answer is “a customer would complain eventually,” you have found your first gap.
What happens when it is expensive wrong about something expensive? Trace one irreversible action end to end. A payment. A rejection. An access grant. Find out who or what can stop it, and how fast. If the answer involves someone noticing a dashboard, that is not a control.
Who checked it, and were they the same people who built it? There is nothing wrong with the build team running the first pass. There is something wrong with them running the only pass.
None of these take a quarter to answer. They take an afternoon. Most teams have never spent that afternoon.
The uncomfortable summary
The build side of AI has matured at remarkable speed. The tooling is good. The frameworks are good. The orchestration layers improve every month.
The assurance side has not kept up. Worse, the gap is widening exactly as the stakes rise.
We are handing probabilistic systems control over underwriting, claims, and access. Then we are checking them with a level of rigour we would consider negligent on a checkout page.
The companies that get burned will not be the ones that moved fast. They will be the ones that moved fast while assuming two very different claims were the same thing. “It worked in the demo” is not “it works in production, continuously, under drift, at volume.”
Those are not the same claim. Test accordingly.
