From Tests to Feedback Loops: QA in the AI Era
AI-focused* software development has changed how I think about quality, but many of the fundamental problems remain unchanged. I’ve had the opportunity to work on a heavily AI-focused project for almost a year now and have found a new perspective on it. These things are definitely not new, but just how I see things evolving currently.
The most visible change is of course speed. AI can dramatically reduce the time from an idea to implementation. But faster implementation does not necessarily mean faster delivery.
Let's look at a development ticket with an unclear acceptance criteria as an example. An agent can turn it into a working implementation very quickly, but the questions that were missing from the ticket are still missing. What should happen in an edge case? What does a good result actually look like? What shouldn't the feature do? Instead of discovering these questions while implementing the feature, they might now appear during review, testing or even in production.
This puts more emphasis on making requirements testable and thinking about evaluation before implementation starts. The faster implementation gets, the less useful it is to discover afterwards that everyone had a different understanding of what was supposed to be built.
* Here AI-focused means not only a product which heavily relies on AI, but where also the whole development process is supported by AI tools.
Separate testing from evaluation
One distinction that has been useful for me is separating scripted regression testing from evaluation of AI behavior and implementing them both where it makes sense. Much of an AI product should still behave deterministically, even if the generated answer itself does not.
Did the integration work? Was the correct action triggered? Did the application state change as expected? Can the user complete the flow?
Tools such as test automation frameworks (for example Playwright for web-based flows) are useful here. The focus should be on stable contracts and observable behavior rather than exact AI-generated outputs or incidental data.
For example when thinking of a chatbot, an integration test might verify that a certain type of question triggers the expected tool call and that the result returned to the user is related to the correct topic. The test does not need to understand everything about the question or validate the exact answer. It only needs to verify enough of the stable behavior to give confidence that the integration has not regressed.
The AI output itself is a different problem. Two answers can be equally correct without being identical. This makes exact-output assertions fragile, but it does not mean AI behavior should go untested.
This is where evals come in.
Evaluation datasets and benchmarks can measure things such as correctness, groundedness, tool selection and usefulness across a broader set of cases. Some of these can be evaluated deterministically, while others can use an LLM-as-a-judge to score the response against defined criteria. This makes it possible to evaluate behavior at a scale that would be difficult to review manually or with test automation.
These results can provide directional signals or even act as release gates when suitable thresholds can be defined.
The difficult part is keeping both the evaluation cases and the evaluation criteria useful. Products evolve, requirements change and new failure modes appear. And an LLM judge is only useful if what it is being asked to judge actually represents what “good” means for the product. Both of these take human effort even with the help of an AI.
AI can make some of that maintenance easier. Production traces can be analyzed for interesting failures, variations of existing cases can be generated and potential new evaluation cases can be suggested automatically. The important part is having a process for deciding which cases actually represent something the product should be able to do.
Exploration still matters
Automated tests and evals are good at checking behaviors that have already been identified as important. But someone still needs to find the behaviors nobody thought to put into either of them. This is where I think exploratory testing remains just as important as before, if not more so.
With AI products, exploration can mean trying different ways of expressing the same intent, following unusual conversation paths, combining actions in unexpected ways or simply asking "what happens if...?".
The goal is not to turn every finding into another automated test. Some findings might expose an unclear requirement. Others might reveal a missing benchmark case or something worth monitoring in production.
In that sense, exploratory testing is part of discovering what should be tested or evaluated in the first place. And even with good exploration, there will still be things that only become visible once real users start using the product.
Production is part of the quality loop
We have never been able to test every possible user behavior before releasing software. With generative AI, this becomes even more obvious. Users can ask unexpected questions, generated outputs vary, external data changes and integrations create many different paths through the system.
This makes production observability increasingly important.
Traces can show what happened: what context was retrieved, which tools were called and what response was generated. Combined with automated evaluation, user outcomes and human review, they provide evidence about how the system is actually behaving.
For QA, this means production is not only the place where incidents are monitored. It becomes another source of test cases. A bad answer, unexpected tool call or unusual user flow found from production can become an exploratory testing scenario, an automated regression test or a new benchmark case.
This creates a feedback loop:
01
Perhaps this is more useful than thinking only in terms of shift-left or shift-right. Quality increasingly comes from closing the loop.
The limiting factor is changing
AI can generate code, tests and evaluation cases. It can analyze traces and help maintain benchmarks.
The challenge is no longer only producing things. It is having ways to determine whether they are useful:
An AI agent can generate thirty tests quickly. Someone still needs to know whether they are the right thirty tests. An AI agent can implement a ticket quickly. Someone still needs to know whether the ticket described the right thing.
Implementation is becoming less of a bottleneck. Good requirements, representative examples, meaningful evaluation criteria and human judgment are not.
For QA, I think this shifts some of the work away from simply creating more tests. It becomes even more important to ask: What should be deterministic and protected by regression automation? What needs evaluation instead? What can only really be understood from production? And how do findings from each of those places get back into development?
In practical terms, that means keeping deterministic regression tests stable, maintaining representative evaluation sets, making production behavior visible and turning what is learned there back into requirements, tests and evals.
That feedback loop needs to become faster alongside development. Otherwise AI just gives us the ability to create changes faster than we can understand the effect on the quality of our product.