AI Testing: Rethinking the Testing Strategy
Five guiding principles from QA practice
AI agents are accelerating code development. How can testing keep up? Five guiding principles show how AI testing is changing testing strategies, how much the testing effort has decreased across three projects, and why final approval remains in human hands.
What Is AI Testing?
AI testing, also known as AI-based testing, refers to the use of artificial intelligence to support software testing. Depending on the tool and use case, AI derives test cases from requirements, creates test scripts, assists in fixing failed tests, and helps distinguish real errors from false positives.
AI testing at a glance:
Responsibility
People remain responsible for quality, even when AI tests are specified and automated.
Automation
In a Forrester report published in 2026, companies using autonomous testing platforms reported an average automation rate of between 51% and 60%. Forrester refers to approximately 25% as a historical industry plateau (Forrester, 2026).
Effort
In three Arvato Systems projects, the teams estimated that the effort required for test design and test automation decreased by approximately 50 to 90 percent, albeit under favorable conditions.
Obstacles
Implementation requires certain organizational and technical prerequisites: securing buy-in, clarifying roles, preparing projects, ensuring integration and reliability, managing costs, and complying with data protection regulations.
What Software Development Can Learn from Manufacturing
Many people are skeptical of a novel, a song, or an image generated by a language model. When it comes to smartphones or televisions, however, hardly anyone asks whether a human or a robot assembled the device. What matters there are price and functionality. Software falls into this second category: Who wrote or tested the code is of little consequence to customers. This raises the question of testing strategies involving AI anew.
The manufacturing industry has already answered this question. When robots took over assembly, the industry also industrialized quality assurance: with fixed tolerances, continuous process monitoring, and an inspection at the end of the assembly line. Trust in the product depends on this inspection chain. Software development now faces a comparable challenge.
Assessing Your Current Status: How Ready Is Your Testing Process for AI?
Many testing strategies date back to a time when people designed and executed test cases manually. They work, but they’re time-consuming and leave gaps. Use these five questions to find out if your strategy still follows this pattern:
- Are test results not available until long after development is complete?
- Do testers manually run regression tests before every release?
- Are entire functional areas left untested because there isn't enough time?
- Are licenses for testing tools expensive, while switching is considered too risky?
- Are there no clear rules regarding where and how AI may be used in testing?
If you answer “Yes” to several questions, you’ll find some starting points in the following guidelines. Our article “Software Testing Strategy: Quality, Tools & Methods.”
Why Is AI Testing Becoming Important Now?
What language models are capable of today was almost unimaginable in late 2022, when ChatGPT was launched. They analyze requirements, derive test cases, and write test scripts. As a result, AI is increasingly taking over testers' intellectual tasks.
According to Arvato Systems, AI is now technically capable of creating and automating tests for many applications largely on its own. More complex domains are likely to follow soon. Nevertheless, AI is still missing from the day-to-day testing activities of many projects, mostly because its adoption within teams and processes is progressing slowly.
Then there's the issue of speed. If AI speeds up code development but testing remains manual and slow, testing risks becoming a bottleneck.
The Principle: Responsibility Lies With the Individual
One fundamental principle applies to all five guiding principles: “Humans Own Quality.” Responsibility for quality remains with the individuals and teams in charge. Trust in AI agents when it comes to release decisions has actually declined recently. According to the Tricentis Quality Transformation Report 2026, only 34% of IT and QA managers now leave release decisions to AI agents. A year earlier, that figure was 48%.
That is why a risk-based approach makes sense. For critical functions, testers delve into the application itself and look for errors or gaps in the AI tests. Complete autonomy is currently conceivable only for very non-critical applications or functions. How far it will extend in the future remains to be seen. A good compromise: AI specifies and automates the tests, while humans review both. This creates a verified and reproducible foundation for all future executions.
What Will Change Specifically
If teams consistently take advantage of the possibilities offered by AI, their testing strategy will change fundamentally. The following guiding principles point the way—and are intentionally exaggerated.
-
Guiding principle 1: test early, test always – test early and continuously
Shift-Left and Test-Driven Development have been considered best practices for years. In day-to-day project work, however, their consistent implementation often fails due to the effort involved, existing processes, or differing perspectives within the team.
With AI, unit tests and acceptance criteria can be generated in minutes, lowering the barrier to entry. AI agents also consistently adhere to agreed-upon rules—such as “test-first”—even under time pressure. This enables teams to consistently implement a test-driven approach, with tests serving as an executable specification, similar to Spec-Driven Development.
The AI can test completed work immediately. If an automated test is missing, an AI agent checks the change until a reproducible test is in place. This also includes Vibe Testing: Business units describe in their own words what they want to test, and an agent performs the test. Structured testing remains the foundation. Vibe Testing complements it very well—and, above all, quickly.
After a rollout, the automated scripts run, and—in a completely new development—the AI agent performs exploratory testing. A triage process separates genuine errors from noise. The agent-based exploration complements the reproducible tests but does not replace them.
-
Guiding principle 2: no manual testing – no more manual testing
When AI specifies a test case, it also develops the test automation. In the best-case scenario, this completely eliminates the need for manual test execution. The AI could also run the tests directly without automation code, but that consumes a lot of tokens and can be slow at times. Once generated, however, test scripts can run as often as needed without the AI.
For years, test automation rarely accounted for more than a quarter of all tests. According to Forrester, companies using autonomous testing platforms now achieve 51-60% (Forrester, 2026). Manual testing was often still necessary, however, because scripts do not check all aspects of an application, such as layout and text. AI can now handle even these visual checks.
Until now, risk assessment has primarily determined what remained untested due to time constraints. In the future, it will guide human attention. The AI largely handles secondary functions—such as layout variations or help texts—on its own. This frees up time for testers to focus on core functions. They understand the technical logic behind these functions and specifically look for errors in processes that affect revenue, security, or service delivery.
-
Guiding principle 3: close the coverage gaps – close gaps in testing
With AI, creating an additional test costs next to nothing. This allows teams to fill in the gaps they previously didn’t have time to address, such as threshold values or rare data combinations. Meta uses language models to generate tests targeting specific error classes, such as data protection errors. In test-a-thons for Messenger and WhatsApp, developers accepted 73% of these tests (Meta, 2025). This additional coverage becomes valuable when it addresses the relevant risks.
-
Key point 4: AI breaks up the tool market – AI opens up the market for testing tools
AI has now replaced many of the functions of commercial test suites, such as script generation, capture-and-replay, and self-healing. Open-source tools are reaching new heights with AI and are, in some cases, replacing expensive off-the-shelf solutions.
Switching tools also becomes affordable if the AI supports existing test cases during migration. Whether the switch is worthwhile depends on the feature set, integrations, and maintenance requirements. At the same time, new dependencies on the providers of language models and agent platforms may arise.
For years, Arvato Systems has relied primarily on open-source tools such as Robot Framework and Cucumber. Both can be used directly with agent-based AI. This makes new AI capabilities immediately available without requiring a vendor to adapt its product first. Combined with standardized interfaces such as the Model Context Protocol (MCP), this creates a powerful and cost-effective infrastructure for test automation.
-
Guiding principle 5: always with AI – AI in every testing strategy
AI is used in every testing strategy, in both B2B and B2C business, as well as in critical infrastructure (KRITIS) and regulated markets. The tasks it performs depend on the context. It delivers value everywhere. Where required by law, acceptance testing is conducted without AI, and humans approve the results. AI supports all the steps leading up to this, thereby accelerating the acceptance process.
AI Testing in Practice: Three Projects by Arvato Systems
At Arvato Systems, teams work with ROLF (Real Outcomes Less Friction), a framework for agent-based software development that the company developed in-house. Four specialized agent groups cover the entire process from concept to tested code:
- AIRE(Requirements Engineering) organizes ideas and input into requirements, user stories, and use cases.
- EARTH (Software Architecture) analyzes the architecture, models the business domain, and documents architectural decisions before implementation begins.
- FIRE(Development) plans the implementation using a test-driven approach, generates code, and reviews it during code reviews.
- AQUA (Quality Assurance) derives test cases, automates them using tools such as Robot Framework, and verifies test coverage. Each test remains linked to its corresponding requirement in AIRE.
Teams don't have to use all four groups; many start with just one. A person approves the plan and reviews each key step. The testing strategy serves as a guideline for every AI agent. Three projects demonstrate the benefits of this approach:
- New Development in Healthcare: Based on formal requirements, approximately 400 test cases were developed using AI. The effort required was reduced by about 50%.
- Existing Healthcare Application: The agents derived approximately 150 automated test cases for the graphical user interface from the existing application. The effort required was reduced by about 70%.
- In-house development: Using the complete ROLF chain, approximately 500 automated test cases were generated. The effort required was reduced by about 90%.
These figures represent the project teams’ estimates regarding test design and test automation under ideal conditions. The effect is less pronounced when considered across the entire development process. It was greatest in the project where all four agent groups collaborated from requirements through testing.
By the end of 2026, at least 80% of all new test cases and test automations at Arvato Systems are to be created using AI. For changes to legacy tests, the target is 50%. At the same time, the division is preparing for an “AI-first” strategy in which manual test execution will be phased out gradually.
Challenges for the AI Testing Strategy
The World Quality Report, 2025 quantifies the gap: 89% of the companies surveyed use or are testing generative AI in quality engineering, but only 15% have scaled it company-wide. Adoption therefore does not depend solely on the performance of the models. Integration, data protection, reliability, and expertise are equally important hurdles.
New skills take practice, and many people wonder what will become of their roles. Technically oriented testers often start with the basics: How does a large language model (LLM) work? How do you use GitHub Copilot?
Until now, roles in testing have been clearly defined: some design test cases, others execute them, and still others automate them. With AI, these boundaries are blurring. In the future, there will be a demand for quality engineers who can combine all three tasks. This won’t happen overnight, because those who have only handled one of these tasks so far will first have to learn the others. Test management remains just as important in large and critical projects.
Every project requires its own AI implementation strategy, ranging from training and context setup to the pilot phase and rollout. Existing projects require additional preparatory work because agents must first analyze existing tests. According to the Google’s DORA Report 2025, AI amplifies what already exists within a team—both strengths and weaknesses. Our article on Modernization with AI-Supported Software Development.
Each agent run consumes tokens. At Slack, an agent-based E2E test workflow cost between $15 and $30 per run (Slack Engineering, 2026).
The return on investment is usually very good, but initially these costs are added to the existing ones. Ideally, this directly generates revenue because a team delivers more with less effort. Otherwise, costs simply rise.
Many customers specify which models, hosting locations, and data are permitted. Personal test data is often not allowed in cloud models. Synthetic or anonymized data, as well as EU-hosted or locally operated models, can help in this regard.
Implementing AI Testing: First Steps
- Define the framework: Before the project begins, the client's requirements are aligned with the models and test data, and it is determined who will review and approve the AI tests.
- Empowering the team: Training sessions cover the basics—from large language models to working with agents—before the first project begins.
- Establishing context: The team prepares requirements, the test strategy, and existing tests so that the AI can use them. In existing projects, additional preparatory work is required.
- Launching a pilot: A manageable, low-risk area with clear requirements is a good place to start. The AI specifies and automates the tests, while humans review them. The effort involved, test quality, and AI costs are measured.
- Scaling: The lessons learned are incorporated into the test strategy and future projects. Measurable goals, such as the percentage of test cases generated by AI, make progress visible.
Arvato Systems has had positive experiences developing its AI approach independently of the constraints of day-to-day business operations. This is also how ROLF came about—starting with the question of what an “AI-first” approach looks like and what it can achieve to the fullest extent possible.
Outlook: Agentic Testing Gains Ground
AI agents are becoming more specialized. Testing agents explore an application, create test plans, and fix failing tests. Playwright provides such agents as open source (Playwright Test Agents). Other vendors and the open-source community provide their own agents and skills. A task such as “Test the login; the test data is in folder X” already works in many development environments today. With more efficient models, the cost per agent run is likely to decrease. Then, agentic testing will also be cost-effective for individual changes.
The more tests the AI generates, the more important their quality becomes. Mutation testing measures this quality using artificially introduced errors. Meta is already using this method.
Arvato Systems' QA experts will evaluate your existing testing strategy, identify where AI testing will be most beneficial, and use that information to develop a roadmap for implementation.
Written by
Dr. Rene Stehr has been testing software for years, with a focus on test automation, including in regulated and critical industries. As Senior Manager of Quality Assurance, he leads the QA division at Arvato Systems, which has approximately 80 employees across three countries. He is responsible for the strategic development of the division, particularly the implementation of AI.