Synthesized has announced Test Data Agent, an infrastructure capability being developed to create realistic test data and system conditions for evaluating AI agents before they are deployed into production environments.
The product entered limited availability for existing clients and ecosystem partners recently, with general availability planned for later in the third quarter of 2026. Synthesized said the technology is also being tested with Tier 1 global bank design partners.
The tool is designed to work alongside agent development, evaluation, testing, and orchestration frameworks. Rather than scoring an agent’s output, it provides the data, business relationships, permissions, and application states needed to test whether an agent can complete enterprise processes correctly.
Why agent testing is different
Testing an AI agent introduces different requirements from testing deterministic software. Traditional automated tests can often check whether a known input produces an expected output, while agents can make a series of decisions, call external tools, and alter application state before reaching an outcome.
Anthropic said in its January 2026 guidance on agent evaluations that agents operate across multiple turns, use tools, modify their environment, and adapt to intermediate results. It also noted that model outputs can vary between runs, requiring multiple trials to establish how consistently an agent succeeds at the same task.
AWS has identified the same issue in its guidance for production agent testing. The company said identical inputs can lead to different outputs because agents make context-dependent decisions, making single-trial testing less reliable than it is for deterministic applications.
These behaviours affect the scope of testing. An agent interacting with enterprise applications can encounter missing records, unusual transactions, access restrictions, and dependencies between systems that may not appear in controlled evaluations.
Testing the final response alone can also miss errors earlier in the process. An agent can produce a plausible answer even when it has selected the wrong tool, passed incorrect parameters, or taken an incorrect action.
Anthropic distinguishes between an agent’s transcript and the resulting state of the environment. Its guidance notes that checking what an agent says it accomplished is insufficient when the task also requires a corresponding change to a system, application, or database.
AWS similarly recommends evaluating tool selection, parameter passing, execution paths, and task completion alongside the final output. Its July 2026 production blueprint separates these checks to identify failures that may not be visible from the response alone.
Building repeatable test environments
Reproducing enterprise operating conditions creates another challenge. Non-production environments can be incomplete or outdated, while security and regulatory requirements can restrict the use of production data in development and testing.
Test Data Agent generates or provisions data and system states around a specified business scenario, according to Synthesized. It identifies the records, relationships, and conditions required for a test before creating the environment in which the agent is evaluated.
The system can generate synthetic data, mask sensitive information, or create subsets of production-representative datasets. Synthesized said it is intended to preserve referential integrity, statistical characteristics, and business rules across connected systems.
Teams can also create normal, exception, failure, and adversarial scenarios, then refresh those environments as applications and datasets change.
Using synthetic data addresses the sensitivity of production information, but generated datasets still need to reproduce the characteristics of the environments they are intended to represent.
Microsoft Research examined this issue in its SynAE research published in May 2026. The researchers said internal production datasets used to evaluate tool-calling agents can be unsuitable because they contain sensitive or proprietary information, while available datasets can also be too sparse for comprehensive pre-deployment testing.
SynAE evaluates synthetic datasets for validity, fidelity, and diversity across task instructions, intermediate responses, tool calls, final outputs, and downstream evaluation results.
Synthesized said its system is intended to preserve the records, relationships, and system conditions relevant to a workflow without requiring raw production data to be transferred to an external testing service.
“Building an agent is becoming easier. Proving that it can be trusted with a real business process is not,” Baldin said.
Repeatability also matters for regression testing. If the agent and its test environment both change between runs, it becomes harder to determine whether a different result came from the model, prompt, tool, orchestration logic, or underlying data.
Anthropic recommends running evaluations in stable environments and starting individual trials from a clean state. Leftover files, cached data, shared state, or infrastructure constraints can otherwise affect the result independently of the agent.
According to Synthesized, Test Data Agent can recreate the same environment across multiple runs, allowing model, prompt, tool, or configuration changes to be tested against the same underlying data and system state.
Microsoft also recommends maintaining evaluation suites throughout an agent’s lifecycle. Its May 2026 guidance identifies model changes, major knowledge-base updates, new tool or connector integrations, pre-production deployments, and production incidents as triggers for running a full evaluation suite.
Anthropic distinguishes capability evaluations, which test what an agent can do, from regression evaluations, which check whether previously working tasks continue to succeed after changes.
Regression testing therefore extends beyond checking the final response. An update can change the tool selected, parameters passed, sequence of actions, or resulting system state even when the final output still appears acceptable.
Repeatable environments can also support comparisons between models or agent configurations. Synthesized said competing versions can be tested against the same enterprise scenarios while the surrounding data, permissions, dependencies, and application states remain fixed.
Bringing agent validation into release pipelines
Test Data Agent can be called through REST APIs and CI/CD pipelines, allowing existing testing and evaluation systems to request the environments they need.
Microsoft’s Copilot Studio supports automated agent evaluations through a REST API and identifies agent updates, release validation, regression testing, and CI/CD pipelines as use cases. Its documentation says predefined test sets can be triggered programmatically during development to detect behavioural regressions.
AWS has also demonstrated agent evaluation as part of deployment gating. Its July 2026 production blueprint evaluates tool use, reasoning, and output quality during development and can block a release when defined thresholds are not met.
Test Data Agent is designed to supply the data and application state used by these types of evaluation systems rather than score the agent itself.
The system can run inside customer-controlled on-premises, private-cloud, and hybrid environments. Synthesized said data generation, masking, and provisioning remain subject to an organisation’s existing identity, networking, security, and governance controls.
According to the company, this setup is intended to support production-representative testing without requiring raw customer, employee, financial, or commercially sensitive data to be exported to an external service.
Synthesized is also developing support for SAP environments, where business processes can span multiple tables, applications, authorisation rules, and organisation-specific configurations.
An invoice-processing agent, for example, can depend on linked supplier records, purchase orders, invoices, payment terms, currencies, and access permissions. Synthesized said the system can recreate those conditions so an agent can be tested against standard transactions and exceptions before it receives access to live financial systems.
“The difficult part of testing an SAP agent is not demonstrating that it can navigate a workflow,” Baldin said. “It is recreating the relationships, exceptions and business conditions that determine whether the resulting action is actually correct.”
(Photo by Growtika)
See also: AWS DevOps Agent traces pipeline failures to GitHub commits
Want to dive deeper into the tools and frameworks shaping modern development? Check out the AI & Big Data Expo, taking place in Amsterdam, California, and London. Explore cutting-edge sessions on machine learning, data pipelines, and next-gen AI applications. The event is part of TechEx and co-located with other leading technology events. Click here for more information.
Developer Tech News is powered by TechForge Media. Explore other upcoming enterprise technology events and webinars here.
