AI agent for test generation: a playbook

The problem

A test suite that falls behind the code is a quiet cost center. Teams skip tests near a deadline, a bug slips out, and the fix takes longer than the test would have. An AI agent can read a module, propose cases and open a draft pull request. That is useful, and easy to get wrong. Raw generation produces volume, not confidence. The agent can write tests that mirror the current implementation, pass on the first run and catch nothing. It can mock away the part that matters or edit production code so a test passes.

The hard part is not typing a test. It is choosing the behavior worth pinning down, finding a trustworthy oracle, and deciding which tests earn a place. A good agent needs repository access through approved APIs, a sandbox runner, coverage and mutation tools, and a firm boundary: it writes candidates to a branch and never merges. Each candidate carries a plan: target behavior, input, expected result, risk covered and the source of that result. If the agent cannot say why the expected value is right, the test is not ready.

Legacy code makes this sharper. Old modules may be hard to instantiate, so the agent will propose seams or wrappers. Those are production changes and need human design review. If you tell the agent to raise coverage, it will chase lines. If you tell it to catch defects in payment rounding and date parsing, it will make choices you can review. Start with one module and one class of risk. The guide AI agent for test generation: a playbook works when the scope is narrow enough to measure.

What a good solution looks like

A good setup looks like a small test factory with a review desk. The agent reads the module, existing tests, the issue tracker and recent bug fixes. It writes a test plan before code, listing behaviors, edge cases and risks. It then adds tests using the framework the repository already uses, such as pytest, Jest, JUnit, Playwright or Cypress. It does not add a new framework, mocking library or test dependency without approval. It runs the tests in a clean environment and attaches the output, coverage report and runtime to the draft pull request.

Human approval sits at three gates. A reviewer approves the test plan or asks for a narrower target. A reviewer reads the tests and checks the oracle. A code owner approves the merge. The agent may answer review comments and push commits, but it cannot merge, delete tests or change branch protection. For production code changes, it opens a separate pull request with a reason and a rollback note. That separation keeps test generation from becoming an unreviewed refactor.

Tool access should be boring and explicit. The agent gets a repository token with read access and branch write, not admin. It gets a CI runner or a container with the same image as CI, plus API access to the issue tracker and coverage report. It should not have production credentials, customer data or a browser session into an admin console. If a test needs seed data, provide factories or a sanitized fixture set. Log every tool call: file read, file write, test run, coverage read, pull request create.

How to build it

Pick one module with clear behavior and weak tests. A pricing calculator, date parser, permission check or import mapper works better than a sprawling controller. Collect the last several bug fixes in that module, including issue text and patch. That is your evaluation set. Ask the agent to generate tests for those old bugs. If it cannot reproduce the bug with a failing test before the fix, it is not ready for new work. That step separates a demo from a tool your team trusts.

Connect tools in this order: repository read, branch write, test runner, coverage report, issue tracker and CI status. Use approved APIs, not screen automation. Give the agent a command allowlist: install from the lockfile, run tests, run coverage, run a mutation tool on a small directory. Block network access except to the package registry and the APIs it needs. Set a token budget and a wall clock limit per run. When either limit is hit, the agent stops and posts what it has.

Write test conventions down. Say which fixtures to use, which mocks are allowed, how to name tests and where files go. Tell the agent to prefer real objects over mocks and keep one behavior per test. Tell it to avoid sleep calls, shared state and order dependence. Require repeat runs and a report of any failure. If a test fails, the agent must show the failure and leave the assertion alone. Loosening an assertion to get a green run is a human decision. Add a review checklist: oracle clear, no production edit, no secret, no new dependency, runtime within budget. Roll out in stages: plans only, then tests with edits, then tests with light edits.

Risks and how to manage them

The main risk is false confidence. A test that asserts the current output of a broken function will pass forever. Mutation testing helps: change a line in a copy of the code, run the new tests, and count how many changes they catch. If the new tests kill few mutants, they are decoration. Run mutation testing on touched files, not the whole repository. Keep runtime in check and treat surviving mutants as review notes, not a gate that blocks every merge.

Over-mocking is the second risk. Agents like to mock dependencies because it makes tests easy to run. That can hide the integration that fails in production. Set a rule: mock at the boundary you do not own, such as a payment gateway or email provider. Use a fake or local stub for internal services where you can. For database code, use a real test database in a container, not a mock that returns a list. Reviewers should ask what the test would catch if the real dependency changed.

Security and cost need hard limits. Never give the agent production secrets, customer records or admin access. Use short lived tokens, a dedicated CI service account and an isolated network. Scan generated files for secrets and copied code. Cap tokens, tool calls and runtime per run and per day. Alert when a run hits the cap. A runaway agent that loops on a failing test can fill a branch with commits. Set branch protection so required checks pass and code owners review. Give the agent no merge button. Quarantine flaky tests instead of deleting them. A test that fails a repeat run goes back with the log.

How to measure success

Measure at the pull request level. Count accepted candidates, rejected candidates and candidates that needed a human edit. Track rejection reasons: weak oracle, over-mocking, flake, runtime, production edit, unclear plan. That list tells you what to fix. Count tests that found a real defect before release, not just tests that passed. A test that never failed and never caught a bug is a cost until proven otherwise.

Use coverage as a directional signal, not a target. Record the delta in covered lines and branches for the touched module. Pair it with mutation kills on the same files. A rise in coverage with no rise in mutation kills means the agent writes tests that execute code without checking behavior. Track runtime delta for the suite and the count of flaky tests. If review time per accepted test climbs, the agent is producing noise. Review time is a real cost and belongs on the same dashboard.

Watch the module after release. Count escaped defects and regressions in the touched area and compare with the period before the agent started. Look at bug reports that mention behavior the new tests should have covered. Feed those cases back into the evaluation set. That is how the agent gets better without vague prompt tuning. Hold a monthly review with the test owner, the reviewer and the module engineer. Decide whether to expand, hold or stop. If you want outside help, Cloudverg's AI agent build package is $19,500 at a fixed price, runs 5 to 6 weeks and starts within 10 business days. It covers one production workflow with a written success measure, tool access through approved APIs, human approval for actions that spend money or message customers, an evaluation set from your past cases, token and time budgets with alerts, full logging, and a monitoring dashboard plus a 30 day tuning window. Treat it as one option among building in house or hiring a contractor.

Frequently asked questions