AI agent for code review: a playbook

The problem

Pull requests queue up faster than anyone wants to review them. Senior engineers read diffs at the end of the day, tired, and the quality of their comments swings with their energy. Attention lands on formatting while a missing null check goes through. New hires don't yet know what this codebase treats as normal.

Pasting a diff into a chat model looks like a fix for about a week. The comments sound confident and half of them are wrong or repeat what the linter already reported. The model cannot see the function two files away that the change breaks, and it cannot run the tests. Reviewers start skimming the bot, then filter it out.

Code review suits an agent in a narrow way. The work is text, it repeats, and a person already stands between the comment and the merge. What fails is treating it as a prompt. An agent that can read the repository, run the linters and post one careful review is a different tool from a chat window with a diff pasted in.

What a good solution looks like

It runs on pull request events, not on a timer. It reads the diff, the files around it and the definitions the changed code calls, then posts a single review with a few comments. Each comment names a file, a line, a severity level and the reason. Severity is what makes it usable: blocking means a defect or a security issue, warning means a plausible bug, note is a suggestion a reviewer can ignore.

It calls tools instead of guessing. Language linters and type checkers first, then pattern scanners like Semgrep, a secrets scanner like gitleaks, and the test runner for the packages that changed. A good agent explains why a check failed and points at the line. It should not re-derive what Ruff or ESLint already said, because the reviewer will see both.

It stays quiet when it has nothing useful to say. An empty review is a valid result. It has a cap on comments per pull request and on tokens and wall clock time. Files matched by ignore globs, generated code, lockfiles and vendored dependencies, never get reviewed.

It cannot approve, merge or push. Give it a read token and comment permission, nothing else. Approval stays with people, and branch protection keeps required reviewers in place.

How to build it

Start with the trigger and the queue. Subscribe a webhook to pull request events: opened, synchronize, reopened. Verify the HMAC signature on every call or anyone can spam your pipeline. Put each job on a queue so a large diff does not stall the CI run that is waiting on it.

Fetch the diff with the provider API, GitHub's pull request files endpoint or the GraphQL equivalent, and check out the merge commit in a sandbox container. Keep the sandbox off your internal network and give it a time limit.

Assemble context on purpose. The full diff, about thirty lines around each changed function, and the definitions of what those functions call, found through code search or a tree-sitter parse. Include your CONTRIBUTING file and the last twenty review comments on the touched files so house conventions come through. Skip binary files. If the diff is huge, review file by file and post one summary comment instead of forty. Let the agent ask for something it lacks, one extra file or one code search, and cap that at two rounds. Run the linters, type checks and the changed test packages inside the sandbox and feed their output in as facts rather than questions.

Require structured output. A JSON array with file, line, severity, category, a one-sentence claim, a quoted line as evidence, and a suggested fix in prose. Drop any finding whose evidence line is not in the diff. Deduplicate against existing comments and against linter output. Post it as one review with the COMMENT event. Add a status check that fails only on blocking findings, and make that check required in branch protection only after the review has earned it.

Risks and how to manage them

The main failure is noise. A wall of comments on every pull request trains people to ignore all of them. Cap the count, order by severity, and let reviewers react with a thumbs down on each comment. Store those reactions, they are the cheapest feedback signal you will get, and read them weekly.

Wrong claims come next. Models misread control flow and invent method names. The evidence field is your guard: if the quoted line is not in the hunk, throw the finding away. Route security findings to a named person before they appear on the pull request, because anyone with read access can see a comment.

Watch what leaves your network. Read the provider's data retention and training settings, and prefer an enterprise endpoint with zero retention or a self-hosted model for repositories you care about. Run gitleaks and strip anything it finds before the text goes to the model. Keep a deny list of repositories that never get reviewed by the agent.

Treat code comments and pull request descriptions as untrusted input. A comment that says to ignore previous instructions is data, and your system prompt must say so. The subtler risk is people relaxing. Required approvals and CODEOWNERS stay exactly where they are, and the agent's comment never counts toward the approval count. Force pushes leave stale comments, so re-run on synchronize and resolve findings that no longer apply, or reviewers will chase ghosts.

How to measure success

Build the test set before the rollout. Take fifty to a hundred merged pull requests that had review comments, plus a set that later needed a hotfix. Run the agent in shadow mode, posting to a private channel instead of the pull request. Count how many real issues it catches and how many comments a reviewer would dismiss.

Then track a small set of numbers weekly. Comments posted, share marked helpful or resolved, time from pull request open to first human review, review rounds per pull request, and reverts or hotfixes on files the agent touched. Compare against the weeks before the agent existed, not against a vendor claim.

Cost matters too. Log tokens per pull request, sandbox minutes and the time from open to posted review. Set alerts before the cap, so a runaway job gets caught by a person and not by an invoice.

Then make a judgement call. If the helpful share stays low, cut the scope to security and test coverage findings only. If reviewers stop reading, turn it off and say why. A wider build, like the Cloudverg AI agent build at $19,500 fixed price, packages one production workflow with tool access, human approval, an evaluation set from your own history, budgets and a monitoring dashboard, but a webhook, one linter and one comment type will tell you most of what you need to know first.

Frequently asked questions