Playbook

Taking an AI pilot to production

How to move an AI pilot into production: one workflow, a frozen eval set, approval gates, cost caps and a staged rollout.

By the Cloudverg engineering team · · 5 min read

The problem

Most AI pilots do not fail at the model. They fail at the handoff. The demo ran on a laptop with a pasted API key, three sample documents and a person watching every output. Nobody wrote down what a correct answer looks like, so nobody can say whether the thing works.

Scope is the other problem. Pilots tend to cover a whole job: handle our invoices, answer our support mail. Production needs one workflow with a clear start and a clear end. Categorize an inbound invoice, match it to a purchase order, then post it or send it to a human. That is a workflow. Do accounts payable is not.

Then the unglamorous part. Permissions, retries, timeouts, a place to look when something breaks, and a named person who gets called. Pilots skip all of it because the pilot ends when the meeting ends.

What a good solution looks like

A production AI feature has four things a demo does not: a written success measure, a fixed set of test cases, limited tool access, and cost limits. Everything else is detail.

The success measure is one or two sentences with numbers from your own records. Something like: nine in ten invoices are coded and posted without a human edit, no payment goes out wrong, and the median case finishes in under a minute. Write it before you build. If two people read it and picture different outcomes, rewrite it.

Tool access runs through approved APIs with scoped credentials. Not a shared login, not a bot that clicks through screens. Every function the model can call gets its own permission and its own dry-run mode. Anything that spends money, deletes a record, or messages a customer outside the team goes behind an approval step.

Cost limits belong in the design, not in a spreadsheet you check on Fridays. Set a token budget per case, a step limit per run, and an alert when a single case crosses a threshold. Add a switch that sends all traffic to humans. You will use it.

How to build it

Start with the workflow map. Write down every step a person takes now, the system of record for each step, and what happens when a step fails. You are looking for the input shape, the output shape and the failure path. If the output cannot be written into the system of record in one call, the workflow is not ready.

Then build the evaluation set. Pull 100 to 300 past cases from your own queue, including the ugly ones: duplicates, missing fields, a supplier that changed its name, attachments that are scans of scans. Label the correct outcome by hand. Have two people label the first 30 separately and compare. Every disagreement is a gap in your success measure, so fix the measure, not the label.

Build the test harness before you tune anything. Every change to a prompt, a tool description or a model version gets a full run, and you record which version produced which result. Pin the model version. When the provider ships a new one, rerun the set before you switch.

Last comes the human step. Build the review queue as a real screen: the source document on one side, the drafted output on the other, one key to approve and one to edit. If approving takes longer than doing the task from scratch, the pilot dies there, however good the model is. Run the whole thing in shadow mode for a couple of weeks beside the people doing the work now, compare outputs daily, then let it act.

Risks and how to manage them

The main risk is a confident wrong answer. The model returns something that reads well and carries the wrong invoice number. Guard against it with schema validation on every output, a cross-check against the system of record for key fields, and a rule that low confidence means handoff, not a guess. Never let a model invent an ID, an amount or a date. Pull those from the tool response.

Cost creep comes from loops and retries. A model that keeps calling the same tool with the same arguments can burn a budget in minutes. Cap the steps per run, cap the wall clock, and log every tool call with its arguments so you can see the loop. Set the alert low enough that you hear about it before finance does.

Ownership is the risk people skip. Name a primary owner and a backup. Write down what to do when the queue backs up, when the API is down, and when the output looks wrong. That page should fit on one screen and should be tested once, badly, on a Friday.

Scope creep arrives in week two: can it also send the email, can it also handle credit notes. Keep a list and say no for now. Every added field widens the eval set and the failure surface. Watch what data leaves your network too. Send the minimum, redact what the task does not need, and keep the raw records in your own store.

How to measure success

Measure two things and keep them apart. The eval score on the frozen set tells you whether the system still does what it did on day one. The business number tells you whether the workflow got better. If the eval holds and the business number does not move, the problem is usually upstream: wrong cases, wrong queue, or a step nobody owns.

Track a few operating numbers weekly: share of cases finished without a human edit, median time to approve, cost per completed case, how often the model hands off. Look at the mix of failures, not just the rate. Ten wrong supplier matches matter more than a hundred formatting fixes.

Set a review at 30 days. Decide to expand, hold or stop, and write the reason down. Expansion should mean one more workflow, not a bigger prompt.

Buying the build is one option among several. Cloudverg sells an AI agent build at $19,500/month that covers one production workflow, the evaluation set, approval steps, budgets and a 30 day tuning window. Weigh that against your own timeline and the cost of pulling engineers off other work.

Frequently asked questions

How long does it take to get an AI pilot into production?

One workflow usually takes a few weeks of focused work, not a few months. The slow parts are labeling the evaluation set, building the approval screen and getting API credentials from the systems team. The model choice is rarely the bottleneck.

What is an evaluation set for an AI feature?

It is a fixed batch of past cases with the correct outcome labeled by hand. You run it on every change to the prompt, the tools or the model version, so you can see whether a tweak helped or hurt. Keep it frozen and add new hard cases to a separate set so your scores stay comparable.

Should an AI agent send customer emails on its own?

Not at first. Put an approval step in front of anything that messages a customer, spends money or changes a record. Once drafted messages get approved without edits most of the time and the failure mix is boring, you can loosen it for narrow cases with a clear rollback.

Monthly plan

Want help with this? AI agent build

$19,500/month

5 to 6 weeks · Starts within 10 business days

Five to six weeks to put one AI agent into production on a real workflow, with tools, approvals, evaluations and cost limits in place.

Renews monthly. Cancel any time.
What is included

Related guides

More ai and agents guides   AI and machine learning

Smallest way to start

Get a senior architect on your problem this week

Ninety minutes a month on the cloud, AI or security question that is blocking you, a pre-read of your material and a written summary within two business days.

$450/month

Renews monthly. Cancel any time.