When AI automation is worth building — and when it isn't

Most companies asking about AI automation have already seen a demo that impressed them. The harder question is which of their own processes are worth automating, and that is a commercial question before it is a technical one.

We use four tests when assessing a process. A process that fails any of them is usually better left alone, or fixed with ordinary software instead.

1. Volume justifies the engineering

Automation has a fixed cost: integration, evaluation, monitoring, and the ongoing maintenance of a system that talks to a model whose behaviour changes between versions. A process that runs eleven times a month rarely repays that. A process that runs eleven times an hour usually does.

Before scoping anything, count the occurrences and the minutes each one takes. If the annual hours saved do not comfortably exceed the build and running cost, stop there.

2. The cost of a wrong answer is bounded

Language models are probabilistic. Any process you automate will occasionally produce a wrong output, and your design has to assume that rather than hope otherwise. The relevant question is what happens when it does.

• Drafting a reply a human sends: a wrong answer costs a few seconds of editing.
• Categorising a support ticket: a wrong answer costs a routing correction.
• Issuing a refund or changing a price: a wrong answer costs money and trust.

The first two are good candidates. The third needs a human approval step — which is fine, and still saves most of the time, but it changes the design and should be priced in from the start.

3. Success can be measured

If you cannot describe what a correct output looks like, you cannot evaluate the system, and without evaluation you have no way of knowing whether a prompt change or a model upgrade made things better or worse.

In practice this means assembling a test set of real examples with known-correct outputs before building anything. Fifty is usually enough to start. That set becomes the regression suite that lets you change the system with confidence later.

4. The data is actually reachable

AI automation projects stall on plumbing far more often than on model quality. The information the system needs sits in a CRM with no API access, or a shared drive nobody has audited, or a database whose fields stopped matching their names three years ago.

Establish where the data lives and how you will get to it before committing to a timeline. This is where estimates go wrong.

What this looks like in practice

A process that passes all four tests gets built in a specific order: assemble the evaluation set, wire up data access, build the narrowest useful version, measure it against the set, then add scope. A process that fails one gets a smaller intervention — often a well-designed form, an integration, or a scheduled job, none of which involve a model at all.

The most valuable outcome of an automation assessment is sometimes a short answer explaining why the process is not worth automating yet.

That answer costs a fraction of a build and saves considerably more.