Two systems

Three years ago I was asked to solve a problem that most operations teams eventually hit: inbound email arriving faster than any reasonable number of people could answer it, across eleven separate business portals, each with its own tone and its own policy.

Every message had to be understood, categorised, matched against internal records, answered, and logged. The existing approach was the obvious one — more people, working faster, making more mistakes as volume grew.

The system now drafts more than 10,000 replies a day.

I designed, built, deployed and now monitor the system that replaced most of that work. Here is what it does, what it cost to get right, and what I'd tell anyone considering the same thing.

The architecture

Twelve specialised agents, orchestrated rather than chained. Each does one narrow job well.

Intent detection decides what the message is actually asking for — which is rarely what it appears to ask for. Aspect-based sentiment analysis flags the ones that are angry, because those need a human within minutes. Query extraction pulls the specific entities involved. Retrieval finds the relevant records across both a vector store and a structured database. A drafting agent composes the reply. A summarisation agent compresses long threads so nothing is lost mid-conversation. An auditing agent reads the draft before it goes out and blocks anything it can't support with retrieved evidence.

Response time under two seconds. Full tracing on every run.

The part nobody talks about

Building agents that work in a demo takes days. Building agents that work on the ten thousandth message of a Tuesday takes months, and almost all of that time goes into three things nobody puts in a pitch deck.

Evaluation. If you cannot measure whether the system got better or worse after a change, you are not engineering, you are guessing. We needed a scored test set before we needed anything else. Every prompt change ran against it. Roughly a third of the changes that felt like improvements measurably weren't.

Guardrails. A language model asked a question it cannot answer will answer anyway. In a customer-facing pipeline that is not a quirk, it is a liability. The auditing agent exists because confident wrongness is more expensive than silence — a system that escalates to a human is inconvenient; a system that invents a delivery date is a lost account.

Monitoring. These systems degrade quietly. Model versions change under you. Input distributions drift as the business changes. Costs creep. Accuracy falls a percentage point a month and nobody notices until a customer complains. Full tracing was not a nice-to-have; it was the difference between operating the system and hoping.

What it produced

Measured by the business, not by me: 70–80% of manual intervention eliminated. Turnaround time down by more than 90%. Sub-two-second responses sustained at full volume.

What matters more than the numbers is what didn't happen. Nobody was laid off. The team stopped doing the repetitive 80% and started handling the exceptions — the escalations, the angry customers, the genuinely complicated cases where a human being is worth the money. That is the honest version of what this technology does well.

What I'd tell you if you're considering this

Don't start with a chatbot. Start with your highest-volume, most repetitive inbound process and ask how much of it is genuinely judgement and how much is lookup and formatting. Usually it's 80% lookup.

Budget for evaluation before you budget for the build. A system you can't measure will fail slowly and you won't know why.

Design the human escalation path first, not last. Every worthwhile system in this category is a human–machine system, and the ones that fail are the ones where the machine had no way to say “I don't know.”

And be suspicious of anyone who shows you a demo without showing you what happens when it's wrong.