How to build AI agents that actually work in prod
At Orus, we built an AI agent to handle the termination of a customer's previous insurance contract after they subscribe to a new one. The process covers roughly 500 cases a month. Internal estimates put the previous handling time at 15 to 20 minutes of human work per case.
On paper, it looked like a great fit for an agent. Read a contract, collect missing information, prepare a mandate for signature, send a registered letter, and keep the customer informed. The process was documented and repetitive. Very little of it seemed to require judgment.
Then we replayed real conversations. The documented process and what our operations team actually did diverged in roughly 30% of cases. Documents were incomplete. Customers replied out of order. Humans added attachments without explaining what they meant. Third-party APIs failed after an action had been approved. The same conversation could be picked up by several executions. Some cases waited for a signature for days.
The model was not the hard part. Building a system that could survive all of this was.
In its first six weeks of production, which happened to span the summer slowdown when volumes run well below the yearly average, reviewers made 659 validation decisions on the agent's proposals. They accepted 60% unchanged, corrected 35%, and rejected 5%. We review every customer-facing action, which gave us a safe rollout path and a dense feedback loop. As evidence accumulates, execution can become automatic, backed by deterministic checks, for actions where the risk justifies it.