Structured outputs, always
Never parse free-form text to drive an action. Use function calling or schema-constrained output, then validate it in code before anything happens.
Every conference has a talk on AI agents with a demo that works perfectly. Then you deploy it, and reality files a bug report. Here's what it takes to get from the demo to something you'd trust with your ERP.
By Anvexis Stack Engineering8 min read
A demo runs on the inputs its author picked. Production agents face everything else: odd documents, half-written emails, adversarial prompts, latency spikes, rate limits, and model updates that quietly change behaviour. None of that shows up on stage.
The fix is architectural, not a better prompt. Treat the language model as an unreliable component inside a reliable system, not as the system itself. The surrounding code decides what the model may see, what it may do, how its output is checked, and what happens when it gets something wrong.
Often, no. Anthropic draws a useful line between workflows, where language models and tools are orchestrated through predefined code paths, and agents, where the model decides its own stepsSource [1]. Its advice is to find the simplest solution possible and only increase complexity when neededSource [1].
In practice, many “agent” projects are better as a fixed workflow: classify the email, extract the fields, look up the customer, draft a reply. Each step is a model call with a narrow job, and your code decides the order. Workflows are cheaper, faster, and far easier to test. Reach for open-ended agents when the steps genuinely can't be known in advance, such as research across several systems, and accept the extra testing that comes with them.
Never parse free-form text to drive an action. Use function calling or schema-constrained output, then validate it in code before anything happens.
Log every prompt, completion, tool call, latency, and cost. You can't debug, or explain, what you can't see.
Set confidence thresholds and approval rules. When the model is unsure, or the action is expensive, route it to a person.
Build a red-team dataset before launch. Your users, and the occasional attacker, will find inputs you never imagined.
The most reliable agents we've built do one well-defined job. They classify incoming requests, prepare a draft, or gather information from a few systems. They don't “run customer service” or “manage purchasing”. Narrow scope makes behaviour predictable, evaluation possible, and failures easy to understand.
Write the job down as you would for a new employee: what triggers it, what information it may use, which actions it may take, what a good result looks like, and when it must ask a person. That description becomes the agent's instructions, the list of tools it gets, and the starting point for its evaluation set. If you can't write it in a page, the job is too big.
An agent is only as safe as the tools it can call. Each tool should do one thing, accept strictly typed arguments, validate them, and return clear results or clear errors. Anthropic recommends investing in the agent-computer interface through thorough tool documentation and testing, the same care you'd put into an API for human developersSource [1].
A tool called update_delivery_date that takes a date and an order reference is far safer than a generic execute_sql or call_api. OWASP's guidance on excessive agency makes the same point: offer only the extensions an agent needs, prefer granular tools over open-ended ones such as shell access, and enforce authorization in the downstream system rather than trusting the model to decide what's allowedSource [3].
result = llm.call(tools=[create_quote], input=email_text)
action = CreateQuote.model_validate(result.tool_args) # schema check
if action.total > APPROVAL_LIMIT or result.confidence < 0.8:
queue_for_review(action, reason="limit_or_low_confidence")
else:
erp.create_draft_quote(action) # drafts onlyGive each agent its own credentials with the minimum permissions its tools need, ideally acting in the context of the user it's working for. OWASP lists excessive functionality, excessive permissions, and excessive autonomy as the root causes of agents doing damage, and recommends limiting permissions to the minimum necessary, such as read-only database roles where writes aren't neededSource [3].
Irreversible actions, such as payments, deletions, and messages to customers, should require a person to approve them. Notice the snippet above only ever creates draft quotes. When something goes wrong, and eventually something will, scoped credentials limit the damage and make the audit trail clear.
OWASP ranks prompt injection first in its 2025 Top 10 for LLM applicationsSource [2]. The direct kind comes from a user typing instructions at your agent. The indirect kind is sneakier: instructions hidden in a website, document, or email the agent reads while doing its jobSource [2]. If your agent processes supplier invoices or customer emails, it reads text written by people outside your organization every day.
No prompt fully solves this. The mitigations OWASP lists are mostly architectural: constrain the model's role, define and validate output formats in deterministic code, enforce least privilege, require human approval for high-risk actions, clearly separate untrusted content from instructions, and run adversarial tests regularlySource [2]. Treat external content as data, never as instructions, and assume some injection attempts will get through the prompt layer. Your permissions are what stop them doing harm. Our AI security and governance work starts from exactly this assumption.
Without an evaluation set, every change to a prompt or model is a guess. Collect real examples of inputs with the expected outcome, including the awkward ones that tripped the agent up before. Run the whole set on every change and track how many pass.
Score more than the final answer. Check whether the agent called the right tools with the right arguments, how many steps it took, and how much it cost. Many regressions show up as extra steps or unnecessary tool calls before they show up as wrong answers. Add every production failure to the set once it's understood, so the same mistake can't come back unnoticed.
Agents will misunderstand inputs, call tools that time out, and occasionally loop. Production systems need step limits, timeouts, retries with backoff for transient errors, and a clean hand-off to a person when the agent can't finish. Anthropic notes that stopping conditions such as a maximum number of iterations are common for exactly this reason, alongside extensive testing in sandboxed environmentsSource [1].
Every hand-off should include the full trace, so the person can pick up where the agent stopped instead of starting over. A good hand-off is the difference between “the AI failed” and “the AI did the boring 90% and asked for help with the hard part”.
A trace of every step, tool call, input, and output is essential, but it only helps if people can read it. Make traces searchable by task, customer, and outcome, and show them in a form a support person can follow without reading raw JSON. When a customer asks why the agent did something, the answer should take a minute to find.
Keep logs for as long as your policies require, protect them like the data they contain, and review a sample regularly. Patterns in failures, such as the same tool erroring or the same type of request being escalated, point directly at the next improvement.
Agents can call a model many times for a single task: to plan, to choose a tool, to read the result, and to decide what's next. At a few tasks a day, nobody notices. At thousands, model costs and latency become real constraints, and OWASP even lists unbounded consumption as its own risk categorySource [4].
Measure cost and time per task during the pilot, and set hard budgets per task. Use smaller, cheaper models for simple steps such as classification, reserve larger models for reasoning that needs them, and cache results that don't change. A well-designed agent often costs far less than the first prototype.
Launch agents in a supervised mode where they propose and a person approves. This builds the evaluation set from real cases, shows where the agent struggles, and lets the team trust it gradually. It's the same pattern we use for human-reviewed document AI.
Increase autonomy one action at a time, starting with the safest and most frequent, and keep sampling automated decisions for review. If quality drops after a model or prompt change, the evaluation set and the sampling catch it quickly. Model choice matters less than most teams expect: a well-scoped job with good tools usually works with several models, and the evaluation set tells you which gives the best balance of quality, cost, and speed.
Written by Anvexis Stack
We build Odoo, AI, web, and cloud systems for growing companies, and we write about the patterns we use. This is engineering guidance, not a product guarantee: check vendor details for your version before you decide. Last reviewed .
LLM agents
Production LLM agents that call your tools and APIs with guardrails, permissions, audit logs, and human approval for risky actions.
AIAI security & governance
Deploy AI safely: private model hosting, data minimization, access controls, prompt-injection defenses, audit logs, and usage policies.
InsightsHuman-reviewed document AI
Design extraction workflows with validation rules, confidence handling, exception queues, and a path for reviewer corrections.
Next step
Thirty minutes with an engineer, no slide deck required. You'll leave with a clearer picture, whether or not we work together.