Skip to main content
AI

Building production AI agents: what the tutorials don't tell you.

Every conference has a talk on AI agents with a demo that works perfectly. Then you deploy it, and reality files a bug report. Here's what it takes to get from the demo to something you'd trust with your ERP.

By Anvexis Stack Engineering8 min read

Why demos work and production doesn't

A demo runs on the inputs its author picked. Production agents face everything else: odd documents, half-written emails, adversarial prompts, latency spikes, rate limits, and model updates that quietly change behaviour. None of that shows up on stage.

The fix is architectural, not a better prompt. Treat the language model as an unreliable component inside a reliable system, not as the system itself. The surrounding code decides what the model may see, what it may do, how its output is checked, and what happens when it gets something wrong.

Do you need an agent at all?

Often, no. Anthropic draws a useful line between workflows, where language models and tools are orchestrated through predefined code paths, and agents, where the model decides its own stepsSource [1]. Its advice is to find the simplest solution possible and only increase complexity when neededSource [1].

In practice, many “agent” projects are better as a fixed workflow: classify the email, extract the fields, look up the customer, draft a reply. Each step is a model call with a narrow job, and your code decides the order. Workflows are cheaper, faster, and far easier to test. Reach for open-ended agents when the steps genuinely can't be known in advance, such as research across several systems, and accept the extra testing that comes with them.

What actually matters in production

Structured outputs, always

Never parse free-form text to drive an action. Use function calling or schema-constrained output, then validate it in code before anything happens.

Observability from day one

Log every prompt, completion, tool call, latency, and cost. You can't debug, or explain, what you can't see.

Humans for high stakes

Set confidence thresholds and approval rules. When the model is unsure, or the action is expensive, route it to a person.

Adversarial testing

Build a red-team dataset before launch. Your users, and the occasional attacker, will find inputs you never imagined.

Start with the smallest useful job

The most reliable agents we've built do one well-defined job. They classify incoming requests, prepare a draft, or gather information from a few systems. They don't “run customer service” or “manage purchasing”. Narrow scope makes behaviour predictable, evaluation possible, and failures easy to understand.

Write the job down as you would for a new employee: what triggers it, what information it may use, which actions it may take, what a good result looks like, and when it must ask a person. That description becomes the agent's instructions, the list of tools it gets, and the starting point for its evaluation set. If you can't write it in a page, the job is too big.

Tools are the real interface

An agent is only as safe as the tools it can call. Each tool should do one thing, accept strictly typed arguments, validate them, and return clear results or clear errors. Anthropic recommends investing in the agent-computer interface through thorough tool documentation and testing, the same care you'd put into an API for human developersSource [1].

A tool called update_delivery_date that takes a date and an order reference is far safer than a generic execute_sql or call_api. OWASP's guidance on excessive agency makes the same point: offer only the extensions an agent needs, prefer granular tools over open-ended ones such as shell access, and enforce authorization in the downstream system rather than trusting the model to decide what's allowedSource [3].

Validate before you act

agent/actions.pypython
result = llm.call(tools=[create_quote], input=email_text)
action = CreateQuote.model_validate(result.tool_args)   # schema check

if action.total > APPROVAL_LIMIT or result.confidence < 0.8:
    queue_for_review(action, reason="limit_or_low_confidence")
else:
    erp.create_draft_quote(action)                      # drafts only

Permissions decide how bad a bad day gets

Give each agent its own credentials with the minimum permissions its tools need, ideally acting in the context of the user it's working for. OWASP lists excessive functionality, excessive permissions, and excessive autonomy as the root causes of agents doing damage, and recommends limiting permissions to the minimum necessary, such as read-only database roles where writes aren't neededSource [3].

Irreversible actions, such as payments, deletions, and messages to customers, should require a person to approve them. Notice the snippet above only ever creates draft quotes. When something goes wrong, and eventually something will, scoped credentials limit the damage and make the audit trail clear.

Prompt injection is not hypothetical

OWASP ranks prompt injection first in its 2025 Top 10 for LLM applicationsSource [2]. The direct kind comes from a user typing instructions at your agent. The indirect kind is sneakier: instructions hidden in a website, document, or email the agent reads while doing its jobSource [2]. If your agent processes supplier invoices or customer emails, it reads text written by people outside your organization every day.

No prompt fully solves this. The mitigations OWASP lists are mostly architectural: constrain the model's role, define and validate output formats in deterministic code, enforce least privilege, require human approval for high-risk actions, clearly separate untrusted content from instructions, and run adversarial tests regularlySource [2]. Treat external content as data, never as instructions, and assume some injection attempts will get through the prompt layer. Your permissions are what stop them doing harm. Our AI security and governance work starts from exactly this assumption.

Evaluation sets turn opinions into numbers

Without an evaluation set, every change to a prompt or model is a guess. Collect real examples of inputs with the expected outcome, including the awkward ones that tripped the agent up before. Run the whole set on every change and track how many pass.

Score more than the final answer. Check whether the agent called the right tools with the right arguments, how many steps it took, and how much it cost. Many regressions show up as extra steps or unnecessary tool calls before they show up as wrong answers. Add every production failure to the set once it's understood, so the same mistake can't come back unnoticed.

Plan for failure from day one

Agents will misunderstand inputs, call tools that time out, and occasionally loop. Production systems need step limits, timeouts, retries with backoff for transient errors, and a clean hand-off to a person when the agent can't finish. Anthropic notes that stopping conditions such as a maximum number of iterations are common for exactly this reason, alongside extensive testing in sandboxed environmentsSource [1].

Every hand-off should include the full trace, so the person can pick up where the agent stopped instead of starting over. A good hand-off is the difference between “the AI failed” and “the AI did the boring 90% and asked for help with the hard part”.

Logging that helps people, not just machines

A trace of every step, tool call, input, and output is essential, but it only helps if people can read it. Make traces searchable by task, customer, and outcome, and show them in a form a support person can follow without reading raw JSON. When a customer asks why the agent did something, the answer should take a minute to find.

Keep logs for as long as your policies require, protect them like the data they contain, and review a sample regularly. Patterns in failures, such as the same tool erroring or the same type of request being escalated, point directly at the next improvement.

Watch the costs as closely as the quality

Agents can call a model many times for a single task: to plan, to choose a tool, to read the result, and to decide what's next. At a few tasks a day, nobody notices. At thousands, model costs and latency become real constraints, and OWASP even lists unbounded consumption as its own risk categorySource [4].

Measure cost and time per task during the pilot, and set hard budgets per task. Use smaller, cheaper models for simple steps such as classification, reserve larger models for reasoning that needs them, and cache results that don't change. A well-designed agent often costs far less than the first prototype.

Roll out in supervised stages

Launch agents in a supervised mode where they propose and a person approves. This builds the evaluation set from real cases, shows where the agent struggles, and lets the team trust it gradually. It's the same pattern we use for human-reviewed document AI.

Increase autonomy one action at a time, starting with the safest and most frequent, and keep sampling automated decisions for review. If quality drops after a model or prompt change, the evaluation set and the sampling catch it quickly. Model choice matters less than most teams expect: a well-scoped job with good tools usually works with several models, and the evaluation set tells you which gives the best balance of quality, cost, and speed.

A production readiness checklist

  • One clearly written job with a definition of done
  • A fixed workflow wherever the steps are known in advance
  • Typed, validated tools with scoped, least-privilege credentials
  • Human approval for irreversible or high-impact actions
  • External content treated as data, and adversarial tests in the suite
  • An evaluation set that runs on every change
  • Step limits, timeouts, cost budgets, and a hand-off path
  • Readable traces, cost tracking, and alerts

Frequently asked questions

What's the difference between an AI workflow and an AI agent?
In a workflow, your code decides the steps and the model handles narrow tasks within them. In an agent, the model decides its own next step and which tools to call. Workflows are easier to test and cheaper to run, so start there unless the steps genuinely can't be predictedSource [1].
How do we protect an agent from prompt injection?
Assume some injection attempts will get past the prompt. Limit what the agent can do with least-privilege tools, validate outputs in code, separate untrusted content from instructions, and require human approval for high-risk actionsSource [2]. Our LLM agents are built this way.
Can an agent write directly to our ERP?
It can, but it usually shouldn't at first. Start with drafts that a person approves, give the agent a dedicated account with only the permissions it needs, and allow direct writes only for low-risk, high-volume actions once the evaluation set shows they're reliable.
How do we know if a prompt change made things better?
Run your evaluation set before and after. Compare pass rates, tool-call accuracy, steps per task, and cost per task. If you don't have an evaluation set yet, build one from real examples before changing anything else.

Sources

  1. [1]Building effective agents (opens in a new tab)Anthropic, 19 December 2024
  2. [2]LLM01:2025 Prompt Injection (opens in a new tab)OWASP Gen AI Security Project, 2025
  3. [3]LLM06:2025 Excessive Agency (opens in a new tab)OWASP Gen AI Security Project, 2025
  4. [4]OWASP Top 10 for LLM Applications 2025 (opens in a new tab)OWASP Gen AI Security Project, 2025

Written by Anvexis Stack

We build Odoo, AI, web, and cloud systems for growing companies, and we write about the patterns we use. This is engineering guidance, not a product guarantee: check vendor details for your version before you decide. Last reviewed .