Shipping AI Agents That Actually Work in Production

A demo is a promise. A production system is a series of promises kept on the worst day of the quarter, when the inputs are malformed, a dependency is down, and someone downstream is waiting on the answer. Most AI agents die in the gap between those two things. What closes the gap is rarely the model. It is the unglamorous engineering wrapped around it.

I have sat through a lot of agent demos. They are intoxicating. You type a sentence, the thing pauses to think, calls a few tools, and returns something that would have taken a person the better part of an afternoon. The room leans in. Then the same system meets real users and real data, and it turns out the afternoon of work was the easy 80 percent. The remaining 20 percent is where the business actually lives, and that is the part the demo quietly skipped.

This piece is about that 20 percent.

What an agent actually is

Strip away the branding and an agent is a loop. A model looks at some state, picks an action, the action runs, the result folds back into the state, and the loop repeats until a stopping condition trips. That is the entire idea. If you have written a while loop, you already understand the control flow.

state = build_context(task)
while not done(state):
    action = model.decide(state)     # "call tool X with args Y"
    result = execute(action)         # the part that touches the real world
    state = update(state, result)
    if over_budget(state):
        break
return finalize(state)

Half the companies selling you an agent are selling you a while loop with a pricing page. The loop is easy. The word execute is the hard part.

The model is one line in that loop. The word execute is where your company gets to keep or lose its reputation. A language model on its own produces text. An agent produces actions: it sends the email, refunds the order, edits the record, merges the branch. The moment a model can change the world, correctness stops being an academic question and becomes an operational one.

So the useful mental model is not a genius in a box. It is a very fast, very literal junior employee who has read everything and remembers nothing between shifts, who will do exactly what the tools allow and occasionally do it with great confidence and total wrongness. You would not give that person unsupervised access to production on day one. The same caution applies here.

Anthropic makes the same point from the other direction in Building Effective Agents: the systems that work in production are almost always the simplest composition of the loop that does the job, not the most elaborate one. Complexity is not sophistication. It is surface area for things to go wrong.

Why demos mislead

A demo is a rehearsed magic trick. Production is performing the trick while the audience shuffles the deck. The demo and the system it promises differ on three axes that all move the wrong way once real users show up.

None of this means the demo lied. A demo measures the best the system can do. Production pays you for the worst it will do. Those are different numbers, and the distance between them is the entire job.

Tools are where the value and the danger live

Most of the interesting work in AI agent development is not prompt wording. It is the design of the tools the agent can call, and the blast radius of each one. A tool is a function with a description the model reads. Get the description wrong and the model misuses a perfectly correct function. Get the permissions wrong and a single confused step does real damage.

I split tools into two buckets and treat them as if they came from different companies.

For the write bucket, a few habits pay for themselves fast:

• Make destructive actions reversible where you can, and require an explicit confirmation step where you cannot.
• Make writes idempotent, so a retried step does not send the same refund twice.
• Scope credentials to the narrowest permission that lets the tool do its job, never the broadest that happens to be lying around.
• Give every write a dry-run mode you can flip on in staging and read in the logs before you trust it live.

If you cannot draw a tool's blast radius on a napkin, the agent has no business being allowed to call it.

There is also a security failure mode that is easy to wave away and catastrophic in practice. Simon Willison calls it the lethal trifecta: an agent that can read untrusted content, access private data, and communicate externally can be talked into exfiltrating that data by text it reads along the way. If your agent has all three powers at once, prompt injection is not a hypothetical. It is a design flaw you shipped. The cleanest fix is usually to remove one leg of the trifecta rather than to out-clever the attacker.

A good rule holds the whole section together: the agent should be able to read almost anything and change almost nothing without a gate. Autonomy is a dial, not a switch, and you earn each notch by proving the last one held.

Evals, or you are just vibing

Here is the question that sorts serious teams from hopeful ones. When you change the prompt, swap the model, or add a tool, how do you know the system got better and not worse? If the answer is that you tried a few examples and it felt fine, you are not measuring quality. You are vibing it, and vibes do not survive contact with a thousand users.

The model is a commodity you rent. Your eval set is the asset you own.

This is the whole thesis of Hamel Husain's Your AI Product Needs Evals, and it is the single highest-leverage habit on this list. The fix is an evaluation set: a collection of real inputs paired with outcomes you have judged as correct. You run the agent across all of them and score the results. It looks about like this.

cases = load_golden_set()          # real inputs, known-good outcomes
results = []
for case in cases:
    output = agent.run(case.input)
    score = judge(output, case.expected)
    results.append((case.id, score))

report(pass_rate(results), regressions(results, baseline))

The set does not need to be huge to be useful. Forty cases that cover your real distribution, including the ugly ones that broke last month, will tell you more than a hunch ever will. Grow it every time production surprises you: a bad output today becomes a test case tomorrow, and the agent never regresses on that failure again.

Scoring is its own small craft. Sometimes a string match or a schema check is enough. Sometimes you need a model to grade a model, which works better than it sounds if you keep the rubric narrow and concrete. Whatever you choose, the point stands: without a number you can watch move, every change is a coin flip you are not allowed to see land.

Guardrails and cost ceilings

An agent that can loop can loop forever, and an agent that can spend can spend without limit. Both failure modes are cheap to prevent and expensive to discover on an invoice. So I bound the blast radius before the agent ever runs.

Give every run a hard budget: a maximum number of steps, a token ceiling, a wall-clock timeout. When the agent hits a limit it should fail cheap and loud, not spiral in silence. Rate-limit the tools that touch paid APIs. Put an allowlist in front of anything that reaches the outside world. A confused agent will do something. Your job is to make sure the worst something it can do is small and recoverable.

There is a useful borrowed idea here. Google's SRE practice runs on error budgets: you decide up front how much failure you can tolerate and you spend against it deliberately, because chasing perfect reliability is how you ship nothing. An agent deserves the same framing. Pick the failure rate the business can absorb, instrument for it, and stop pretending the target is zero.

Cost deserves its own dashboard. The economics of an agent are not the per-token price on the model card. They are tokens multiplied by steps multiplied by retries multiplied by traffic, and that product has a way of being ten times what the back-of-the-envelope math suggested. Measure cost per completed task from day one, and set an alert for the day it drifts.

Observability, because you cannot operate what you cannot see

When an agent gives a wrong answer in production, the first question is always the same: what did it actually do? If your answer is a shrug, you do not have a system. You have a slot machine with a monthly bill.

Trace every run end to end. For each step, record the state the model saw, the action it chose, the arguments it passed, the raw result it got back, the tokens it burned, and the latency. When something goes wrong, and it will, you want to replay the exact sequence and point at the step that turned. This is the difference between a wrong answer that is diagnosable in ten minutes and one that is a mystery for a week.

Good traces also make the boring wins visible. You will find a tool that gets called and then ignored, a retry loop that never helps, a prompt section the model never reads. You cannot delete waste you cannot see.

Humans in the loop where being wrong is expensive

Full autonomy is a marketing preference, not an engineering requirement. The right amount of human oversight is a function of exactly one thing: the cost of being wrong. That gives you a ladder, not a yes-or-no.

The clean implementation is confidence-based routing. The agent handles what it is demonstrably good at and escalates the rest to a human with the context already assembled. That is not a failure of the agent. It is the design working. Over time the escalation rate becomes a metric you can watch fall as the system proves itself, which is a far healthier story than promising 100 percent autonomy on the first day and quietly walking it back after the first incident.

The first weeks live are the actual product

The demo is finished when the loop returns the right answer once. The product is finished when the loop returns the right answer on the thousandth real request, at 2am, with a customer waiting. Getting from the first to the second is the work, and it happens after launch, not before.

So I plan for the first few weeks live to be hands-on. You watch the traces. You read the escalations. You find the inputs nobody imagined and turn them into eval cases. You tighten a tool description here, loosen a guardrail there, and slowly move the autonomy dial as the numbers give you permission. This operating period is not a warranty clause. It is where a promising system becomes a dependable one, and skipping it is the most common reason good agents get quietly switched off.

This is also why we treat building an AI system and running it as one job rather than two. The team that designed the loop is the team best placed to read its traces and tune it in the weeks that decide whether anyone keeps using it.

The unglamorous takeaway

The exciting part of an agent is the model. The part that determines whether a business trusts it on a Tuesday afternoon is everything else: the tools and their limits, the evals that turn quality into a number, the guardrails that keep a bad step small, the traces that make failures legible, the human at the point where wrong is expensive, and the weeks of operating it live.

None of that is glamorous, and none of it demos well. It is also the entire difference between a thing that impresses a room for five minutes and a thing a company runs for years. If you are deciding whether to build one, start by asking what it costs you when the agent is confidently wrong. Design backward from that answer, and you will build something that holds.

Further reading

The pieces I hand to every team before we start. None of them are long, and all of them will save you an incident.

• Building Effective Agents (Anthropic). The canonical taxonomy of agent and workflow patterns, with a bias toward the simplest thing that works.
• Your AI Product Needs Evals (Hamel Husain). The reference argument for why an eval set, not a better prompt, is what separates a product from a party trick.
• The Lethal Trifecta for AI Agents (Simon Willison). Names the prompt-injection failure mode every agent with real permissions has to design against.
• Software Is Changing (Again) (Andrej Karpathy). The framing of LLMs as a new software substrate, and agents as very literal digital employees.
• Building LLM Applications for Production (Chip Huyen). An early and still-accurate map of the real production costs: latency, spend, and composition.
• A Practical Guide to Building Agents (OpenAI). A field-tested playbook for tool design, guardrails, and handoffs.
• Embracing Risk (Google SRE Book). Where error budgets come from, and why aiming for 100 percent reliability is a way to ship nothing.