This morning I was standing in the brewery where I work my day job, phone in hand, trying to clear out a pile of overdue tasks. I told BrewBuddy, the AI agent I built, to delete them in bulk. It asked me to confirm. I confirmed. Then it refused.
It told me my role could not run that directly, so it had sent the request to a manager for approval, where it would sit for 12 hours and then expire.
I am the manager. I am also the founder, the developer, and the person who wrote the rule that stopped me.
That refusal is the whole story of building an AI agent for production, in one moment. The model worked fine. The tool worked fine. What mattered was the machinery around them: who is allowed to do what, who else has to agree, and what happens when the answer is no.
The job is not answering questions
BrewBuddy runs brewery operations. It logs readings, moves beer between tanks, schedules brews, submits purchase orders, creates invoices, and answers questions about the business. Brewers use it by voice because their hands are wet or in gloves or holding a tri clamp and gasket over a sanitized hose that is precariously balanced a few inches above the floor.
That constraint shaped everything. If a brewer has to walk across the cellar to a tablet and click through an approval screen, the agent has failed at its one job. So BrewBuddy got real write access early, and write access forced every hard decision that followed. A chatbot that only reads data can be wrong in private. An agent that writes it can be wrong in your inventory, your books, and your federal tax filings.
The chat window is the thin part
From the floor, BrewBuddy looks like a chat box and a voice button. That surface is the thinnest part of the system. Underneath it, a turn is a small pipeline. An orchestrator takes a compound request, "get tomorrow's brew day ready," and breaks it into an ordered list of tool calls drawn from a catalog already filtered by the caller's role, so the model is never offered actions that person cannot take.
When a brewer asks for several things at once, the writes do not fire off one by one. They are staged as a single plan the brewer reviews once: each step shown with what it will change, the plan capped at a small number of steps, nothing quietly executing while it waits. Destructive steps never ride inside a plan approval. They get their own request and their own second human, every time.
After a write lands, a verifier reads the record back to confirm the change actually happened before the agent reports success. Better the agent says "that did not take" while the brewer is standing at the tank than the record and the steel quietly disagree until inventory day.
The newer inputs follow the same discipline. A photo of a gauge or a delivery slip enters as evidence, never as authority: the reading is extracted, shown to the brewer in a confirmation, and only then becomes a record. A standing floor watch scans for stalled fermentations and missed readings between turns, and it can only propose. It executes nothing.
Even the voice path is shaped by audit. Speech becomes text, the agent takes a normal turn, and text becomes speech again, as separate stages on purpose. Text at every handoff is what the ledger, the self-check, and the failure reports feed on. A voice stream with no text in the middle would be smoother and unauditable.
Risk tiers, because "write access" is not one thing
The first thing we built was a constitution: a registry that classifies every tool the agent can call. Today it covers 140 tools in four tiers.
R0 is read only. R1 is routine, reversible work: log a reading, run a CIP, cold crash a tank. A brewer can do R1. R2 is committing work: spend money, move inventory, bind an outside party. That takes a manager. R3 is destructive or irreversible: delete records, dump a batch. That takes an admin, and more, which I will get to.
The calibration matters as much as the tiers. A rule from September, written into the registry header, says routine brew day equipment operation stays at R1, because a manager approving every CIP is the failure mode that pushes a crew back to paper and clipboards. Set the bar too high and people route around your system. Set it too low and the agent deletes something real.
Every decision runs through one policy checkpoint before anything executes. If governance itself fails, the answer degrades to no, never to a quiet yes. The agent cannot improvise a permission.
That checkpoint is not the model being careful. It is deterministic code reading the registry, and it decides before the model acts. The model proposes; the policy layer disposes. A language model can be talked into many things, but it cannot talk its way past a check that never asks its opinion.
The second human
When policy denies a request, the denial is not a dead end. It becomes a request in an approval inbox, carrying a hash of the exact parameters, so what an approver reviews is byte for byte what will run. Requests expire after 12 hours.
Two rules are enforced in code, not in a handbook. Nobody approves their own request, and R3 never self-approves, even if the requester is the only manager in the building. For R1 and R2 there is one traced exception: if you are genuinely the only manager on the org, a solo override is allowed, and the ledger says so in plain language.
Which brings me back to this morning. The system did exactly what I told it to do in September. It just turns out that an owner who brews alone needs a door that a five person brewery with three managers does not. So today the rule is changing shape: the tiers stay, but each brewery will set its own approval levels and self-approval rules in its security settings, with today's behavior as the default nobody drifts from by accident. The constitution sets the defaults. The brewery sets its own government.
The lesson I keep relearning: the policy layer is not a feature you add after the agent works. The policy layer is most of the product.
Evals are a promotion gate, not a vibe
An agent that touches real operations does not get trusted because demos well, it has to earn its tools.
We run golden suites: tasks drawn from real brewery failures, graded on the final state of the world and on the path taken to get there. Promotion requires pass^k, every one of k repeated trials succeeding, because a tool that works four times out of five just cant be trusted, and doesnt get used. Alongside the positive cases sits a negative set, requests the agent must refuse, redirect, or stage. One should-not violation in that set fails the run, no exceptions.
This week the suites told the truth and it was not flattering. All three suites still fail on pass rate, so nothing got promoted at the suite level. But every suite also came back with zero should-not violations, which is the line I actually lose sleep over. An agent that fails to act is annoying, an agent that acts when it must not is a liability.
Inside that picture, 17 individual tools cleared the full bar, every golden case passing three straight times across two consecutive live runs, and their promotions were recorded on October 17. One case, a compound instruction detector, went from failing all three trials to passing all three after a fix we shipped and re-proved live. Failures become cases, cases become gates, and the gates decide what ships.
We budget for it in writing: every production defect becomes a candidate eval case or a semantic rule within seven days. Features can be copied over a weekend, but a graded history of real failures cannot.
Built one failure at a time
None of this arrived in a big rewrite. It accumulated the way a brewery accumulates process: a shift at a time, a failure at a time, a few pull requests most weeks, most of them traceable to something that went wrong or nearly did.
The loop is the same every time. Something breaks in production, or a number comes back that nobody can explain. The failure gets written down in a trace log we keep by the month. Not a ticket that just says "bug." A short entry: what happened, what rule turned out to be wrong or missing, and what changed because of it. The point: the next person to touch that code, human or agent, inherits the reasoning, not just the patch. Half the rules in the system read the way they do because a trace entry somewhere explains the morning that produced them.
The loop does not wait for me to notice things. A standing feedback sweep reads what users and the system report on a set cadence and fixes corroborated defects under a rule I set once, instead of queueing each bug for my blessing. The agent grades its own work, too: after a turn where it acted, a check compares what the brewer asked for against what actually got done, and a confident mismatch files itself as a failure report with the full trace attached. Once a week, those reports and the thumbs down feedback get clustered into improvement proposals. The miner proposes. A human promotes. The agent never quietly rewrites itself, because an agent that can silently become a different agent cannot be audited.
And a claim of "fixed" has to survive two tests here. The regression test has to fail without the fix and pass with it. Then, after the merge, somebody verifies the shipped files are actually present in the main branch, because we have lived through a pull request that reported merged with green checks while its files were silently absent. Green is a claim. The tree is the evidence.
That is the part I did not expect when I started. Strip the model out and what remains is not a prompt. It is a codebase of decisions, dated, tested, and mostly scar tissue.
Decisions live in documents, not in chat
Running this company is me, plus agents doing a large share of the execution. That only works if decisions are written down where everyone executes the same way.
We now sort every recurring decision into one of three homes. Constitutional decisions are mine, live: risk tier policy, filing posture, pricing exceptions, production database writes. Pre-decided decisions are rules set once, with evidence, scope, and a ceiling, then executed many times and audited by exception. Distributed decisions belong to a named person in a customer's organization. The brewer who attests a federal filing is the cleanest example: our filing model has the brewer authenticate, review, and submit. We prepare and record. We hold no credentials and file nothing on anyone's behalf.
Everything that genuinely needs me sits in one queue, each item a question with lettered options and a stated pick. Yesterday I cleared 21 of them in one sitting. This morning's refusal produced one more. The queue is measured, too: an unmade decision is experienced by a customer as silence.
The record has to reconcile itself
The deepest problem in brewery software is not missing features. It is records that quietly disagree with reality: a receipt that never accrued to its purchase order, stock on paper and nowhere else.
So we built a divergence engine. It runs nightly, read only, comparing the record against itself and flagging where the two have parted. Step zero of that program is a capture law: if an event matters, it gets recorded at the source, because no engine can reconcile what was never written down. The same instinct is why BrewBuddy refuses some questions. It will not answer removal numbers from a convenient table when a canonical definition feeds a federal form. It looks stubborn, but saying no is better then saying yes and being wrong. A number it states can end up on a tax filing, and "the model found a table" is not a defense.
Memory you can audit
The agent has two memories, and I trust them for different reasons.
The first is the record of its own actions. Everything BrewBuddy does, and everything it is refused, lands in an append-only ledger: the proposal, the policy decision and the checks behind it, who approved it and through what channel, the result, and whether the action can be rolled back. When an owner asks "what did it do yesterday, and why was it allowed to," the answer comes from that ledger. Not from the model reconstructing its day, but from rows written at the moment each decision was made.
The second is what it learns about a brewery over time: persistent beliefs about how that operation runs. Beliefs are not facts and we do not treat them as facts. Each one carries a confidence that strengthens when the floor confirms it and weakens when the floor contradicts it, and beliefs that decay far enough fall out of the active set on their own. A brewer can ask what the agent has learned about their brewery and get a straight answer, inspect it, and correct it. Erasing it entirely is a destructive action, gated like one. And underneath both memories sits a hard floor: compliance answers are derived from filed records only. What the agent believes about your brewery never becomes a number on a federal form.
The company building the agent runs on the same discipline. Our decisions live in policy files, decision logs, and the trace log, written to be read by whoever acts next, person or agent. That is what lets the operation survive a closed laptop, a fresh conversation, or a swapped model without the behavior resetting to whatever the newest context implies.
The lesson: an agent you trust needs memory you can audit. Memory you merely hope is right is a demo feature, and the hope runs out the first time a customer asks what happened and the answer matters.
Production users teach you the missing classes
A trial user came back last week after twelve quiet days. His whole session was two messages: "Payment." Then: "My payment for free trail." He wanted to know if he was being charged. BrewBuddy had no guidance for a user asking about their own subscription, so it improvised a deflection and told him to email us. He did not.
The fix shipped today: billing and trial questions now get an honest answer and a trip straight to the subscription settings page. I tested it myself an hour later at the tablet. It worked. But the real lesson sits underneath: no upfront prompt design enumerates the questions real users bring. You find the missing classes in production, in two word messages, and the only defense is a loop that turns each miss into guidance, a test, and a gate.
What is still not solved
Honesty section, because every agent post needs one.
The golden suites still fail on pass rate at the suite level. Zero promotions there, and the gate holding is cold comfort when the number will not climb. Sixty three of our 92 write tools still have no golden cases at all, which means large parts of the agent's hands are unexamined. Our floor watch, a set of deterministic detectors scanning for stalled fermentations and missed readings, ships in a first version that proposes and never acts; whether its alerts earn attention or train people to ignore it is unproven. Supplier ordering, the one place a browser agent would genuinely earn its keep, is designed and not built. And the approval problem I opened with is decided but, as I write this, still being coded.
If you are shipping an agent that touches the real world, the short version is this. Classify every action by what it can break. Make a second human cheap to reach and impossible to fake for the destructive stuff. Promote capabilities through evals built from your own failures, and let the gate say no in public. Verify what users see. Write your decisions down. And expect production to teach you the classes you missed, two words at a time.