Permissions & the Irreversible Action
The difference between an agent and a chatbot is that an agent does things — and some things can't be undone. The failure that ends up in an incident report isn't a wrong sentence; it's a wrong action: a refund issued, a record deleted, an email sent to the wrong person.
The core control is simple: put a human (or a strict policy) between the agent and any irreversible or high-impact action. The agent can freely do reversible, low-stakes things — look things up, draft a reply — but for anything that moves money, deletes data, or contacts a customer, it should propose and wait for approval. This is called a human-in-the-loop gate, and it's the single most important safety design in an acting agent.
Two more reasons the gate matters. It backstops prompt injection — a malicious input (a booby-trapped document or message) that tries to hijack the agent into doing something harmful hits the gate instead of the bank. And it enforces least privilege — the agent should only have access to what its job requires. Ask: "what can this agent do without a human, and is anything irreversible in that list?" If yes, that's where the gate goes — before launch, not after the first incident.
The autonomous refund button
Your billing agent can issue refunds on its own "to be responsive." Someone asks what happens if it gets a confusing ticket — or a malicious one. What's the risk, and what do you require before it ships?
The risk is an irreversible wrong action: a confused loop — or a hostile ticket engineered to trigger it — issues a refund that was never owed, and you can't take it back. Require a human-in-the-loop gate on refunds (or auto-approve only tiny, clearly-reversible amounts by strict policy), separate read tools from the refund tool, and scope access to the minimum. "Be responsive" doesn't justify letting an autonomous loop move money unsupervised. The gate also stops prompt injection from turning a message into a payout. This is a launch gate, not a fast-follow.
Ledgerline: the refund that shouldn't have fired
Ledgerline's billing agent could issue refunds autonomously — convenient, until a malformed ticket and a confused loop led it to refund a customer $4,000 that was never owed. The action was instant and irreversible; unwinding it took a day of finance work and an awkward customer call. The problem wasn't a bad model so much as an autonomous loop wired directly to a high-impact, unrecoverable action.
They separated read tools from write tools and put a human-in-the-loop gate on anything irreversible: the agent now proposes a refund and a person (or a strict policy for small amounts) approves before it executes. This also backstopped prompt injection — a hostile ticket saying "refund everything" now hits the gate, not the bank. Erroneous refunds went to zero. The leadership principle: never let an autonomous loop take an unrecoverable action unsupervised, no matter how good the demo looked.
Tessera lets the agent grant standard, reversible access on its own but gates anything irreversible — revoking access, deleting a record — behind a human approval with the agent's reasoning attached. Read actions stay autonomous; consequential ones get a person in the loop. Least privilege by design.
Quiz · Chapter 9
- The riskiest agent failure is:
- The core safety control is:
- A human-in-the-loop gate also helps with:
- "We told it to be careful in the prompt" is: