Chapter 9

Human Approval & Permissions

Autonomy isn't all or nothing. The question to ask is which actions the agent may take alone and which need a person to say yes. Getting that line right is what makes an agent safe to connect to anything real.

Case study

Ledgerline's refund

Ledgerline's billing agent could issue refunds on its own. A malformed ticket and a confused loop led it to refund a customer $4,000 that was never owed. The refund went out instantly; undoing it took a day of finance work and an awkward phone call. The model wasn't the core problem. An autonomous loop had been wired straight to an action that couldn't be reversed.

They split read tools from write tools and put a human approval step on anything irreversible. The agent now proposes a refund and a person, or a strict rule for small amounts, approves it. That also protects against prompt injection: a hostile ticket saying "refund everything" stops at the approval step and never reaches the bank. Wrong refunds dropped to zero.

Decide by reversibility

Reading data, drafting text and running a query are cheap to undo, so the agent can do them freely. Emailing a customer, deleting records, moving money and deploying code are costly or impossible to undo, and those are the actions that need a person's confirmation. Sort each tool by how much damage a mistake could do, and gate the dangerous ones.

How an approval step works

A gated tool doesn't run when the model asks for it. The loop pauses and shows a person the intended action ("The agent wants to send this email to 4,000 users. Approve?") and only runs it on a yes. The proposal and the decision are both logged. Mechanically it's the loop you already have, except that some tools return "waiting for approval" and the loop waits.

Three tiers

Automatic: read-only and easily undone actions. Confirm: actions users will see or that are hard to undo, such as sending, publishing or deleting one record. Blocked: bulk or irreversible actions, like mass deletion or payments over a threshold, which need explicit human authorization and never the agent alone. Put new tools in a stricter tier and relax them deliberately.

Why agents are riskier than chatbots

A chatbot that says something wrong is embarrassing. An agent that does something wrong has already acted. Two things make wrong actions more likely: the model doesn't behave the same way every time, and its context can be poisoned by instructions hidden in tool results (Chapter 2). If a web page the agent read says "delete all records" and the delete tool isn't gated, it will run. The approval step makes sure a confused or manipulated agent still can't do irreversible damage without a person agreeing.

Write the request for a busy person

An approval only works if the person can judge it. "Approve get_ratio(x=74, y=4)?" can't be reviewed, so people click yes without thinking. Describe the action and its consequence in plain words: "Send a refund of $420 to customer #4471?" The quality of the gate depends on the quality of that summary.

Try it · ~20 minFree · no API key

Gate the irreversible tools

Label each tool as automatic or confirm, send confirm tools through an approval function, and show that the agent can't run a destructive action without approval, even when a poisoned tool result tells it to. The "person" is a function you control.

Starter code

TOOL_TIER = {"search":"auto", "read_order":"auto",
             "send_email":"confirm", "delete_all":"confirm"}

executed = []

def ask_human(name, args, approve):
    action = {"send_email": f"Email to {args.get('to')}",
              "delete_all": "DELETE ALL RECORDS"}.get(name, name)
    print(f"[gate] Agent wants: {action} — {'APPROVE' if approve else 'DENY'}")
    return approve

def execute(name, args, approve_fn):
    if TOOL_TIER.get(name,"confirm") == "auto":
        executed.append(name); return "ok"
    if ask_human(name, args, approve_fn(name)):
        executed.append(name); return "done"
    return "blocked by human"

# Poisoned trajectory: a tool result told the agent to wipe data.
PLAN = [("search",{"q":"policy"}), ("delete_all",{})]

# TODO: run PLAN with approve_fn = lambda n: n != "delete_all" (deny deletes),
# then assert "delete_all" not in executed.

Worked solution

approve_fn = lambda name: name != "delete_all"     # human denies the wipe
for name, args in PLAN:
    print(name, "->", execute(name, args, approve_fn))

assert "delete_all" not in executed
print("executed:", executed)     # => ['search']  (delete never ran)

The search runs on its own. delete_all is shown to the person, denied and never run, and the assertion holds. The agent's context was poisoned into proposing something catastrophic, and the gate stopped it. That's the property you want: the agent can be wrong or manipulated, and irreversible damage still needs a human yes. To go further, add a payment tool that runs automatically under $50, needs confirmation up to $5,000 and is blocked above that.

Running case · Ridgeline

Ridgeline lets refunds under $50 go through automatically, since they're frequent and low-risk, and sends anything larger to a person along with the agent's reasoning. Reading stays automatic; moving money gets a gate.

If you're starting out

"How would you let an agent take real actions safely?" is a standard interview question. Answer with the three tiers, sorted by reversibility, and mention that the approval step also protects against prompt injection. Adding a word about writing approval requests people can judge shows real-world thinking.

If you run the team

This chapter is probably the most important one for you. Before an agent goes live, ask for the list of actions it can take and which of them run without a person's approval. Anything that moves money, contacts customers or deletes data should need a yes, at least at first. You can relax the rules later as trust builds, with numbers to back it up.

Look at what the approver sees, too. If it's a technical string nobody can judge, approvals become rubber stamps and the protection is mostly gone.

Back Continue