Interview Simulation

An agent is a model running in a loop: it's given tools, it picks one and calls it, it sees the result, and it decides whether to act again or stop. A single LLM call is one turn with no feedback. The agent-ness — and all the hard parts — come from that loop: each turn costs tokens, the loop can fail to terminate, and it can take a wrong action with real consequences. If a task can be done in one call or a fixed script, I wouldn't use an agent at all.

First a hard step/cost budget so it can never run unbounded — that's a safety floor, not a fix. Then I diagnose why: is the stopping condition ill-defined, so it never recognizes "done"? Is it repeating the same failing tool call because it can't see it already failed? Is the task underspecified? I'd add an explicit definition of done, make tool results legible in the context, detect repeated no-progress states, and then measure average steps across many tasks to confirm the fix rather than trusting one run.

Tools should be few, well-named, and hard to misuse. Each needs a clear description, typed parameters, and a result the model can actually read and act on — including good error messages, because "file not found: check the path" steers the next turn while a raw stack trace derails it. I prefer a small set of composable tools over dozens of overlapping ones, and I make destructive actions separate and gated. Bad tool design is one of the biggest hidden causes of flaky agents.

I build a set of tasks with checkable success conditions — did the test pass, did the row get written, did the answer match — and run the agent over all of them, reporting task-completion rate, average steps, and cost per task, with a sample size. I also evaluate the trajectory, not just the end: did it call the right tools, avoid destructive mistakes, and stop when done? A single successful demo run tells me almost nothing; the distribution over many tasks tells me whether it's shippable.

Rarely, and only when the work genuinely splits into independent sub-tasks or needs distinct tool sets or context that would blow one agent's window — e.g. a coordinator delegating parallel research threads. Multi-agent adds coordination cost, more failure surface, and harder debugging. I'd start with one well-instrumented agent, measure where it breaks, and only split when a single loop demonstrably can't handle the context or parallelism. Complexity has to be earned with evidence.

Irreversible or high-impact actions get a human-in-the-loop gate: the agent proposes the action and a person (or a strict policy) approves before it executes. I separate read tools from write/destructive tools, scope credentials to least privilege, and prefer reversible operations (soft delete, dry-run first). This also backstops prompt injection — if a malicious document tells the agent to delete everything, the gate stops it. You never let an autonomous loop take an unrecoverable action unsupervised.

Cost is tokens-per-turn times turns, so I attack both. Per turn: keep the context lean — don't resend the entire history every step, summarize or drop stale tool output, and use a smaller model for routing or simple sub-steps. Turns: a step budget and a clear stopping rule so it doesn't wander. Then I measure cost per completed task across the eval set, because a cheap-per-call agent that takes 30 steps can cost more than a pricier one that finishes in 5.

Tool failures are expected, not exceptional, so the loop has to survive them. I return a clear, actionable error into the context so the model can adapt, add bounded retries for transient failures (with backoff), and detect when it's stuck repeating the same failed call so it changes strategy or escalates instead of burning the budget. Unhandled, one flaky tool turns into an infinite retry loop; handled, it's just another observation the agent reasons about.

Real inputs are messier than my test tasks, so completion rate drops and I need real trajectories sampled back into evaluation. Tools flake and rate-limit, so retries and budgets become load-bearing. Cost and latency now matter per run. And I need observability — full traces of every step, tool call, and decision — plus online signals like completion rate, step distribution, and escalation rate, because a loop can fail in ways a single output never would.

Walk through your portfolio agent: the task (fix a failing test in a small repo), the failure you found (it kept re-running the same broken command because it couldn't see the last result), the fix (surfaced tool output cleanly and added no-progress detection), and the measured result (completion over 30 tasks went from 55% to 80%, average steps from 14 to 8). Concrete task, concrete failure, concrete fix, concrete numbers — that's the whole interview.