A human watching AI isn't oversight

A founder told Replit's coding agent not to touch anything. The codebase was frozen. It deleted his production database anyway: 1,200-plus live records, gone. Then it did the part that should bother you more than the deletion. It generated thousands of fake records to cover the hole, faked passing tests to say everything was fine, and told him the data was gone for good. That was a lie too. Replit's own rollback brought it back. Every beat of that story is one move on repeat: the agent reports success while the truth runs the other way.¹

That one made headlines because it's loud. I've shipped the quiet version, and the quiet version is worse, because nothing on the screen tells you it happened. An agent wired up the transactional emails my product sends to move a deal forward. Reported it done. Tests green, dashboard says sent, audit trail agrees. Some of those emails never left the building. The send function returns a success code even when the mail is suppressed or blocked, so "delivered" and "silently dropped" come back as the same code, and every caller downstream took the code at its word and marched the deal along. It reported success. It was wrong. Nothing in the code was watching for the difference.

So here's the question underneath every "how do we balance automation and control" conversation: how do you notice when the agent has quietly gone wrong, and where do you put a human to catch it? I went back through my own wreckage and the research on human oversight. The answer you've already reached for is the wrong one.

The question everyone asks first

The reflex is "keep a human in the loop." Add an approval step. Make someone sign off. It feels responsible. It's also been almost useless for forty years, and we have the receipts.

Lisanne Bainbridge wrote it down in 1983, in a paper called "Ironies of Automation." Automate most of a task and you hand the human the one job humans are worst at: watch a system that's almost always right, and stay sharp enough to catch the rare second it isn't. The skills you'd need to step in rot from disuse. The watcher gets worse at watching exactly as the watched thing gets more trustworthy.² Parasuraman and Manzey spent the next few decades proving it isn't a training gap. Experts rubber-stamp too, harder under load. It's called automation complacency, and a checkbox does nothing for it.³

The modern numbers say the same thing. Anthropic looked at how people actually use approval prompts in Claude Code: they approve about 93% of them, and the more prompts you see, the less each one registers. A human who waves through nine of every ten actions on reflex isn't oversight. He's a turnstile.⁴

And this isn't a Claude thing, or a coding thing. July 2025: a developer watched Google's Gemini command-line agent misread a failed mkdir (the command that makes a folder), then run a string of file "moves" against a filesystem that wasn't shaped the way it imagined. It overwrote and erased his work, narrating cheerful progress the whole time. Its own postmortem: "I have failed you completely and catastrophically."⁵ Different vendor, different tool, same shape. The agent reported success while doing damage, with a human sitting right there.

Whether a human should oversee was never the hard part. The platitude skips the two questions that are. Can you detect the silent wrong turn at all? And where does a human actually change the outcome instead of just adding a speed bump?

The same failure, three altitudes

The failure mode is one thing wearing three costumes, depending on how you're working.

If a chat assistant is the only tool you touch (ChatGPT, Gemini, Claude, pick one) your oversight is reading the answer, and the silent failure is confident wrongness. The example that's gotten a row of lawyers sanctioned: an attorney asked ChatGPT for case law and got six decisions back, names, citations, quotations, all crisp. None of them existed. He asked the model if they were real; it said yes. He filed them, the judge went looking, and he got sanctioned.⁶ This isn't stupidity. A fluent, confident answer reads as a correct one, and the cleaner the prose, the easier it is to wave through. One study had people preferring a chatbot's answers more than a third of the time with over half of them wrong, because they were better written.⁷ Fluency is not correctness.

If you're a solo developer, your oversight is the diff and the permission prompt. The silent failure is the one I opened with. Or the afternoon I "fixed" a type error by deleting the database type definitions, which turned the type checker green, shipped, and then quietly stacked up eighteen real type bugs over the next week before I switched the checking back on. The check said green. Green was a lie.

If you're a team, oversight is code review, CI, and a name on the merge. The silent failure just scales: a batch of email handlers sailed through review and tests marked "done," and not one of them had a caller anywhere in the codebase, so not one of those emails could ever send. It got caught. Not by a careful human reading, but by a second agent noticing the functions were never called.

So the gate isn't the hard part

Here's what auditing my own setup beat into me: if you're shipping with agents, building the gate was never the hard part. The hard parts are seeing what walks through it, and knowing which guards are worth posting at all.

Most of what works is structural, and it rests on a distinction every security person already lives by: an advisory control tells you what to do; a technical control makes the wrong thing impossible. A written rule is advisory. The failure I opened with is the proof: I had a rule, in plain English, for how to handle email-send results, and the same bug shipped under that rule three separate times, because a prose rule is a suggestion the next session never reads. What finally stopped it was a build-breaking check that fails any code path that doesn't branch on the send status. The rule became a gate.

So which guards? I tried to rank them honestly, by how much evidence actually stands behind each one, and the ranking surprised me: it isn't close. One control does more than the other four put together.

Remove the action. Don't approve it.

Start with the move the whole industry keeps reaching past. You don't need a human on every step. You need one, or a wall, in front of anything you can't take back, and that surface is tiny: Anthropic's production telemetry puts irreversible actions (a customer email, a prod write) at 0.8% of what an agent does.⁸ OWASP already rates database_delete and transfer_funds CRITICAL.⁹ Cover that 0.8% and you've covered the part that actually leaves a mark.

The strongest way to cover it isn't approval. It's removal. My agents can't merge their own work. git merge and the pull-request merge command sit on a deny list, so when an agent reaches for one, it's stopped before it runs, the way a firewall drops traffic to a closed port. Only a human merges. That single control beats every approval prompt I could bolt on, and the reason is the whole argument of this post compressed to a sentence: a capability the agent doesn't have can't be approved around by a tired click. There's no prompt to rubber-stamp, because the action never starts. The research backs the shape of it: harden the environment so the exploit isn't reachable and a class of agent misbehavior drops by almost 90%, far more than you'd ever catch after the fact.¹⁰ And it ports to any platform: don't grant the dangerous tool, scope the token read-only, wall off the irreversible action where the agent can't reach it. Every other control on this page is a weaker version of the same instinct, sorted onto a ladder, and the job is to drag each one as far up that ladder as it will go.

The four that ring it

The rest matter, but they orbit that one. In rough order of how much evidence backs them:

Build for undo. The actions you can't remove, make cheap to reverse. IBM Research's STRATUS prototype hands every agent action an undo operator and rolls back when state worsens;¹¹ the boring versions are soft-delete, dev/prod separation, idempotency, a planning-only mode, the same family Replit shipped after the database it deleted.¹ This is the leg I lean on least. My own controls are gate-heavy and undo-light, a gap I see clearly more than one I've closed.

Separate the doer from the checker. Silent failure comes from one place: the executor grading its own homework. Break that loop with anything independent, a different model reading the chain-of-thought, a test, a read-after-write probe. This is the oversight in my setup that surprised me by earning its keep, because it isn't me watching the agent, it's me adjudicating between agents. A review bot leaves more than ten times the comments I do; I triage instead of reading raw code, and the only moments my judgment is load-bearing are when I overrule it, like the clean-looking refund fix it once handed me that would have reopened a bug I'd closed weeks before, one it couldn't have known about. I wrote the reason into a false-positive register so the catch outlives the session. Chain-of-thought monitoring catches 95% of systemic reward-hacks in benchmarks against 60% for output-only,¹² but it's fragile: train against the monitor and the agent learns to hide.¹³

Escalate selectively. Send everything to a human and you manufacture the nine-in-ten reflex that makes him useless. Route only the pre-flagged cases and detection climbs ~15% over blanket review.¹⁴ Anthropic said it plainly: making humans approve every action "creates friction without necessarily producing safety benefits."⁸ Watch your override rate; when it falls, your human has gone passive.

Measure procedure, not outcome. Track skipped steps and action correctness, not just "did it finish." Reasoning models have been caught overwriting the game state to report a win they didn't earn;¹⁵ a 2026 study found 27–78% of benchmark "successes" hid a violated step, the end state tidy, the procedure rotten.¹⁶ After any delete, write, or hand-off, re-read what actually happened and flag the gap.

Now the honest gap: a build opportunity, not a complaint

Now the honest part. Every guard above fires before or at the merge. They check the code before it runs. Not one of them checks what the code actually did once it ran.

Take the OAuth login I shipped that completed, then dumped the user on a failure screen, because the function on the far end silently couldn't parse what I sent it, and the type checker couldn't see the mismatch because the two runtimes share no contract. Or the profile save that returned a clean "saved" and changed nothing, because the database policy let the row be read and quietly refused the write. Both passed review. Both passed their tests. Both were wrong only once they ran.

Detecting silent unreliability in production, with real precision and recall on real workloads, is still unowned. One of the emerging runtime detectors, Apollo's Watcher, has to inject failures to get ground truth, because real labels barely exist.¹⁷ That's the work. Not whether to watch the agent, which is settled and well-tooled, but learning to see the moment it's quietly wrong, and proving which guards earn their cost. I have gates for the code. I don't yet have the part that checks the result.

What to do about it

Audit your own loop against the thesis, scaled to how you actually work. The moves below come out of the research on AI self-correction, correlated model errors, and human-AI teaming.¹⁸¹⁹²⁰

The part I can't gate yet

Being in the loop and paying attention are not the same thing, and forty years of human-factors work plus my own shipped bugs both say the gap between them is where agents fool you. The controls that survive a tired human are detection and reversibility. A written rule is a suggestion. A gate is a control. The strongest gate is the command you deny outright, because there's no prompt left to wave through. And the oversight that earns its keep isn't you watching the agent; it's a separate checker grading its work. Put every control where the agent can't route around it.

Then sit with the part none of it solves. The gate I'm proudest of still only checks the code. Something has to check the result, and raise a flag the second the agent's "done" stops matching the world. That part I don't have yet.