Microsoft ran 507 business tasks 20 times each and found that most failed agent runs looked perfectly fine from the outside. Of 79,853 attempts that failed their checks, 67.24% ended cleanly, called a tool that changed data, and reported no error at all. That single number is the clearest statement of the AI agent reliability problem I have read this year, because it describes exactly the failure I cannot see when Claude Code tells me a job is done. The agent says “finished.” The files, the rows, the records may say something else.
In this article
- Why a clean agent run is not evidence of a correct one
- What ThinkingBox measures that other benchmarks skip
- Why AI agent reliability means passing 20 times, not once
- How the cost of a dependable task reorders the price list
- Five AI agent reliability checks I run on my own scripts
A Clean Agent Run Is a Claim, Not Proof
A finished agent run proves only that the agent stopped. That is the core argument of Microsoft’s ThinkingBox write-up on Hugging Face, published October 3, 2026, and it puts the line more bluntly than I would: “A trajectory is a claim. Database state is the evidence. Repetition is the trust test.”
The example they lead with is a customer support case. The agent made nine well-formed tool calls and read the refund policy correctly. It then closed the ticket as “resolved” when the correct end state was “on hold,” because a carrier exception was still open. It also never answered the customer’s actual question. Every log line looked healthy, and the result was still wrong.
The failure breakdown explains why this is so easy to miss. Among the failed attempts in that 121,680-trial ablation across 12 models:
- 77.61% had wrong field values
- 43.30% had unintended extra effects
- 25.36% were missing required effects
Those categories overlap, so one bad run can do all three. None of them shows up in a transcript that ends with a polite summary.
AI Agent Reliability Is Measured in the Database, Not the Chat Log
ThinkingBox grades AI agent reliability on the final state of the backend, not on what the agent said. Each attempt gets an isolated MCP session with freshly initialized data. A simulated user holds private context and only releases it when asked. Then a side-effect extractor works out what actually changed, and deterministic judges compare that against the required end state.
That design choice is the useful part for anyone who cares about AI agent reliability outside a lab. Of the 507 tasks, 477 are graded on state alone, with only 30 also using a response rubric. The agent never sees the golden state or the assertions. It cannot talk its way to a pass.
I have not run ThinkingBox, and I am not going to pretend the benchmark maps neatly onto a one-person consultancy. Every public task is a synthetic reconstruction, and the customers are not real. The framework is open source under MIT on GitHub, but the setup asks for Docker, a pinned data checkout, a search server and three model endpoints. That is a weekend project, not a Monday check.
What transfers is the frame. When I last wrote about the security rules I keep for a solo agent stack, the question was what an agent is allowed to touch. This is the companion question: after it touches something, how do you know it touched it correctly?
AI Agent Reliability Means Passing 20 Times, Not Once
The single-attempt score flatters almost every model, and the repeat score is where AI agent reliability actually shows up. ThinkingBox runs each task 20 times from the same clean state, then asks how many tasks pass every single time.
A model that solves a task once has shown you it can. A model that solves it 20 times in a row has shown you it will.
The gap between those two is large. Kimi-K3 solves 93.89% of tasks at least once, 476 of 507. Only 68 tasks, or 13.41%, pass all 20 attempts. Claude Opus 5 solves fewer tasks at least once, 79.09%, but passes 241 of them, 47.53%, on every attempt.
The retention numbers make the same point from another angle. GPT-6 Astra keeps roughly 78% of its single-attempt rate across 20 runs. Claude Opus 5.5 and Opus 5 keep about 71%. GLM-5.1, Kimi-K2.6 and DeepSeek-V4-Pro keep about 8%. On overall pass@1, Claude Opus 5.5 leads at 67.16%, with Opus 5 at 66.50% and Kimi-K3 at 57.37% as the top open-weight model.
Domain matters for AI agent reliability too. Claude Opus 4.6 scores 68.62% on retail tasks and 8.30% on auto insurance. Same model, very different reliability, depending on how tangled the business rules are.
The Cheapest Correct Answer Is Not the Cheapest Dependable One
Cost per dependable task reorders the price list in a way per-token rates never will. Microsoft estimated two numbers from OpenRouter list prices, and they disagree with each other on purpose.
Per successful attempt, GPT-5.6 Sol is cheapest at about $0.127, GPT-5.4 is close at $0.131, Claude Opus 5.5 is around $0.276, and Opus 5 sits near $0.475. Measured that way, the Claude models look expensive.
Per dependable task, meaning the full 20-run campaign cost divided by tasks that pass 20 out of 20, the order shifts. GPT-5.4 comes in at $6.80, GPT-6 Astra at $7.45, Claude Opus 5.5 at $7.80, and GPT-5.6 Sol at $9.76. The cheapest model per success becomes the most expensive per dependable result in that group. The authors put it in one line: “The cheapest way to get a right answer is not the cheapest way to get a dependable one.”
That changes how I read every price cut. When Anthropic dropped the rate on its flagship, I walked through what the lower Opus 5.5 price actually changed for daily use. I priced it per token. I now think AI agent reliability belongs in that sum too, because a cheap run that I have to check and redo is not cheap. The same goes for hidden reasoning token spend: the meter you watch is rarely the one that decides the bill.
Five AI Agent Reliability Checks I Run on My Own Scripts
Most agent failures are recoverable tool errors, so my checks focus on state after the run, not on the transcript. ThinkingBox traces 79.9% of failure signatures to tool usage, 10.3% to wrong state updates, 7.0% to incomplete user resolutions and 2.9% to no state-changing action at all. The authors read that as mainly a retry and error-recovery problem.
My setup is tiny next to an enterprise help desk. Claude Code runs file automation and the scripts behind this site every day, and Cursor handles occasional rename and CSV jobs. Here is what I check, scaled to that size:
- Read the end state, not the summary. After a rename or move, I list the folder myself. The agent’s “done” is the claim; the directory is the evidence.
- Diff anything that edits data. For a CSV cleanup, I compare row counts before and after, because extra effects were the second most common failure type.
- Rerun the boring jobs. A script that works once on Monday has not earned trust. I run recurring automations a few times on a copy before I let them touch client folders.
- Treat silence as unverified. A run with no error message is exactly the case ThinkingBox flagged, so “no complaints” counts as a pending check, not a pass.
- Make the agent prove the result. I ask Claude Code to print the changed state, such as a file list or a count, rather than describe what it did.
None of this needs a benchmark harness. It needs a habit of asking where the evidence lives. Good AI agent reliability, at solo scale, is mostly the discipline of not taking the last message at face value.
For me, the most useful line in this report is not a leaderboard position. It is the reminder that a clean finish and a correct finish are different events, and only one of them shows up on screen. I will keep using Claude Code daily, and the Opus 5.5 retention numbers make me more comfortable with that, not less. But I will treat AI agent reliability as something I check in the files, every time, rather than something I assume from a confident summary.
Sources
- Hugging Face Blog — The Agent Said It Was Done. The Database Disagreed. (Microsoft ThinkingBox)
- GitHub — microsoft/thinkingbox
AI-assisted research and drafting. Reviewed and published by ToolMint.