The Agent Checked Its Own Work and Called It Done

When an agent fails, you usually get something to look at: a stack trace, a timeout, a red row in the workflow view. The failure these six papers measured looks like status 0.

RECEIPT went looking for the gap between a tool call that finished and a tool call that did what it was asked. The authors collected 91,469 production tool calls and 89,047 public agent trajectories, built a catalogue of 158 checked command-line invocations and ran them against eight widely used harnesses. With the program’s input closed, 11.5% of tested invocations returned status 0 without performing the requested action. Seven of the eight harnesses close input when no terminal is attached.

The mechanism is dull, which is probably why it took a paper to find it. Plenty of commands ask a question before they act: confirm this, overwrite that, continue yes/no. The harness either closes stdin or leaves it hanging at a terminal nobody types into. Closed, the program reads end-of-file, gives up, and exits successfully. The agent reads the zero and files the task as done.

That is a success signal that is not a success, and it is not a rare edge case. Across 158 invocations it works out to roughly one call in nine.

What the harness lets the agent see

RECEIPT’s repair is the most useful engineering result in this set, because it requires no new model, vendor or runtime. It keeps the input open, reads what each process is waiting for out of the system calls, and hands that state back to an unmodified agent through a model-checked protocol. False claims of success drop 73.5% across five models, and it completes all 25 guarded tasks on a second harness.

Two other numbers in that abstract matter. Leaving input open instead of closing it multiplies a task’s token use by 2.9, which is the cost of the fix and the reason nobody fixes it. The input connection alone “moves an LLM from 7th to 3rd place among 8.” A configuration detail of the harness moved a model four places in a ranking of models.

The authors put the conclusion more carefully than I would: “An agent’s reliability therefore depends on what its harness lets it observe as well as on its LLM.”

In September we published a piece on Dan Luu’s pre-registered experiment, which found that instructing an agent to use TDD, Lean, TLA+, fuzzing or mutation testing made its code no more correct. The result was a null across 26 prompt conditions. I read that at the time as agents being unable to check their own work even when told how. RECEIPT suggests a reason the instruction was useless: on some fraction of calls the check was passing on evidence that did not exist, because the command exited 0 without running. Better testing instructions cannot rescue you from a test that never executed. I am connecting two findings here rather than reporting one. RECEIPT did not study Luu’s setup, and this is a hypothesis about why his null was so flat.

95% alone, 51% in a chain

BOBA-Bench is the sharpest measurement of the group. The authors built 280 chains of 20 consecutive requests drawn from 140 real open-source projects and ran ten coding agents on both the isolated requests and the chains. Taken one at a time, the agents complete 95% of requests. Chained, they complete 51%. All ten do worse.

Worse code is not the cause. Once a reviewer checks the output, agent code is nearly as good a base for the next request as human code. The cause is the mistake the agent did not catch, because it declared the request done when its own tests passed. A request built on broken code fails about twice as often, and the mistake is rarely found later. Nor does asking the agent to review fix it. Asked directly whether an implementation is correct, the agents are right 53% of the time, which the authors describe as “barely better than a coin flip.”

One external “that’s wrong” after a failed request, plus one repair round, lifts chain completion from 51% to 59%, and a repaired mistake stops hurting the requests that follow it. Eight points is not a triumph, but it is the cheapest intervention in the paper and the only one that moved the outcome, and it required somebody outside the loop to say no.

BOBA’s authors put their conclusion plainly: “coding agents are only as reliable as the checks they can reach.” The thing being measured against is the checks, not the model and not the runtime.

The runtime question finally got a test

In August I argued that durable execution was a real need, and I called the claim that it was the missing abstraction “vendor positioning” (that post is here, with the runtime stack and the assembly guide following it). What none of those posts had was a controlled comparison, because none existed.

Permission to Retry ran one. A prospectively frozen campaign executed 336 histories on native LangGraph and Temporal hosts, crossing 14 schedules, three provider contracts and four matched application policies, and scored each history against a provider effect ledger the recovering application could not see. All 168 paired host cells have identical checked outcomes. Which host you ran on did not separate anything.

What separated the outcomes was the recovery policy the application implemented. Retained-ledger reconciliation passed all 42 histories. Cached retry passed 30, revalidation before retry 36, stable-key retry 31.

The detail I would put in front of anyone designing this is what “passed all 42” contains: 21 checked completions, 15 denials, and six unknown outcomes with reservations retained. The policy that won was not the one that resolved every case. It was the one that could tell the difference between a case it had settled and a case it had not, and recorded the second kind as unresolved instead of retrying into it. The paper separates four obligations that are easy to collapse into one retry button: checkpoint state, permission to dispatch, evidence of an external effect, permission to reconcile.

I would be a fool to read this as vindication. The paper is seven pages, both authors are at Arya.ai, their component study exercises their own company’s platform, and it states that full public reproduction “is therefore not claimed for this version.” It is one conformance study, not a survey of the category. But the specific claim it tests is the one we could not test in August, and the answer is that the host was not the variable and the policy was.

Rankings that cannot see this

When Agents Fail makes an argument about measurement that undercuts most agent leaderboards. It separates reliability from accuracy with a metric called Valid Completion Rate: how reliably an agent finishes normally and delivers an evaluable result, regardless of whether the task was done correctly. A run that crashes, hangs or emits something you cannot score is a reliability failure regardless of whether the model got the task right.

Across 111 evaluation settings and 52,168 trajectories, covering four harnesses with four models plus two native model-harness systems on six benchmarks, harness rankings move around with the model and the benchmark. Similar task scores come with distinct execution profiles. OpenCode ranked first on cross-model performance, though that ordering will date quickly. What does not date is the consequence: a leaderboard sorted by task success is not sorted by reliability, and a harness that wins on one model can lose on the next one you try.

The record can be wrong about what happened

On 29 August we published an argument that capture is not audit, setting out three properties that separate a trace from an audit trail: completeness, tamper-evidence and portability. It closed on a gap it could not resolve, that traces record what an agent did and not why.

These papers point at a fourth property, and it is more basic than the other three: the record has to be true. A log line is evidence that a process exited. It is a much weaker thing than evidence that the work happened. RECEIPT’s recommendation to builders is one sentence: “Agent builders should report what each tool call did, not only how it ended.”

Monitor-Validity Failures shows the same problem at the safety layer. The author confirmed 18 evidence-path failures across 11 LLM or agent safety project groups and four ecosystems, and 14 of them reach an enforcement boundary. In one representative case a regex detector returned a positive match and the framework stranded the result before the block decision. The abstract puts it better than I can: “a framework path may remain operational while policy-relevant evidence never reaches its decision point.” Nothing crashed, the control fired, and the evidence never arrived at the decision it was meant to inform.

Compact-Bench covers the state version of the same loss. Comparing full-history and summary-only continuations across 35 SWE-bench repairs, 146 agentic sessions and 106 production chains, LLM summaries retain at most 16.0% of recurring details against 79.1% for mechanical extraction, and production summaries retain 15.7% at the first boundary. Even the prompt configuration that retained the most recurring details only reached 15.0%, and damage across 14 summarizer backbones ranges from 5% to 64% of 22 full-history successes. For anyone treating a transcript as a record, the authors’ own conclusion is the relevant one: preserving relevant information is not the same as preserving the state needed to finish.

What the people running these systems report

Temporal’s State of Development 2026, published 25 August, ran through Qualtrics and ended with 554 respondents. It puts the top blockers to using agents more, in order, as tracking state (35.7%), debugging and managing costs. Meanwhile 41.1% encounter issues with agents daily or more, and 51.3% say they get from prototype to production-ready code in hours or faster. Those two figures sit in the same dataset: people shipping fast and hitting the same friction every day.

The caveats belong in the same breath as the numbers. Temporal sells durable execution and state tracking, so its number one blocker is the product. The report labels its successful cohort as self-reported data. I have not seen the questionnaire. Use it as a shape rather than a measurement.

The market’s answer has split into separately sold layers, which one vendor guide describes plainly: sandboxes isolate code, runtimes run agents as services, orchestration layers record each step so a run can resume. On recovery it is candid. “This usually comes from an orchestration layer, not the runtime.” And on the division of labour: “Sandbox-only products leave the runtime to a second platform, and orchestration layers save progress but do not run your services.”

The same guide sells a runtime on the strength of active-active across regions and a 99.999% SLA for agents running at least two replicas in at least two locations. That is a real number for the thing it measures, and it is not the thing these papers measure. Five nines of availability is compatible with every failure above. You can have a platform that never goes down and a log that is clean and wrong.

What I would wire

None of these is a platform decision.

Verify from outside the agent. The one intervention that moved chained completion was an external correction, and the observation repair that cut false success claims lived in the harness rather than the model. Honest reporting is not free. RECEIPT measures a 2.9 times token cost for leaving a command’s input open, against 11.5% of invocations returning a false success when it is closed. A team that has to pay the first number to avoid the second will be tempted not to.

Record what each call did, not only how it ended. Exit codes, stdout and a duration are not a record of an effect. If a command was supposed to create a row, check the row.

Treat recovery as four permissions rather than one retry. Checkpoint state, permission to dispatch, evidence of an effect, permission to reconcile. The policy that passed everything in the study above was the one that could say “unknown” and stop.

Do not compact with a summary and assume continuity. Mechanical extraction retained five times the recurring detail of an LLM summary, and even the best configuration lost most of it.

Watch the evidence path, not the process status. Monitor-Validity’s finding is not that monitors are broken. It is that a monitor can be up, correct and disconnected, and the only way to know is to check that its judgment reaches the thing that acts on it.

What these six papers do not show

All six are first versions on arXiv with no peer review behind them. I reproduced none of it. Both BOBA authors are at Amazon and the benchmark ships under the amazon-agi organisation, which makes it checkable rather than neutral. Permission to Retry is partly a study of its authors’ own product. The survey and the vendor guide are both vendor publications, and neither is neutral about the category it sells into.

The scope is also narrower than “AI agents are unreliable.” These are specific failure modes in specific task families: chained coding requests, command-line and web service operations, context compaction, safety evidence paths. The defensible claim is the one the numbers support. In the tasks these people measured, reliability was governed by what the harness let the agent observe and verify, not by the model it ran or the host underneath it.

That is a smaller claim than the one I was making in August, and I think it is a better one to build against.