---
title: "The 'Longhaul-Bench' Claim Doesn't Check Out — But the Real Evidence on Agent Reliability Is Worse"
description: "A viral claim about a 40-step benchmark and a 34% ceiling isn't traceable to any published evidence. The benchmarks that do exist paint an even less comfortable picture for anyone running autonomous agents in production."
canonical: "https://voiceagentinsurance.com/p/agent-benchmark-reliability-evidence"
site: "Voice Agent Insurance"
category: "agent benchmark reliability"
published: "2026-08-26T12:46:58.062311Z"
updated: "2026-08-26T12:46:58.046604Z"
reading_minutes: 8
keywords: []
---

## A benchmark that doesn't appear in the record

A claim has been circulating that a benchmark called 'Longhaul-Bench' scores agents on 40-step tasks and that top models stall at a 34% success rate. We looked for this benchmark, this task structure, and this figure across our evidence, and found none of it. There is no fact in our source set that names a benchmark called Longhaul-Bench, describes 40-step tasks, or reports a 34% ceiling for any model on any task. We are not asserting any of those details here, and readers should not treat them as established. If you have seen this figure repeated elsewhere, treat it the same way: as an unverified claim until someone points to the paper, the leaderboard, or the dataset that produced it.

That said, the underlying question — what happens to agent performance numbers when you look past the first stated score — is a live one, and there is real published evidence bearing on it. Two separate strands are worth walking through: one on how a single leaderboard number can be inflated by weak evaluation, and one on how unreliable agent benchmarking is as a discipline once you decompose where the variance actually comes from. Neither strand is comforting for anyone treating a headline agent score as a deployment green light.

## What happens when you stress-test the evaluation itself, not just the model

SWE-Bench Verified is a widely cited benchmark built from 500 instances of real software-engineering tasks, and its leaderboard has been described as approaching saturation, with the top system reported at 78.80% ([arxiv.org/abs/2604.16788v1](https://arxiv.org/abs/2604.16788v1)). A leaderboard that looks nearly solved is exactly the kind of number that gets quoted in procurement decks and vendor comparisons.

A re-evaluation using a framework called SWE-ABS put that number under pressure. SWE-ABS is an adversarial framework that strengthens test suites through a two-stage pipeline: coverage-driven augmentation using program slicing to target untested code regions, and mutation-driven adversarial testing (fact 548). Applied to SWE-Bench Verified's 500 instances, it strengthened 50.2% of them (fact 550), which the authors describe as a 25.1x improvement over prior work in strengthening test suites (fact 551). The point of strengthening a test suite is to check whether patches that were marked 'solved' actually solve the problem, or whether they merely pass a test that wasn't looking hard enough.

The result: one in five 'solved' patches from the top-30 agents on SWE-Bench Verified turned out to be semantically incorrect, passing only because weak test suites failed to expose their errors (fact 547). Once the stronger test suites were applied, 19.71% of previously passing patches were rejected (fact 552). The top agent's own score fell from 78.80% to 62.20% (fact 553), and the leaderboard reshuffled enough that the previously top-ranked agent dropped to fifth place (fact 554).

That's a 16.6-point swing on a benchmark treated as close to solved — produced not by retraining any model, but by asking harder questions of the existing test infrastructure. A leaderboard score is a statement about the benchmark's tests as much as about the agent: if the tests are weak, the score is inflated, and nobody deploying on the strength of that score would know it.

## The reliability problem goes deeper than any one leaderboard

A separate line of work looked past individual benchmark scores entirely and asked a more structural question: when you evaluate the same agents on the same tasks repeatedly, how much of the variance in outcomes is actually attributable to the agent, versus everything else — the specific task, the interaction between agent and task, and noise?

Using a four-facet Generalizability Theory variance decomposition — fit with three independent estimators (Henderson Method-I, REML via lme4, and a Bayesian binomial GLMM) that agreed to three decimal places (fact 567) — researchers analyzed three open agent-trace benchmarks: TheAgentCompany, tau²-bench, and AppWorld ([arxiv.org/abs/2509.16941v2](https://arxiv.org/abs/2509.16941v2)). The headline finding is stark: the agent main effect — how much of the outcome is explained by which agent you used — accounted for less than 3% of total variance in every dataset and check type (fact 565). The agent-by-task interaction, meaning how a given agent happens to perform on a given task in a way that doesn't generalize, accounted for 7-23% of variance (fact 566). In plain terms, most of what looks like a performance difference between agents is actually noise tied to specific task instances, not a stable property of the agent itself.

It gets worse on harder tasks. On the hardest quartile of tasks in tau²-bench, aggregate reliability collapsed: a measure called E-rho-squared on action checks fell from 0.752 to 0.000 (fact 568). And the designs that appeared most reliable during training were the ones that generalized worst — training-cell reliability correlated with held-out reliability at r = -0.90 on tau² (fact 569), meaning the setups that looked best on paper were systematically the least trustworthy once tested on new data.

There is a partial silver lining. Some diagnostics do transfer across benchmarks: a capability-gap ratio stayed stable at 0.35-0.40 across enterprise benchmarks (fact 570), and failure-mode profiles under the MAST taxonomy generalized well at the cell level (MAE = 0.056, r = 0.83) even though they were idiosyncratic at the trace level (MAE = 0.261) (facts 571, 572, 573). But per-family agent rankings inverted across benchmarks (fact 570) — meaning the agent that looks best on one enterprise benchmark can look worst on another, for the same underlying population of tasks.

## What this means if you're deploying agents now

Put the two strands together and a pattern emerges that has nothing to do with any specific unverified number like the one attributed to 'Longhaul-Bench.' It has to do with what a single benchmark score can and cannot tell you.

First, a headline pass rate can be inflated by the weakness of the test suite grading it, not just by the competence of the agent being graded — that's what the SWE-ABS re-evaluation demonstrated on SWE-Bench Verified, turning a 78.80% into 62.20% without touching the model. Second, even a rigorously graded benchmark score is dominated by task-specific variance rather than a stable, transferable measure of agent capability — the agent main effect explaining under 3% of variance means that ranking agents on aggregate scores is, statistically, mostly ranking which agents happened to draw easier task subsets. Third, reliability is not uniform across difficulty: it can collapse entirely on the hardest tasks, which are usually the ones closest to what an agent will actually encounter in an unscripted production environment. Fourth, the benchmark configurations that look most trustworthy during evaluation design can be the ones least likely to hold up out of sample.

For anyone running procurement or deployment decisions off a single leaderboard number, the practical implication is to ask what happened to that number under an adversarial test suite, and to ask how much of it is agent-attributable versus task-attributable. Neither question is currently answered by a standard leaderboard page. Both are the kind of question the SWE-ABS and Generalizability Theory studies were built specifically to surface.

## What these findings don't tell you

Neither study is a general indictment of agent capability, and neither should be read as such. SWE-ABS's re-evaluation is specific to SWE-Bench Verified's software-engineering task format; it says nothing directly about agent reliability on customer-service workflows or long-horizon robotic manipulation. A real benchmark for the latter, LongBench, exists separately — it evaluates six state-of-the-art policies across more than 1,000 real-world episodes (facts 522, 523, 525) — but it measures something different from either study discussed here, and isn't a source for the disputed 34%/40-step claim.

The Generalizability Theory decomposition is drawn from three specific open agent-trace benchmarks — TheAgentCompany, tau²-bench, and AppWorld — and its numbers describe variance structure within those datasets. It does not establish a universal ceiling on multi-step agent task success, nor does it name a specific step count (such as 40) at which performance stalls. If a number like that exists in a legitimate published source, we have not seen it in the evidence available here, and it should not be attributed to either of the studies described above.

What both studies do establish is that the gap between a benchmark's reported top-line number and what that number predicts about behavior on a new, harder, or differently-distributed task is larger than a single percentage figure suggests — and that gap is measurable, even if the 'Longhaul-Bench' framing being asked about is not.

## FAQ

### Is there a real benchmark called Longhaul-Bench?

There is no verified evidence in the article's source set for a benchmark by that name, a 40-step task structure, or a 34% success rate. The article identifies it as an unverified claim and points readers to the real, published evidence on agent benchmark reliability instead.

### How much did SWE-Bench Verified scores drop under adversarial testing?

When an adversarial test-suite framework called SWE-ABS was applied, 19.71% of previously passing patches were rejected, and the top agent's score fell from 78.80% to 62.20%. One in five 'solved' patches turned out to be semantically incorrect.

### What does variance decomposition reveal about agent benchmark reliability?

A Generalizability Theory decomposition across three benchmarks found the agent main effect accounted for under 3% of total variance, while task-specific interactions accounted for 7–23%. Most apparent performance differences between agents reflect task-instance noise rather than stable agent capability.

### Do agent benchmark results hold up on harder tasks?

Reliability collapsed on the hardest tasks: on the toughest quartile of tau²-bench tasks, E-rho-squared on action checks fell from 0.752 to 0.000. Additionally, the configurations that looked most reliable during training generalized worst, with a correlation of r = -0.90.

### What should teams deploying AI agents ask before trusting a benchmark score?

Procurement and deployment decisions should not rely on a single leaderboard number. The article recommends asking whether the score survived adversarial testing and how much of it is agent-attributable versus task-attributable — questions that standard leaderboard pages do not currently answer.