Your coding agent's harness mostly decides the bill, not the score

If you pick a coding agent because its harness scored higher on a leaderboard, you are probably choosing on noise. If you pick it because of what it costs per task, you are choosing on something real. That is the uncomfortable split in What Does a Harness Buy? Tokens, Mostly by Yangze Liu and Zhongyi Han, posted on 3 October.

A harness is everything wrapped around the model: the system prompt, the tool definitions, the context management. Vendors ship harness releases constantly and attach pass-rate gains to them. Leaderboards mix harnesses freely. The paper asks the question almost nobody controls for: hold the model fixed, swap only the harness, and how much does the score actually move?

The setup

Five models, three production harnesses (Claude Code, mini-SWE-agent and OpenCode), SWE-bench Verified. The part that makes this paper worth reading is that the authors also reran identical configurations. That gives them a noise floor: how much a score moves when literally nothing changes except the run.

What they found

Heavy and light harnesses tie. On 447 tasks with two models, Claude Code (the heaviest harness tested) and mini-SWE-agent (the lightest) land within five points of each other. Five points is inside what the authors can resolve at that sample size, so it is a tie.

Swapping the harness looks exactly like rerunning it. On a 45-task hard subset across all five models, changing the harness flips 13% of task outcomes. Rerunning the same harness also flips 13%. And the tasks a harness "wins" in one run are not the tasks it wins in the next. No harness owns a stable set of problems. The wins just churn.

The only effect that clears the noise is a loss. OpenCode trails by up to 9 points on the large pool. For one model, about half of that gap comes from runs cut short by its output cap. So the one real harness effect in the study is partly a configuration limit on output length.

The bill is where harnesses really differ. Same model, same tasks, same price list: cost per task varies by up to 3x across harnesses. The paper traces this to the preamble, meaning the system prompt plus tool schemas that get resent on every step, multiplied by step count. How fast context grows per step and how much each tool call returns differ far less. Cached-input pricing scales the bill but does not change which harness is cheapest.

The number most people should take away

The rerun data lets the authors compute what a harness comparison actually needs. At the disagreement rate they observe, 45 tasks detect a 13-point gap only half the time, and fail to reach 80% power for any gap. Even 447 tasks only resolve differences of about 5 points.

Hold that against the harness changelogs you have read this year. A release note claiming "+3% on SWE-bench" is reporting a difference smaller than the full Verified set can reliably detect, unless the vendor ran many repeats and did not say so. This is the same lesson as the replay study I wrote about last week: an agent that passes on rerun has often just been resampled. Here it applies one level up, to the tooling itself.

Where I would push back

The paper is narrower than its title suggests, and it is honest about that in the abstract if you read closely.

• SWE-bench Verified is one shape of work. Single-issue Python fixes with a test oracle. Harness features like planning modes, subagents, long-session memory or project instruction files are built for multi-hour, multi-file work that this benchmark barely exercises. "Harnesses do not move the score here" is not the same as "harnesses do not matter."
• The large-pool comparison covers two models. The five-model results come from 45 tasks, which the paper itself says cannot catch a 13-point gap reliably. The finding is strongest as a statement about measurement, weaker as a ranking.
• Three harnesses is a small sample of a fast-moving field, and harness versions change weekly.

None of that undoes the core point. If even a careful study cannot separate these harnesses on pass rate, a vendor blog post with one run per configuration certainly cannot.

What to do with this

If you run coding agents at any volume, the actionable line item is the preamble. Every tool schema you register and every paragraph of system prompt is paid for on every step of every task. A harness that exposes forty tools to fix a typo is spending money the leaderboard will never show you.

Concretely:

1. Measure cost per resolved task, not pass rate alone, when you compare setups. Pass rate differences under about 5 points are probably noise; a 2x cost difference is not.
2. Audit what your harness sends on the first call. Count the tokens in the system prompt and tool definitions. That fixed overhead, times average step count, is most of your bill.
3. Rerun before you believe. If you change a prompt or tool and the score moves, run the old configuration again first. If the old config also moves that much, you have learned nothing yet.

The model is still doing the work. The harness, on this evidence, mostly decides what you pay for it.

Source: Yangze Liu and Zhongyi Han, What Does a Harness Buy? Tokens, Mostly, arXiv:2610.04433, October 2026.

---

Source: https://arxiv.org/abs/2610.04433