A passing rerun is not a fixed agent

If your multi-agent system fails, you rerun it, and it passes, you have learned almost nothing. That is the uncomfortable result in Repair or Resample? Rethinking Failure Debugging in LLM Multi-Agent Systems (Luan et al., August 2026), and it applies well beyond the research tools the paper tests.

The question the authors ask is simple. When a debugging method "fixes" a failed agent run, did it fix anything? Or did it just roll the dice again and get a different outcome, because language models sample?

Their answer, measured: mostly dice.

The numbers

The team built a replay harness, SymTrace, that records every step of a multi-agent run. It can then rebuild the run exactly up to a chosen point from the logs and let the models regenerate only what comes after. That gives them something most agent debugging lacks: a controlled experiment. Same history, same failure point, one variable changed.

They paired it with SymFail, a set of 536 failed runs across three common multi-agent frameworks. Humans annotated each one with where it went wrong, what kind of failure it was, and the trace evidence for that call.

Then they tested the standard approach: rerun the whole thing and see what happens.

• Reproduction: 67.97%. Rerun a failed task and roughly a third of the time the original failure does not come back. The bug you were chasing is simply gone. Nothing changed except the sample.
• Repair: 6.90%. Existing rerun-based repair methods corrected the recorded failure in fewer than one case in fourteen.
• Targeted intervention: 20.15%. Using the annotated symptom to intervene at the specific step where things broke, then regenerating only downstream, repaired about one in five.

The paper headlines that last figure as a "191.89% improvement" over the state of the art. That is accurate and it is also the least useful way to say it. In absolute terms the best method in the paper still leaves four out of five failures unrepaired. Read it as a measurement of how bad the baseline is.

Why a passing rerun is not evidence

The 68% reproduction rate is the finding I would put in front of anyone running agents in production.

Here is the usual loop. A run fails. You look at the trace, form a theory, tweak a prompt or a tool description, and run it again. It passes. You ship the tweak.

If a third of failures vanish on a plain rerun with no change at all, then a meaningful share of those "fixes" were never fixes. You changed the prompt and drew a lucky sample, and the prompt change gets the credit. Do that across a few dozen incidents and your system prompt fills up with superstition: instructions that exist because they happened to be present the one time the dice landed well.

The paper makes a second, quieter point in its introduction. Task-level verdicts, pass or fail on the whole job, do not tell you whether the specific broken step was corrected. A run can succeed by routing around the failure, through a different agent or a different tool path, while the original bad behaviour is still there waiting for the next input. Green at the end is not the same as fixed in the middle.

What it does not show

Three limits worth being straight about.

First, the 20% targeted-repair result leans on human-annotated failure locations. In practice you do not get a labelled root cause handed to you. The paper shows that knowing where it broke helps a lot. It does not show you can find that point automatically at the same quality, and that is the hard part.

Second, three frameworks and 536 traces is a solid study but a specific one. The exact percentages will move with the models, the frameworks, and the tasks. The direction, rerun repair being mostly resampling, is the claim I would trust. The second decimal place I would hold loosely.

Third, faithful replay assumes you can reconstruct the world as well as the conversation. The authors spend an appendix on external state for a reason. If an agent wrote to a database or called a payment API before the failure, replaying "up to the anchor" from logs is not the same as rewinding reality.

What to do with it

The practical lesson needs no new research tooling, only discipline you can apply this week.

Record enough to replay. Log every model input and output, every tool call and result, in an order you can play back. If you cannot rebuild a run up to a given step, you cannot run a controlled test on that step, and every debugging session is a coin toss.

Rerun before you fix. When something fails, run it again unchanged a few times first. If the failure does not reproduce, you are looking at a rate, not a bug, and the fix is about variance (validation, retries with checks, constraining output) rather than about the prompt wording that happened to be on screen.

Test the fix against the failure, not the task. Replay from the step where it broke, with your change, several times. Check whether the bad behaviour at that step is gone. Whether the job eventually went green tells you much less.

None of this is glamorous. But the paper's core number says that without it, the odds your agent fix did anything are roughly the odds of a rerun, and that makes your debugging process a slot machine with a commit history.

Source: Zhongwen Luan, Xiaoyu Zhang, Ming Hu, Yue Yang, Jiongchi Yu, Xiaohong Chen, "Repair or Resample? Rethinking Failure Debugging in LLM Multi-Agent Systems", arXiv:2608.25920, 2026.

---

Source: https://arxiv.org/abs/2608.25920