The agent's instructions rot, and there is no failing test for it

A skill file has no build step. That is the entire problem.

When code goes stale, something breaks loudly. A renamed function throws. A moved endpoint returns 404. CI goes red and somebody fixes it before lunch. When the file you wrote to tell an agent how your repository works goes stale, nothing happens. The agent reads it, believes it, and hands you confident guidance built on a version of the codebase that stopped existing two releases ago.

Thirteen researchers put a number on how often that happens. Repo2Skill-Evo, posted to arXiv on 22 August, treats each release of a repository as a maintenance job: here is the skill set that described version one, here is the official patch that produced version two, now update the skill without destroying the parts that are still true. They ran it over 57 real repositories and 105 release transitions.

Every one of the 105 transitions invalidated part of the version-one skill set. Not most. All of them.

That is the finding worth carrying around, and it is easy to skip past on the way to the more quotable result, which is that six frontier agents managed only 29.9% to 69.7% average macro F1 at repairing the damage. The F1 range tells you today's models are mediocre at a specific chore. The 105 out of 105 tells you something structural about the chore itself. If you keep an instructions file and you cut a release, some of that file is now false. It is not an edge case you might hit. It is the resting state.

Writing it down is what removed the feedback loop

The pitch for agent skills, CLAUDE.md files, cursor rules, or whatever your harness calls them, is that procedural knowledge gets discovered once and reused forever. Which API to call, which script to run, which convention this release expects. Prior work says it improves agent performance and I believe it, because it obviously does.

The cost is subtle enough that I had not thought about it properly until reading this. Knowledge in a person's head gets corrected when the person is surprised. They run the old command, it fails, they update their model of the world. Externalising that knowledge into a file removes the surprise, because the file is never the thing that runs. It sits beside the repository and describes it, and descriptions do not fail.

I have a first-hand version of this. An instructions file in the repository behind this site described one of its own subsystems as built but not yet scheduled. That subsystem had been running daily for three weeks. The line was wrong for twenty-one days, was read by an agent every one of those days, and nothing anywhere noticed, because there is nothing in a markdown bullet that can fail. That is this paper's result without the benchmark around it.

Both ways of being wrong are available

The part of the paper most likely to change what you actually do is the error analysis. Two failure modes dominate, and they pull in opposite directions.

Agents miss files. The skill set spans more of the repository than the agent inspects, so obsolete content survives the edit untouched. That is the quiet failure and it leaves you exactly where you started, except now with a timestamp suggesting the file was reviewed.

Or agents edit too broadly. Recall goes up, precision goes down, and guidance that was still perfectly valid gets rewritten or removed. The agent then has to rediscover that knowledge live, which is the situation the skill existed to prevent.

There is no safe direction to lean, which rules out the tuning knob you would reach for first. You cannot instruct your way out of this with "be thorough" or "be conservative", because those are the two failure modes, and each one is the cure for the other.

It also rules out the obvious automation. "Have the agent refresh its own instructions after each release" is not a workaround for this paper. It is the task the paper measured, and the best score was under seventy percent.

What I would not claim from it

The metric is patch-grounded, meaning the gold standard for what should have been edited is derived from the official diff. That is a reasonable proxy and it is also a judgment call, and a contestable one. Someone else deriving the gold set could reasonably mark different lines as stale.

More importantly, this measures skill maintenance, not downstream damage. The paper does not show that a stale skill causes an agent to fail a task. It shows the skill contains obsolete guidance and that agents cannot reliably remove it. The harm is inferred rather than demonstrated. I think the inference is sound, but it is an inference and the post-hoc version where someone measures actual task failure against skill age has not been written yet.

It is also a preprint with no peer review, and the six evaluated agents are a snapshot that will be out of date within months, which is a mildly funny property for a paper about things going out of date.

The one change worth making this week

Stop treating instruction files as documentation and start treating them as a cache with no invalidation policy.

Record which release each file was written against, as a commit hash or a version, in the file itself. Then make the drift loud rather than silent. If a skill names a script, a command, an environment variable, or an API shape, add a check that asserts those still exist. Most of that is a few lines of grep in CI.

That will not catch semantic staleness, where the command still exists but the right answer changed. Nothing cheap will. What it does catch is the class of rot that is mechanically detectable and currently invisible, and it converts it into the one thing a stale line has never had, which is a way to fail.

A wrong line in an instructions file that breaks a check is a bug report. A wrong line that just sits there is a lie your agent repeats every single day, in a confident voice, to you.

Source: Repo2Skill-Evo: Repository Skills Go Stale in Silence, Chenyuan Duan et al., arXiv:2608.21964, 22 August 2026.

---

Source: https://arxiv.org/abs/2608.21964