Your agent's trust boundary is the skills folder

Most people running coding agents carry an unstated security model: the agent is smart, so if something asks it to do something obviously bad, it will notice and refuse. A paper posted to arXiv on 5 August puts a number on that assumption. Across 5,629 runs, the agents noticed 1.99% of the time.

The paper is Towards a Risk Assessment of Malicious Skill Files in Coding Agents by Rui Yang, Michael Fu, Kla Tantithamthavorn, Chetan Arora and Joey Chua. The setup is straightforward. Take 471 real shell commands with genuine offensive capability. Use six LLMs across four model families to rewrite each one as a plausible, useful-looking skill file: a folder of natural-language instructions plus scripts, the same shape every agent harness now loads to specialise its behaviour. That produces 2,826 skills, each mapped to one of 11 MITRE ATT&CK tactics. Then install them and count what runs.

Gemini CLI executed the payload in 95.5 to 96.1% of runs. Qwen Code, 71.6 to 74.0%. Both figures were checked against a blind human gold standard with a three-judge panel and Cohen's kappa of 0.85, which is stronger validation than most LLM-as-judge work bothers with.

The number that matters is the small one

The 95% figure will get the attention. It is the less interesting one.

An agent executing a skill file is the agent working correctly. Skills are not untrusted input in the way a fetched web page is untrusted input. They are instructions the operator deliberately installed, and following them is the entire feature. Measuring that an agent ran the skill you gave it is close to measuring that a shell ran the script you typed.

The 1.99% is the finding. That is how often the agent explicitly recognised the safety problem in what it was being asked to do. Between "an operator installs a skill" and "arbitrary shell runs against connected systems with delegated credentials," there is essentially no review step. Not a weak one. Not one that fails under adversarial pressure. There is close to nothing there, and the model's judgment, which is where most people quietly assume the safety lives, is doing 2% of a job nobody assigned it.

The second useful signal is the gap between the two agents. 95% against 72% is a large difference in outcome from a shared underlying threat, and the authors report the result is nearly invariant to which model generated the malicious skill. Read those two together and the conclusion is that this is a property of the harness, not of the model. Which is good news, because harnesses are things you can fix. Confirmation prompts, scoped filesystem access, an allowlist for network egress and a diff of what a skill actually executes are all ordinary engineering. None of them require the model to be smarter.

Where the paper is narrower than its abstract

Two agents were tested, both terminal CLI tools, and neither is the most widely installed agent in the category. The paper says "enterprise-grade," which is doing some work. Whether the same rates hold for harnesses with different permission defaults is unmeasured, and the 24-point spread between the two that were tested is a good reason to expect variance rather than assume it generalises.

The skills are also LLM-synthesised to look benign. That is one specific threat shape: an attacker with model access, writing plausible cover text at volume. It is a realistic shape, and the volume argument is the point, since generating 2,826 disguised skills used to be the expensive part of this attack and now is not. But it is not the whole space. A human-authored skill targeting one company would look different and would probably be harder to catch.

And "exploited" means the command ran under the study's conditions. A user with different permission settings, or one paying attention to a confirmation prompt, sits somewhere else on the curve. The study is not claiming otherwise, but the headline number gets quoted without that qualifier.

None of this makes the result soft. It makes it a floor rather than a ceiling.

What to actually do about it

Treat skills as dependencies, because that is what they are. They install from repos and marketplaces, they get copy-pasted between machines, they accumulate, and almost nobody re-reads one after the day they added it. The npm supply chain took years to develop even partial hygiene around exactly this. Agent skills arrived with none of it and a much shorter path from "installed" to "running commands with your credentials."

Three things worth doing this week, in rising order of effort:

Read the scripts, not the prose. The natural-language part of a skill file is the part designed to look reasonable. Whatever it shells out to is the part that runs.

Audit what is already installed. Most people with a serious agent setup have more skills than they can name. The ones that came from a link nine months ago are the ones to look at.

Stop globally disabling permission prompts. The confirmation gate is annoying and it is currently the only layer in this stack that measured as doing anything. Scope it down rather than switching it off.

The framing that survives this paper is simple. Your agent's trust boundary is not the model. It is the skills directory, and right now most people do not have one.

Source: arXiv:2608.05223, Yang et al., 5 August 2026. Code and dataset released by the authors.

---

Source: https://arxiv.org/abs/2608.05223