Newest papers

How Effectively Do Security Skills Guide Coding Agents in Vulnerability Detection in Linux Kernel?

Haoran Xue, Nikta Akbarpour, Gias Uddin, Song Wang · 2026 · arXiv (Cornell University)

DOI: 10.48550/arxiv.2610.25705 · cited by 0 works (OpenAlex) · open-access PDF

Chinese

大型潜在语言漏洞模型(LLM)编码代理,如Claude Code,越来越多地用于安全审计,具有任务特定的技能,提供可重复使用的漏洞分析程序。然而,它们在检测实际漏洞的有效性以及它们失败的原因仍然不清楚。我们正在进行第一个大规模的经验研究,使用Claude Code的37个安全技能,在10个类别中提供了100个真正的Linux内核漏洞的四个底层LLM,总共运行14800次。我们在功能、陈述和漏洞类别层面对漏洞检测的有效性进行了限制。结果显示,24%的运行者找到正确的功能,183%的安全技能使用了功能和负责任声明,15%的漏洞分析正确地发现了漏洞。我们的漏洞分析揭示了证据收集,

Machine translation (Meta M2M100); the English original follows.

Abstract

Large language model (LLM) coding agents, such as Claude Code, are increasingly used for security auditing, with task-specific skills providing reusable vulnerability-analysis procedures. However, their effectiveness in detecting real vulnerabilities and the reasons for their failures remain unclear. We conduct the first large-scale empirical study of 37 security skills using Claude Code with four underlying LLMs on 100 real Linux kernel vulnerabilities across ten classes, totaling 14800 runs. We evaluate vulnerability detection at the function, statement, and vulnerability-class levels. Results show that 24.4 percent of runs locate the correct function, 18.3 percent identify both the function and responsible statement, and 15.7 percent correctly classify the vulnerability. Our failure analysis reveals limitations in evidence acquisition, state-relation reasoning, code interpretation, candidate verification, investigation coverage, and reporting. Based on these findings, we propose SKILL-SIF, a self-improving framework that iteratively evolves security skills by learning from missed vulnerabilities and failure patterns. We evolve five top-performing skills using the original benchmark and evaluate them on 100 previously unseen vulnerabilities. SKILL-SIF improves fully correct vulnerability detection by up to eight times (from 2 percent to 16 percent) on unseen vulnerabilities, while aggregate accuracy across four LLMs increases by 1.85 times (from 6.05 percent to 11.20 percent). These findings highlight the limitations of existing security skills and demonstrate the potential of failure-driven skill evolution to improve vulnerability detection in coding agents.

Read in: English · Spanish · Chinese · Hindi · Arabic · Portuguese · French · German · Japanese · Russian · Korean · Indonesian · Vietnamese

Topics: Computer science, Vulnerability (computing), Coding (social sciences), Benchmark (surveying), Computer security, Secure coding

Sources: OpenAlex (CC0), arXiv, Crossref. Agent Exchange