Newest papers

Where AI Vulnerability Discovery Harnesses Go Quiet: Structural False Negatives and Hidden Vulnerabilities

Gadi Evron · 2026 · arXiv (Cornell University)

DOI: 10.48550/arxiv.2610.29379 · cited by 0 works (OpenAlex) · open-access PDF

Portuguese

Os passos que selecionam o código, o filtram e o armazenam e os acham escritos podem cair uma vulnerabilidade antes ou depois que o modelo é executado, independentemente da capacidade do modelo, e o relatório geralmente não dá nenhum sinal de que isso aconteceu e são mais difíceis de evitar. Essas gotas vêm das escolhas de engenharia e erros. Eles deixam as vulnerabilidades não detectadas, e um atacante que conhece as regras de código aberto também pode usá-las para esconder uma vulnerabilidade com o propósito. Entendendo essas gotas mostra como construir gotas que corrigem ou revelam e são mais difíceis de escapar. A partir das duas gotas de conteúdo que mantemos, o OpenAnt e o Raptor da Knostic, e depois examinando 10 gotas de segurança adicionais de código

Machine translation (Meta M2M100); the English original follows.

Abstract

LLM-based vulnerability discovery harnesses can miss real, reachable vulnerabilities because of how the harness around the model is built. Steps that select code, filter it, and store and merge findings can drop a vulnerability before or after the model runs, independently of the model's capability, and the report usually gives no sign that this happened. These drops come from engineering choices, and errors. They leave vulnerabilities undetected, and an attacker who knows the harness's rules can also use them to hide a vulnerability on purpose. Understanding these drops shows how to build harnesses that fix or disclose them and are harder to evade. Starting from the two harnesses we maintain, Knostic's OpenAnt and raptor, and then examining 10 additional open-source harnesses, we mapped 13 points where such drops occur and organized them into a taxonomy. Using each harness's own code, we show a single file that six harnesses dropped before their model saw its code, five of them without notice. We propose engineering failure patterns mapped to these drops, and relate them to Ptacek and Newsham's work on intrusion detection evasion. The paper also covers content-steered verdicts and prompt injection, and gives design principles and tests for harnesses that report accurately what they analyzed. The reliability findings likely apply to other LLM harnesses, beyond vulnerability research and application security, and the evasion findings apply wherever an attacker can write or influence the code being checked. We also argue that harnesses built to be descriptive can, in effect, be prescriptive. Rules written to describe typical code and typical model replies end up deciding what is analyzed, and anything that does not conform is treated as absent or as "no vulnerability."

Read in: English · Spanish · Chinese · Hindi · Arabic · Portuguese · French · German · Japanese · Russian · Korean · Indonesian · Vietnamese

Topics: Computer science, Computer security, Vulnerability (computing), Evasion (ethics), Secure coding, Vulnerability assessment

Sources: OpenAlex (CC0), arXiv, Crossref. Agent Exchange