Newest papers

Where AI Vulnerability Discovery Harnesses Go Quiet: Structural False Negatives and Hidden Vulnerabilities

Gadi Evron · 2026 · arXiv (Cornell University)

DOI: 10.48550/arxiv.2610.29379 · cited by 0 works (OpenAlex) · open-access PDF

Spanish

Los pasos que seleccionan el código, lo encuentran, y almacenan y fusionan los hallazgos del código pueden caer en una vulnerabilidad antes o después de que el modelo se ejecute, independientemente de la capacidad del modelo, y el informe generalmente no da ningún signo de que esto ocurrió y es más difícil de evadir. Estas gotas vienen de las opciones de ingeniería y los errores. Dejan las vulnerabilidades sin detectar, y un atacante que conoce las reglas de código abierto también puede usarlas para ocultar una vulnerabilidad a propósito. Entendiendo estas gotas muestra cómo construir las gotas que las arreglan o las revelan y son más difíciles de evadir. A partir de las dos gotas de contenido que mantenemos, Knostic's OpenAnt y Raptor, y luego examinando las gotas de código abierto

Machine translation (Meta M2M100); the English original follows.

Abstract

LLM-based vulnerability discovery harnesses can miss real, reachable vulnerabilities because of how the harness around the model is built. Steps that select code, filter it, and store and merge findings can drop a vulnerability before or after the model runs, independently of the model's capability, and the report usually gives no sign that this happened. These drops come from engineering choices, and errors. They leave vulnerabilities undetected, and an attacker who knows the harness's rules can also use them to hide a vulnerability on purpose. Understanding these drops shows how to build harnesses that fix or disclose them and are harder to evade. Starting from the two harnesses we maintain, Knostic's OpenAnt and raptor, and then examining 10 additional open-source harnesses, we mapped 13 points where such drops occur and organized them into a taxonomy. Using each harness's own code, we show a single file that six harnesses dropped before their model saw its code, five of them without notice. We propose engineering failure patterns mapped to these drops, and relate them to Ptacek and Newsham's work on intrusion detection evasion. The paper also covers content-steered verdicts and prompt injection, and gives design principles and tests for harnesses that report accurately what they analyzed. The reliability findings likely apply to other LLM harnesses, beyond vulnerability research and application security, and the evasion findings apply wherever an attacker can write or influence the code being checked. We also argue that harnesses built to be descriptive can, in effect, be prescriptive. Rules written to describe typical code and typical model replies end up deciding what is analyzed, and anything that does not conform is treated as absent or as "no vulnerability."

Read in: English · Spanish · Chinese · Hindi · Arabic · Portuguese · French · German · Japanese · Russian · Korean · Indonesian · Vietnamese

Topics: Computer science, Computer security, Vulnerability (computing), Evasion (ethics), Secure coding, Vulnerability assessment

Sources: OpenAlex (CC0), arXiv, Crossref. Agent Exchange