Newest papers

How Much Does the Harness Matter, and for Whom? A Multi-Model Study of Context in LLM-Driven Vulnerability Discovery

Ganni Galea Curmi, Ben Wilde · 2026 · arXiv (Cornell University)

DOI: 10.48550/arxiv.2610.28802 · cited by 0 works (OpenAlex) · open-access PDF

Portuguese

Quanto um agente broader context harness afeta a descoberta de vulnerabilidades, e seu valor depende do modelo? Estudamos 24 configurações de modelo-runner usando um harness de código aberto em uma base de tarefas de produção cujas vulnerabilidades não eram públicas no momento da varredura. O pipeline combina rastreamento estático, investigação LLM, auto-validação do modelo, mapeamento de causa-raiz, revisão de fonte independente e confirmação de conceito isolada de modelo usando uma base de dados de código aberto. Através de 265 causas de segurança confirmadas, a maior configuração de duas correntes cobre 60.4%, enquanto 14,3% é encontrada por exatamente um modelo. Nenhuma configuração única cobre todas as descobertas observadas. Em um experimento de contexto de quatro modelos, movendo-se de um prompt mínimo a

Machine translation (Meta M2M100); the English original follows.

Abstract

How much does an agent harness affect vulnerability discovery, and does its value depend on the model? We study 24 model-runner configurations using an open-source harness on a production codebase whose vulnerabilities were non-public at scan time. The pipeline combines static screening, LLM investigation, model self-revalidation, root-cause mapping, independent source review, and isolated proof-of-concept confirmation. Across 265 confirmed security root causes, the largest two-run configuration covers 60.4%, while 14.3% are found by exactly one model. No single configuration covers all observed findings. In a four-model context experiment, moving from a minimal prompt to an engineered prompt with a CodeQL-derived call graph raises criticality-weighted yield by 20-86% for smaller and mid-tier models, compared with 2% for the frontier model. This comparison changes both prompt and context. Separating those effects shows that the engineered prompt improves yield for all four models, while adding CodeQL context beyond that prompt improves yield only for the smallest model. A held-out CyberGym subset shows the same direction of target-recall improvement for the combined treatment. Frontier models achieve low false-report rates but do not consistently achieve the highest yield. Strong lower-cost models can deliver similar performance at an order of magnitude lower cost. These results support matching models, prompts, and context to the task, and combining configurations for broader coverage.

Read in: English · Spanish · Chinese · Hindi · Arabic · Portuguese · French · German · Japanese · Russian · Korean · Indonesian · Vietnamese

Topics: Context (archaeology), Computer science, Yield (engineering), Vulnerability (computing), Matching (statistics), Pipeline (software)

Sources: OpenAlex (CC0), arXiv, Crossref. Agent Exchange