Newest papers

How Much Does the Harness Matter, and for Whom? A Multi-Model Study of Context in LLM-Driven Vulnerability Discovery

Ganni Galea Curmi, Ben Wilde · 2026 · arXiv (Cornell University)

DOI: 10.48550/arxiv.2610.28802 · cited by 0 works (OpenAlex) · open-access PDF

German

Wie viel Einfluss hat ein Broader Kontext-Harness auf die Vulnerabilitätsentdeckung, und hängt sein Wert vom Modell ab? Wir untersuchen 24 Modelllaufkonfigurationen mithilfe eines Open-Source-Harnesses auf einer Produktions-Code-Task-Basis, deren Vulnerabilitäten zum Zeitpunkt des Scans nicht öffentlich waren. Die Pipeline kombiniert statische Screening, LLM-Untersuchung, Modell-Selbstvalidierung, Wurzelursache-Mapping, unabhängige Quellquelle-Überprüfung und isolierte Proof-of-Concept-Bestätigung. Über 265 bestätigte Sicherheits-Wurzelursachen erhöht die größte Zwei-Run-Konfiguration 60.4%, während 14,3% durch genau ein Modell gefunden werden. Ke

Machine translation (Meta M2M100); the English original follows.

Abstract

How much does an agent harness affect vulnerability discovery, and does its value depend on the model? We study 24 model-runner configurations using an open-source harness on a production codebase whose vulnerabilities were non-public at scan time. The pipeline combines static screening, LLM investigation, model self-revalidation, root-cause mapping, independent source review, and isolated proof-of-concept confirmation. Across 265 confirmed security root causes, the largest two-run configuration covers 60.4%, while 14.3% are found by exactly one model. No single configuration covers all observed findings. In a four-model context experiment, moving from a minimal prompt to an engineered prompt with a CodeQL-derived call graph raises criticality-weighted yield by 20-86% for smaller and mid-tier models, compared with 2% for the frontier model. This comparison changes both prompt and context. Separating those effects shows that the engineered prompt improves yield for all four models, while adding CodeQL context beyond that prompt improves yield only for the smallest model. A held-out CyberGym subset shows the same direction of target-recall improvement for the combined treatment. Frontier models achieve low false-report rates but do not consistently achieve the highest yield. Strong lower-cost models can deliver similar performance at an order of magnitude lower cost. These results support matching models, prompts, and context to the task, and combining configurations for broader coverage.

Read in: English · Spanish · Chinese · Hindi · Arabic · Portuguese · French · German · Japanese · Russian · Korean · Indonesian · Vietnamese

Topics: Context (archaeology), Computer science, Yield (engineering), Vulnerability (computing), Matching (statistics), Pipeline (software)

Sources: OpenAlex (CC0), arXiv, Crossref. Agent Exchange