Newest papers

Narratives Mislead: An Empirical Study of How LLMs Misjudge CVSS Severity in Bug Bounty Reports

Ruitao Feng, Yuekang Li, Yao Zhang · 2026 · arXiv (Cornell University)

DOI: 10.48550/arxiv.2610.24623 · cited by 0 works (OpenAlex) · open-access PDF

Korean

버그 보너스 보고서가 낮은 잠재적 인 보너스 보고서 플랫폼은 취약성 보고서가 판매자가 그들을 분류 할 수있는 것보다 빠르게 종료되며 대형 언어 모델 (LLMs)는 점점 더 특정한 우선 순위를 추구하는 Common Vulnerability Scoring System (CVSS)의 심각성을 부여하는 것으로 간주됩니다. 사전 평가는 CVE 설명을 짧게하고 보고 정확도를 얻습니다. 그러나 배포는 모델이 얼마나 자주 실수하는지에 따라 달라집니다. CVSS 연구는 각 메트릭에 대 한 형식적 의미를 제공합니다. 연구자들은 이야기 보고서에 영향을 설명하는 방법에 따라 달라집니다. 모델 합성은 두 가지의 차이점이있는 곳에 오류를 집중한다고 제안합니다. 우리는

Machine translation (Meta M2M100); the English original follows.

Abstract

Bug bounty platforms receive vulnerability reports faster than vendors can triage them, and Large Language Models (LLMs) are increasingly considered for assigning the Common Vulnerability Scoring System (CVSS) severity that drives prioritization. Prior evaluations score short CVE descriptions and report accuracy, but deployment depends on how models err, not only how often. CVSS gives each metric a formal meaning that departs from how researchers describe impact in narrative reports, and model rationales suggest that errors concentrate where the two diverge; we ask whether this pattern is systematic. From 4,281 publicly disclosed reports mined from Huntr and HackerOne, we build a benchmark of 240 Huntr reports with vendor-validated CVSS vectors and evaluate eight LLMs under four prompting strategies and fine-tuning, measuring errors by severity class and score and analysing sub-vectors on individual reports. The best prompting configuration reaches only 49.58% level accuracy, ten points above always predicting the majority level, and fine-tuning a 7B model raises this to at most 56.25%, an optimistic upper bound. Errors are directional: models overestimate far more often than they underestimate, and they almost never recognize low-severity vulnerabilities. We trace these errors to three semantic boundaries that the specification draws and narrative reports blur: potential versus realized impact, impact magnitude versus authorization scope, and uncertainty versus the worst case. Because the violations originate partly in the report, we close with an exploratory case study: on four reports, a boundary-aligned, exploit-free report structure lowered a frontier model's score where the narrative invited an unclaimed account takeover, but left the metrics a report does not settle open, and the two published labels disagreed on three of the four reports.

Read in: English · Spanish · Chinese · Hindi · Arabic · Portuguese · French · German · Japanese · Russian · Korean · Indonesian · Vietnamese

Topics: Narrative, Metric (unit), Empirical research, Vulnerability (computing), Benchmark (surveying), Meaning (existential)

Sources: OpenAlex (CC0), arXiv, Crossref. Agent Exchange