Newest papers

SecLog-Bench: A Scalable Benchmark for Cybersecurity Log Reasoning with Scaffolded Evidence

Yifan Wang, Dongliang Xu, Xiaoyong Li, Jing Li, Peter S. Pan, Yue Yao · 2026 · arXiv (Cornell University)

DOI: 10.48550/arxiv.2610.31000 · cited by 0 works (OpenAlex) · open-access PDF

German

Große Sprachmodelle (LLMs) haben ein starkes Potenzial für Cybersecurity gezeigt, was zu einem wachsenden Interesse an der automatisierten Cybersecurity-Log-Analyse auf LLM-Basis führt. Allerdings sind die aktuellen Datensätze (1) in der Größe begrenzt; (2) verlassen sich hauptsächlich auf Einzelquelle-Beweise; und (3) haben nur binäre oder Multi-Klasse-Angriffsetiketten, ohne dichte Anmerkungen des Argumentationsprozesses. Um diese Einschränkungen anzugehen, hat SecLog-Bench drei Schlüsselfunktionen. Erstens bauen wir eine erweiterbare Simulationsplattform auf, die 29 Angriffsszenarien abdeckt, die wiederholte Ausführung und die Integration neuer Szenarien unterstützt und eine skali

Machine translation (Meta M2M100); the English original follows.

Abstract

Large language models (LLMs) have shown strong potential for cybersecurity, leading to growing interest in LLM-based automated cybersecurity log analysis. However, current datasets (1) are limited in scale; (2) rely mainly on single-source evidence; and (3) have only binary or multi-class attack labels, without dense annotations of the reasoning process. To address these limitations, SecLog-Bench has three key features. First, we build an extensible simulation platform that covers 29 attack scenarios, supports repeated execution and the integration of new scenarios, and enables scalable generation of cybersecurity logs with attack-stage labels. Second, we use multi-source logs, including audit logs, application logs, and network traffic, to jointly analyze cyber attacks. Third, we annotate each sample with the essential context for cybersecurity log reasoning and a structured step-by-step evidence chain. We evaluate eight mainstream LLMs on this benchmark for post-detection attack-progress reasoning. Results show that, when provided with scaffolded evidence, current LLMs exhibit modest capability in attack-progress reasoning, with the best model achieving only 44.44% accuracy and 45.07% Macro-F1. Their reasoning remains unreliable; i.e., the highest Stage Reasoning F1 (SRF1) score is only 54.87%. Moreover, even correct predictions may be accompanied by flawed reasoning, and reasoning quality is often misaligned with prediction accuracy.

Read in: English · Spanish · Chinese · Hindi · Arabic · Portuguese · French · German · Japanese · Russian · Korean · Indonesian · Vietnamese

Topics: Computer science, Benchmark (surveying), Scalability, Context (archaeology), Key (lock), Audit

Sources: OpenAlex (CC0), arXiv, Crossref. Agent Exchange