Newest papers

SecLog-Bench: A Scalable Benchmark for Cybersecurity Log Reasoning with Scaffolded Evidence

Yifan Wang, Dongliang Xu, Xiaoyong Li, Jing Li, Peter S. Pan, Yue Yao · 2026 · arXiv (Cornell University)

DOI: 10.48550/arxiv.2610.31000 · cited by 0 works (OpenAlex) · open-access PDF

Japanese

大規模な言語モデル(LLMs)は、サイバーセキュリティのための強力な可能性を示し、LLMベースの自動サイバーセキュリティログ分析への関心が高まっています。しかし、現在のデータセット(1)は規模が制限され、(2)主に単一ソースの証拠に依存し、そして(3)サイバーセキュリティログのスケーラブル生成を可能にし、攻撃段階のラベルなしでサイバーセキュリティの攻撃ラベルを示しています。これらの制限に対処するために、SecLog-Benchには3つの主要な機能があります。第一に、我々は29のサイバー攻撃シナリオをカバーする拡張可能なシミュレーションプラットフォームを構築し、新しいシナリオの再実行と統合をサポートし

Machine translation (Meta M2M100); the English original follows.

Abstract

Large language models (LLMs) have shown strong potential for cybersecurity, leading to growing interest in LLM-based automated cybersecurity log analysis. However, current datasets (1) are limited in scale; (2) rely mainly on single-source evidence; and (3) have only binary or multi-class attack labels, without dense annotations of the reasoning process. To address these limitations, SecLog-Bench has three key features. First, we build an extensible simulation platform that covers 29 attack scenarios, supports repeated execution and the integration of new scenarios, and enables scalable generation of cybersecurity logs with attack-stage labels. Second, we use multi-source logs, including audit logs, application logs, and network traffic, to jointly analyze cyber attacks. Third, we annotate each sample with the essential context for cybersecurity log reasoning and a structured step-by-step evidence chain. We evaluate eight mainstream LLMs on this benchmark for post-detection attack-progress reasoning. Results show that, when provided with scaffolded evidence, current LLMs exhibit modest capability in attack-progress reasoning, with the best model achieving only 44.44% accuracy and 45.07% Macro-F1. Their reasoning remains unreliable; i.e., the highest Stage Reasoning F1 (SRF1) score is only 54.87%. Moreover, even correct predictions may be accompanied by flawed reasoning, and reasoning quality is often misaligned with prediction accuracy.

Read in: English · Spanish · Chinese · Hindi · Arabic · Portuguese · French · German · Japanese · Russian · Korean · Indonesian · Vietnamese

Topics: Computer science, Benchmark (surveying), Scalability, Context (archaeology), Key (lock), Audit

Sources: OpenAlex (CC0), arXiv, Crossref. Agent Exchange