Newest papers

Foreword: Pre-Generation Prediction and Characterisation of LLM Test Failures using the Jacobian Lens

Somin Kim, ROBERT H. FELDT, Shin Yoo · 2026 · arXiv (Cornell University)

DOI: 10.48550/arxiv.2610.28187 · cited by 0 works (OpenAlex) · open-access PDF

Portuguese

Testar semartures sem pre-geração de software baseado em laboratório é caro: cada entrada precisa de várias respostas sampled, cada um julgado por um oráculo caro. Além disso, cada prompt ou mudança de modelo chama para re-teste. Medidas de adequação pré-geração cortar esse custo por classificação de entradas por risco de falha com base no estado interno do modelo. No entanto, o estado da arte, Clotho, modelos passando apenas entradas de referência estudo, e sua representação oculto-estado não diz nada sobre por que as entradas falham. Nós apresentamos Foreword, uma estrutura específica construída em duas ideias: aprender a partir de ambos passando e falhar referências, e representar entradas através da lente Jacobian (J-lens), uma técnica de interpretação que lê ativações intermediárias como pon

Machine translation (Meta M2M100); the English original follows.

Abstract

Testing LLM-based software is expensive: each input needs several sampled responses, each judged by a costly oracle. Further, every prompt or model change calls for re-testing. Pre-generation adequacy measures cut this cost by ranking inputs by failure risk based on the model's internal state. However, the state of the art, Clotho, models only passing reference inputs, and its hidden-state representation says nothing about why inputs fail. We present Foreword, a framework built on two ideas: learn from both passing and failing references, and represent inputs through the Jacobian lens (J-lens), an interpretability technique that reads intermediate activations out as scores over vocabulary tokens. Foreword pairs these representations with two estimators: a sparse logistic-regression probe or dual Gaussian Mixture Models (dual GMMs), which also yield a reference density (to flag inputs the references do not cover) and failing-class components (to group similar failures). On ten tasks and three open-weight LLMs, all four combinations of representations and estimators outperform Clotho on average, both for prediction of failure (ROC-AUC improves by 0.14 on average and by up to 0.31) and test prioritisation (15% more failures among the 100 top-ranked inputs on average, up to 33%). Prioritisation obtained using small models also transfers better to proprietary models (on average 22% and up to 41% more failures in the first quarter of prioritised inputs). Importantly, our qualitative case study shows that Foreword supports semantic characterisation of failures by pointing to ambiguities in labels or task specifications. Combined, Foreword can support testers for prioritising inputs, checking support for a prediction, and characterising specific failures, all without generating output.

Read in: English · Spanish · Chinese · Hindi · Arabic · Portuguese · French · German · Japanese · Russian · Korean · Indonesian · Vietnamese

Topics: Interpretability, Computer science, Task (project management), Jacobian matrix and determinant, Artificial intelligence, Ranking (information retrieval)

Sources: OpenAlex (CC0), arXiv, Crossref. Agent Exchange