Newest papers

Prompt Sensitivity in Tool-Augmented Model Comparisons

Mateusz Modrzejewski, Tomasz Radzikowski · 2026 · arXiv (Cornell University)

DOI: 10.48550/arxiv.2610.31148 · cited by 0 works (OpenAlex) · open-access PDF

Chinese

语言模型被比较在一个词汇中提出每个问题的回答基准上,一个重定义的问题可以更改两个模型中的哪一个得分更高。我们问,是否让模型工具使这种比较更稳定。精度增加不能回答这一点,因为工具同时在问题和得分的答案之间改变了几个步骤。我们提出了一个匹配协议,将这些步骤分开:答案阅读,推理,决定调用,工具的响应,以及没有调用提供的报告。我们将其应用于五个词汇下五个领域的六个开放模型。我们发现工具在一个程序回答问题时提高了精度,并扩大了对许多对比的语法敏感性。扩展跟随模型调用的频率,调用频率跟随某些模型的语法,而不是其他人进行比较。

Machine translation (Meta M2M100); the English original follows.

Abstract

Language models are compared on question-answering benchmarks that pose each question in one wording, and a reworded question can change which of two models scores higher. We ask whether giving the models tools makes such comparisons more stable. An accuracy gain cannot answer this, because tools change several steps between the question and the scored answer at once. We propose a matched protocol that separates those steps: the answer readout, reasoning, the decision to call, the tool's answer, and the report supplied without a call. We apply it to six open models in five domains under five wordings. We find that tools raise accuracy where a program answers the question and widen the wording sensitivity of many pairwise comparisons. The widening follows how often a model calls, and call frequency follows the wording for some models and not others. It also depends on the interface and the instructions: a native tool-calling interface and a second tool description leave no detected widening in molecules, and sensitivity persists where tools return intermediate results. These findings show that deterministic reports do not guarantee stable comparisons. We recommend reporting call frequency and comparison stability under several wordings beside accuracy.

Read in: English · Spanish · Chinese · Hindi · Arabic · Portuguese · French · German · Japanese · Russian · Korean · Indonesian · Vietnamese

Topics: Pairwise comparison, Ask price, Sensitivity (control systems), Stability (learning theory), Computer science, Interface (matter)

Sources: OpenAlex (CC0), arXiv, Crossref. Agent Exchange