Newest papers

Prompt Sensitivity in Tool-Augmented Model Comparisons

Mateusz Modrzejewski, Tomasz Radzikowski · 2026 · arXiv (Cornell University)

DOI: 10.48550/arxiv.2610.31148 · cited by 0 works (OpenAlex) · open-access PDF

French

Les modèles de langue sont comparés sur les critères de référence de réponse à la question qui posent chaque question dans une seule formulation, et une question réformée peut changer lequel des deux modèles obtient un score plus élevé. Nous demandons si donner les outils de modèle rend ces comparaisons plus stables. Un gain de précision ne peut pas répondre à cela, car les outils changent plusieurs étapes entre la question et la réponse notée à la fois. Nous proposons un protocole correspondant qui sépare ces étapes: la lecture de la réponse, le raisonnement, la décision d'appeler, la réponse de l'outil et le rapport fourni sans appel. Nous l'appliquons à six modèles ouverts dans cinq domaines sous cinq formulations. Nous constatons que les outils augmentent la précision

Machine translation (Meta M2M100); the English original follows.

Abstract

Language models are compared on question-answering benchmarks that pose each question in one wording, and a reworded question can change which of two models scores higher. We ask whether giving the models tools makes such comparisons more stable. An accuracy gain cannot answer this, because tools change several steps between the question and the scored answer at once. We propose a matched protocol that separates those steps: the answer readout, reasoning, the decision to call, the tool's answer, and the report supplied without a call. We apply it to six open models in five domains under five wordings. We find that tools raise accuracy where a program answers the question and widen the wording sensitivity of many pairwise comparisons. The widening follows how often a model calls, and call frequency follows the wording for some models and not others. It also depends on the interface and the instructions: a native tool-calling interface and a second tool description leave no detected widening in molecules, and sensitivity persists where tools return intermediate results. These findings show that deterministic reports do not guarantee stable comparisons. We recommend reporting call frequency and comparison stability under several wordings beside accuracy.

Read in: English · Spanish · Chinese · Hindi · Arabic · Portuguese · French · German · Japanese · Russian · Korean · Indonesian · Vietnamese

Topics: Pairwise comparison, Ask price, Sensitivity (control systems), Stability (learning theory), Computer science, Interface (matter)

Sources: OpenAlex (CC0), arXiv, Crossref. Agent Exchange