Newest papers

Prompt Sensitivity in Tool-Augmented Model Comparisons

Mateusz Modrzejewski, Tomasz Radzikowski · 2026 · arXiv (Cornell University)

DOI: 10.48550/arxiv.2610.31148 · cited by 0 works (OpenAlex) · open-access PDF

Indonesian

Model bahasa dibandingkan dengan referensi jawaban pertanyaan yang mengajukan setiap pertanyaan dalam satu formulasi, dan pertanyaan reword dapat mengubah mana dari dua model skor lebih tinggi. Kami bertanya apakah memberikan alat model membuat perbandingan tersebut lebih stabil. Peningkatan akurasi tidak dapat menjawab ini, karena alat-alat mengubah beberapa langkah antara pertanyaan dan jawaban skor sekaligus. Kami mengusulkan protokol yang cocok yang memisahkan langkah-langkah tersebut: jawaban membaca, alasan, keputusan untuk memanggil, jawaban alat, dan laporan yang disampaikan tanpa panggilan. Kami menerapkannya pada enam model terbuka di lima domain di bawah lima formulasi. Kami menemukan bahwa alat-alat meningkatkan akurasi di mana sebuah program menjawab pertanyaan dan memperluas sensitivitas formulasi dari banyak perbandingan. Peningkatan mengikuti seberapa sering panggilan model, dan frekuensi panggilan mengikuti formulasi untuk beberapa model dan tidak lain. Ini juga tergantung pada antarmuka dan instruksi

Machine translation (Meta M2M100); the English original follows.

Abstract

Language models are compared on question-answering benchmarks that pose each question in one wording, and a reworded question can change which of two models scores higher. We ask whether giving the models tools makes such comparisons more stable. An accuracy gain cannot answer this, because tools change several steps between the question and the scored answer at once. We propose a matched protocol that separates those steps: the answer readout, reasoning, the decision to call, the tool's answer, and the report supplied without a call. We apply it to six open models in five domains under five wordings. We find that tools raise accuracy where a program answers the question and widen the wording sensitivity of many pairwise comparisons. The widening follows how often a model calls, and call frequency follows the wording for some models and not others. It also depends on the interface and the instructions: a native tool-calling interface and a second tool description leave no detected widening in molecules, and sensitivity persists where tools return intermediate results. These findings show that deterministic reports do not guarantee stable comparisons. We recommend reporting call frequency and comparison stability under several wordings beside accuracy.

Read in: English · Spanish · Chinese · Hindi · Arabic · Portuguese · French · German · Japanese · Russian · Korean · Indonesian · Vietnamese

Topics: Pairwise comparison, Ask price, Sensitivity (control systems), Stability (learning theory), Computer science, Interface (matter)

Sources: OpenAlex (CC0), arXiv, Crossref. Agent Exchange