Newest papers

VLTest: Finding Joint Image-Text Failures in Vision-Language Models

Bin Duan, Jintao Lin, Guowei Yang · 2026 · arXiv (Cornell University)

DOI: 10.48550/arxiv.2610.30794 · cited by 0 works (OpenAlex) · open-access PDF

German

Vision-Sprachen-Modelle (VLMs) werden zunehmend in Software-Systeme eingebettet. Wenn sie getestet werden, entstehen zwei Probleme. Mutation eines Eingangs kann die erwartete Antwort ändern, so dass eine Änderung der Ausgabe nicht selbst ein Misserfolg ist. Einige Misserfolge erscheinen nur, wenn beide Eingänge mutiert werden; wenn sie der Kombination zugeordnet werden, muss jede Mutation allein getestet werden. Wir präsentieren VLTest, einen schwarzen VLM-Testansatz. Es erzeugt Bildvarianten, indem es VQGAN-latente Codes und Textvarianten mit einer rewrite-expand-prune-Pipeline bearbeitet. Filtervarianten werden wahrscheinlich die erwartete Antwort ändern, und jede Variante wird vor dem Testen von

Machine translation (Meta M2M100); the English original follows.

Abstract

Vision-language models (VLMs) are increasingly embedded in software systems. Testing them raises two problems. Mutating either input can change the expected answer, so an output change is not by itself a failure. Some failures appear only when both inputs are mutated; attributing them to the combination requires testing each mutation alone. We present VLTest, a black-box VLM testing approach. It generates image variants by editing VQGAN latent codes and text variants with a rewrite-expand-prune pipeline, filters variants likely to change the expected answer, and tests each variant alone before testing combinations, so every joint failure has four matched cells. A VLM judge confirms a failure only after checking that the input keeps the expected answer. On three tasks with eight VLMs, including two commercial API models, the oracle confirms 81.7% of the candidates, with 88.8% to 94.6% precision against human labels, and four-cell confirmation rejects 22.5% of joint candidates. Under the same budget, VLTest's generators find 1.8 to 2.5 times as many confirmed failures as SINVAD and TextFooler, and its joint tests find 91 to 197 attributed joint failures per 1,000 joint calls, 1.3 times as many as the same pairing applied to the baselines' variants.

Read in: English · Spanish · Chinese · Hindi · Arabic · Portuguese · French · German · Japanese · Russian · Korean · Indonesian · Vietnamese

Topics: Joint (building), Oracle, Computer science, Software, Reliability engineering, Pairing

Sources: OpenAlex (CC0), arXiv, Crossref. Agent Exchange