Newest papers

VLTest: Finding Joint Image-Text Failures in Vision-Language Models

Bin Duan, Jintao Lin, Guowei Yang · 2026 · arXiv (Cornell University)

DOI: 10.48550/arxiv.2610.30794 · cited by 0 works (OpenAlex) · open-access PDF

Portuguese

Modelos de linguagem de visão (VLMs) são cada vez mais incorporados em sistemas de software. Testá-los levanta dois problemas. Mutar qualquer entrada pode alterar a resposta esperada, de modo que uma mudança de saída não é por si só um fracasso. Alguns fracassos aparecem apenas quando ambas as entradas são mutadas; atribuí-los à combinação requer testar cada mutação sozinho. Nós apresentamos o VLTest, uma abordagem de teste VLM de caixa preta. Ele gera variantes de imagem editando códigos latentes VQGAN e variantes de texto com um tubo de reescritura-expansão-prune, as variantes de filtros provavelmente mudam a resposta esperada e testam cada variante sozinho antes de testar as combinações, de modo que cada fracasso articular tem quatro

Machine translation (Meta M2M100); the English original follows.

Abstract

Vision-language models (VLMs) are increasingly embedded in software systems. Testing them raises two problems. Mutating either input can change the expected answer, so an output change is not by itself a failure. Some failures appear only when both inputs are mutated; attributing them to the combination requires testing each mutation alone. We present VLTest, a black-box VLM testing approach. It generates image variants by editing VQGAN latent codes and text variants with a rewrite-expand-prune pipeline, filters variants likely to change the expected answer, and tests each variant alone before testing combinations, so every joint failure has four matched cells. A VLM judge confirms a failure only after checking that the input keeps the expected answer. On three tasks with eight VLMs, including two commercial API models, the oracle confirms 81.7% of the candidates, with 88.8% to 94.6% precision against human labels, and four-cell confirmation rejects 22.5% of joint candidates. Under the same budget, VLTest's generators find 1.8 to 2.5 times as many confirmed failures as SINVAD and TextFooler, and its joint tests find 91 to 197 attributed joint failures per 1,000 joint calls, 1.3 times as many as the same pairing applied to the baselines' variants.

Read in: English · Spanish · Chinese · Hindi · Arabic · Portuguese · French · German · Japanese · Russian · Korean · Indonesian · Vietnamese

Topics: Joint (building), Oracle, Computer science, Software, Reliability engineering, Pairing

Sources: OpenAlex (CC0), arXiv, Crossref. Agent Exchange