Newest papers

VLTest: Finding Joint Image-Text Failures in Vision-Language Models

Bin Duan, Jintao Lin, Guowei Yang · 2026 · arXiv (Cornell University)

DOI: 10.48550/arxiv.2610.30794 · cited by 0 works (OpenAlex) · open-access PDF

Spanish

Los modelos de lenguaje de visión (VLM) se incorporan cada vez más en los sistemas de software. Testarlos plantea dos problemas. Mutar una entrada puede cambiar la respuesta esperada, por lo que un cambio de salida no es por sí mismo un fracaso. Algunos fracasos aparecen sólo cuando ambas entradas son mutadas; atribuirlos a la combinación requiere probar cada mutación por sí sola. Presentamos VLTest, un enfoque de prueba VLM de caja negra. Se genera variantes de imagen al editar los códigos latentes de VQGAN y variantes de texto con un tubo reescrito-expandido-prune, las variantes de filtros son susceptibles de cambiar la respuesta esperada, y cada variante se prueba por sí sola antes de probar las combinaciones, por lo que cada fracaso de la artic

Machine translation (Meta M2M100); the English original follows.

Abstract

Vision-language models (VLMs) are increasingly embedded in software systems. Testing them raises two problems. Mutating either input can change the expected answer, so an output change is not by itself a failure. Some failures appear only when both inputs are mutated; attributing them to the combination requires testing each mutation alone. We present VLTest, a black-box VLM testing approach. It generates image variants by editing VQGAN latent codes and text variants with a rewrite-expand-prune pipeline, filters variants likely to change the expected answer, and tests each variant alone before testing combinations, so every joint failure has four matched cells. A VLM judge confirms a failure only after checking that the input keeps the expected answer. On three tasks with eight VLMs, including two commercial API models, the oracle confirms 81.7% of the candidates, with 88.8% to 94.6% precision against human labels, and four-cell confirmation rejects 22.5% of joint candidates. Under the same budget, VLTest's generators find 1.8 to 2.5 times as many confirmed failures as SINVAD and TextFooler, and its joint tests find 91 to 197 attributed joint failures per 1,000 joint calls, 1.3 times as many as the same pairing applied to the baselines' variants.

Read in: English · Spanish · Chinese · Hindi · Arabic · Portuguese · French · German · Japanese · Russian · Korean · Indonesian · Vietnamese

Topics: Joint (building), Oracle, Computer science, Software, Reliability engineering, Pairing

Sources: OpenAlex (CC0), arXiv, Crossref. Agent Exchange