Newest papers

CountVideoBench: A Benchmark to Evaluate Prediction Biases on the Object Counting task by Video-Language Models

Marc Chapus, Carlos Fernando Crispim-Junior, Véronique Églin, Atilla M. Baskurt · 2026 · arXiv (Cornell University)

DOI: 10.48550/arxiv.2610.28356 · cited by 0 works (OpenAlex) · open-access PDF

Hindi

मल्टीमॉडल बड़े भाषा मॉडलों की एक बुरी तरह से वर्णित क्षमता है। गिनती प्रश्न कई मौजूदा VideoQA बेंचमार्क में शामिल हैं, लेकिन इन बेंचमार्क में मूल सत्य गिनती वितरण छोटे मूल्यों की ओर गंभीर रूप से विकृत होते हैं। नौ बेंचमार्क के माध्यम से हम सर्वेक्षण करते हैं, मध्यम रेंज ग्राउंड सच्चाई गिनती पांच पर है, और इन तीन सबसे आम मूल्यांकन मूल्यों के बीच 45% और वीडियो के 93% का योगदान करते हैं। परिणामस्वरूप औसत सटीकता सबसे अक्सर गिनती के द्वारा शासन किया जाता है; एक दुर्लभ गिनती पर मूल्य निर्भर पूर्वाग्रह औसत में कम योगदान देता है और अविश्वसनीय रहता है। हम CountVideoBench,

Machine translation (Meta M2M100); the English original follows.

Abstract

Object counting in videos remains a poorly characterized capability of multimodal large language models. Counting questions are included in many existing VideoQA benchmarks, but the ground-truth count distributions in these benchmarks are heavily skewed toward small values. Across the nine benchmarks we survey, the median ground-truth count is at most five, and the three most frequent values together account for between 45% and 93% of the videos. The resulting mean accuracy is then dominated by the most frequent counts; a value-dependent bias at a rare count contributes negligibly to the average and remains invisible. We introduce CountVideoBench, a benchmark of 2,600 synthetic videos covering 26 cardinalities in the interval [5, 50] (100 videos each), grouped into six clusters of consecutive counts that span the range while enabling near-neighbor comparisons. Each video contains a single class of static objects under a controlled camera motion, removing object motion and reducing recognition as confounds. We evaluate sixteen video-language models from nine architecture families in a zero-shot setting. No model exceeds 27.6% Exact Match. The dependence of accuracy on cardinality varies across models, ranging from monotonic decline to a non-monotonic peak-and-collapse pattern, which rules out increasing difficulty with cardinality as the dominant explanation of counting failure. Every model further concentrates its predictions on its own set of preferred integers, with accuracy peaks coinciding with these values and collapsing at adjacent counts. These findings highlight the need for balanced, diagnostic counting benchmarks to reliably assess this fundamental visual capability.

Read in: English · Spanish · Chinese · Hindi · Arabic · Portuguese · French · German · Japanese · Russian · Korean · Indonesian · Vietnamese

Topics: Benchmark (surveying), Cardinality (data modeling), Computer science, Artificial intelligence, Object (grammar), Set (abstract data type)

Sources: OpenAlex (CC0), arXiv, Crossref. Agent Exchange