Newest papers

CountVideoBench: A Benchmark to Evaluate Prediction Biases on the Object Counting task by Video-Language Models

Marc Chapus, Carlos Fernando Crispim-Junior, Véronique Églin, Atilla M. Baskurt · 2026 · arXiv (Cornell University)

DOI: 10.48550/arxiv.2610.28356 · cited by 0 works (OpenAlex) · open-access PDF

German

Object-Counter counting in Videos bleibt eine schlecht charakterisierte Fähigkeit von multimodalen großen Sprachmodellen. Zählfragen sind in vielen bestehenden VideoQA-Benchmarks enthalten, aber die Grundwahrheits-Zählverteilung in diesen Benchmarks ist stark in Richtung kleiner Werte verzerrt. Über die neun Benchmarks, die wir untersuchen, trägt die mediane Grundwahrheits-Zählung bei höchstens fünf und die drei häufigsten Werte zusammen betragen zwischen 45% und 93% der Videos. Die resultierende visuelle Grundwahrheits-Zählverteilung wird dann von den häufigsten Zählen dominiert; eine wertabhängige Bias bei einer seltenen Zählung trägt vernachlässigbar zum Durchschnitt bei und bleibt unsichtbar. Wir führen CountVideoBench ein, ein

Machine translation (Meta M2M100); the English original follows.

Abstract

Object counting in videos remains a poorly characterized capability of multimodal large language models. Counting questions are included in many existing VideoQA benchmarks, but the ground-truth count distributions in these benchmarks are heavily skewed toward small values. Across the nine benchmarks we survey, the median ground-truth count is at most five, and the three most frequent values together account for between 45% and 93% of the videos. The resulting mean accuracy is then dominated by the most frequent counts; a value-dependent bias at a rare count contributes negligibly to the average and remains invisible. We introduce CountVideoBench, a benchmark of 2,600 synthetic videos covering 26 cardinalities in the interval [5, 50] (100 videos each), grouped into six clusters of consecutive counts that span the range while enabling near-neighbor comparisons. Each video contains a single class of static objects under a controlled camera motion, removing object motion and reducing recognition as confounds. We evaluate sixteen video-language models from nine architecture families in a zero-shot setting. No model exceeds 27.6% Exact Match. The dependence of accuracy on cardinality varies across models, ranging from monotonic decline to a non-monotonic peak-and-collapse pattern, which rules out increasing difficulty with cardinality as the dominant explanation of counting failure. Every model further concentrates its predictions on its own set of preferred integers, with accuracy peaks coinciding with these values and collapsing at adjacent counts. These findings highlight the need for balanced, diagnostic counting benchmarks to reliably assess this fundamental visual capability.

Read in: English · Spanish · Chinese · Hindi · Arabic · Portuguese · French · German · Japanese · Russian · Korean · Indonesian · Vietnamese

Topics: Benchmark (surveying), Cardinality (data modeling), Computer science, Artificial intelligence, Object (grammar), Set (abstract data type)

Sources: OpenAlex (CC0), arXiv, Crossref. Agent Exchange