My Chiffon Nguyen, Tianyi Hu, Vinod Anbalagan, Henry Gagnier, Pouya Sadeghi, Farah Abdou, Tidor-Vlad Pricope, Diego F. M. Oliveira, Jocelyn K D'Arcy, Madeline Smith, Oluwaseun A Ajayi, Yusser Al Ghussin, Zena Al-Khalili, Micol Altomare, Shayan Amir, Ahanaf Aziz, Bryan Chen, Ruchira Dhar, Abdellah El Mekki, Rawan Mohammed Elframawy · 2026 · arXiv (Cornell University)
DOI: 10.48550/arxiv.2610.29968 · cited by 0 works (OpenAlex) · open-access PDF
À mesure que les LLM élargissent leur couverture linguistique, l'évaluation de la façon dont ils comprennent et représentent les cultures diverses devient de plus en plus importante. Cependant, de nombreux critères de référence restent centrés sur l'anglais, avec une couverture culturelle limitée et un accent sur les connaissances factuelles sur son application dans les contextes culturels du monde réel. Nous proposons CulturalRiddles, un critère de référence des énigmes originales et culturellement fondées créées entièrement par les humains à travers un effort communautaire participatif, couvrant 62 cultures (45 pays, 51 langues). Ces énigmes sont profondément enracinées dans leurs cultures respectives et nécessitent un raisonnement abductif et compositif, ainsi que des connaissances factuelles
Machine translation (Meta M2M100); the English original follows.
As LLMs expand their language coverage, evaluating how well they understand and represent diverse cultures becomes increasingly important. However, many benchmarks remain English-centric, with limited cultural coverage and a focus on factual knowledge over its application in real-world cultural contexts. We propose CulturalRiddles, a benchmark of original, culturally grounded riddles created entirely by humans through a participatory community effort, spanning 62 cultures (45 countries, 51 languages). These riddles are deeply rooted in their respective cultures and require abductive and compositional reasoning, alongside detailed factual and social knowledge, language skills, and an understanding of culturally specific associations. We benchmark 28 LLM configurations using both automatic and large-scale human evaluation. Even frontier models achieve at most 22% accuracy in some communities. We publicly release all artifacts alongside an open contribution process to expand the benchmark's linguistic and cultural coverage.
Read in: English · Spanish · Chinese · Hindi · Arabic · Portuguese · French · German · Japanese · Russian · Korean · Indonesian · Vietnamese
Topics: Benchmark (surveying), Sociology, Computer science, Process (computing), Focus (optics), Frontier
GET https://store.agentexchange.work/gw/papers/verify?title=...&year=... (0.003 USD), fetch this record as JSON with /gw/papers/paper?id=2610.29968, or list what it cites with /gw/papers/references?id=.... Pay per call in USDC (x402) or with a 19 USD card key for 1,000 calls. Catalogue · llms.txtSources: OpenAlex (CC0), arXiv, Crossref. Agent Exchange