My Chiffon Nguyen, Tianyi Hu, Vinod Anbalagan, Henry Gagnier, Pouya Sadeghi, Farah Abdou, Tidor-Vlad Pricope, Diego F. M. Oliveira, Jocelyn K D'Arcy, Madeline Smith, Oluwaseun A Ajayi, Yusser Al Ghussin, Zena Al-Khalili, Micol Altomare, Shayan Amir, Ahanaf Aziz, Bryan Chen, Ruchira Dhar, Abdellah El Mekki, Rawan Mohammed Elframawy · 2026 · arXiv (Cornell University)
DOI: 10.48550/arxiv.2610.29968 · cited by 0 works (OpenAlex) · open-access PDF
Da LLMs ihre Sprachdeckung erweitern, wird die Bewertung, wie gut sie verschiedene Kulturen verstehen und repräsentieren, immer wichtiger. Allerdings bleiben viele Benchmarks englischzentriert, mit begrenzter kultureller Abdeckung und einem Fokus auf Tatsachenwissen über ihre Anwendung in realen kulturellen Kontexten. Wir schlagen CulturalRiddles vor, ein Benchmark für originelle, kulturell begründete Rätsel, die vollständig von Menschen durch eine partizipative Gemeinschaftsaufgabe geschaffen wurden, die 62 Kulturen (45 Länder, 51 Sprachen) umfasst. Diese Rätsel sind tief in ihren jeweiligen Kulturen verwurzelt und erfordern abstrakte und zusammensetzende Begründung, zusammen mit detailliertem Tatsachen- und sozialen Wissen, Sprach
Machine translation (Meta M2M100); the English original follows.
As LLMs expand their language coverage, evaluating how well they understand and represent diverse cultures becomes increasingly important. However, many benchmarks remain English-centric, with limited cultural coverage and a focus on factual knowledge over its application in real-world cultural contexts. We propose CulturalRiddles, a benchmark of original, culturally grounded riddles created entirely by humans through a participatory community effort, spanning 62 cultures (45 countries, 51 languages). These riddles are deeply rooted in their respective cultures and require abductive and compositional reasoning, alongside detailed factual and social knowledge, language skills, and an understanding of culturally specific associations. We benchmark 28 LLM configurations using both automatic and large-scale human evaluation. Even frontier models achieve at most 22% accuracy in some communities. We publicly release all artifacts alongside an open contribution process to expand the benchmark's linguistic and cultural coverage.
Read in: English · Spanish · Chinese · Hindi · Arabic · Portuguese · French · German · Japanese · Russian · Korean · Indonesian · Vietnamese
Topics: Benchmark (surveying), Sociology, Computer science, Process (computing), Focus (optics), Frontier
GET https://store.agentexchange.work/gw/papers/verify?title=...&year=... (0.003 USD), fetch this record as JSON with /gw/papers/paper?id=2610.29968, or list what it cites with /gw/papers/references?id=.... Pay per call in USDC (x402) or with a 19 USD card key for 1,000 calls. Catalogue · llms.txtSources: OpenAlex (CC0), arXiv, Crossref. Agent Exchange