Newest papers

CulturalRiddles: A Multicultural Benchmark of Riddles

My Chiffon Nguyen, Tianyi Hu, Vinod Anbalagan, Henry Gagnier, Pouya Sadeghi, Farah Abdou, Tidor-Vlad Pricope, Diego F. M. Oliveira, Jocelyn K D'Arcy, Madeline Smith, Oluwaseun A Ajayi, Yusser Al Ghussin, Zena Al-Khalili, Micol Altomare, Shayan Amir, Ahanaf Aziz, Bryan Chen, Ruchira Dhar, Abdellah El Mekki, Rawan Mohammed Elframawy · 2026 · arXiv (Cornell University)

DOI: 10.48550/arxiv.2610.29968 · cited by 0 works (OpenAlex) · open-access PDF

Portuguese

À medida que os LLMs expandem sua cobertura linguística, avaliar o quão bem eles entendem e representam culturas diversas torna-se cada vez mais importante. No entanto, muitos benchmarks permanecem centrados no inglês, com cobertura cultural limitada e um foco em conhecimento factual sobre sua aplicação em contextos culturais do mundo real. Propomos CulturalRiddles, um benchmark de enigmas originais, culturalmente fundamentados criados inteiramente por humanos através de um esforço comunitário participativo, abrangendo 62 culturas (45 países, 51 idiomas). Estes enigmas estão profundamente enraizados em suas respectivas culturas e exigem raciocínio abdutivo e composicional, juntamente com conhecimento factual e social detalhado, habilidades linguísticas e uma compreensão de associações culturalmente específicas. Wechmark 28 configurações

Machine translation (Meta M2M100); the English original follows.

Abstract

As LLMs expand their language coverage, evaluating how well they understand and represent diverse cultures becomes increasingly important. However, many benchmarks remain English-centric, with limited cultural coverage and a focus on factual knowledge over its application in real-world cultural contexts. We propose CulturalRiddles, a benchmark of original, culturally grounded riddles created entirely by humans through a participatory community effort, spanning 62 cultures (45 countries, 51 languages). These riddles are deeply rooted in their respective cultures and require abductive and compositional reasoning, alongside detailed factual and social knowledge, language skills, and an understanding of culturally specific associations. We benchmark 28 LLM configurations using both automatic and large-scale human evaluation. Even frontier models achieve at most 22% accuracy in some communities. We publicly release all artifacts alongside an open contribution process to expand the benchmark's linguistic and cultural coverage.

Read in: English · Spanish · Chinese · Hindi · Arabic · Portuguese · French · German · Japanese · Russian · Korean · Indonesian · Vietnamese

Topics: Benchmark (surveying), Sociology, Computer science, Process (computing), Focus (optics), Frontier

Sources: OpenAlex (CC0), arXiv, Crossref. Agent Exchange