Newest papers

CulturalRiddles: A Multicultural Benchmark of Riddles

My Chiffon Nguyen, Tianyi Hu, Vinod Anbalagan, Henry Gagnier, Pouya Sadeghi, Farah Abdou, Tidor-Vlad Pricope, Diego F. M. Oliveira, Jocelyn K D'Arcy, Madeline Smith, Oluwaseun A Ajayi, Yusser Al Ghussin, Zena Al-Khalili, Micol Altomare, Shayan Amir, Ahanaf Aziz, Bryan Chen, Ruchira Dhar, Abdellah El Mekki, Rawan Mohammed Elframawy · 2026 · arXiv (Cornell University)

DOI: 10.48550/arxiv.2610.29968 · cited by 0 works (OpenAlex) · open-access PDF

Spanish

A medida que los LLM amplían su cobertura lingüística, la evaluación de lo bien que comprenden y representan culturas diversas se vuelve cada vez más importante. Sin embargo, muchos parámetros se mantienen centrados en inglés, con cobertura cultural limitada y un enfoque en el conocimiento factual sobre su aplicación en contextos culturales del mundo real. Proponemos CulturalRiddles, un parámetro de puzzles originales, culturalmente fundamentados creados enteramente por los humanos a través de un esfuerzo comunitario participativo, que abarca a 62 culturas (45 países, 51 idiomas). Estos parámetros están profundamente arraigados en sus respectivas culturas y requieren razonamiento abductivo y de composición, junto con un conocimiento factual y social detallado, habilidades lingüísticas y una comprensión de asociaciones culturalmente específicas. Wech

Machine translation (Meta M2M100); the English original follows.

Abstract

As LLMs expand their language coverage, evaluating how well they understand and represent diverse cultures becomes increasingly important. However, many benchmarks remain English-centric, with limited cultural coverage and a focus on factual knowledge over its application in real-world cultural contexts. We propose CulturalRiddles, a benchmark of original, culturally grounded riddles created entirely by humans through a participatory community effort, spanning 62 cultures (45 countries, 51 languages). These riddles are deeply rooted in their respective cultures and require abductive and compositional reasoning, alongside detailed factual and social knowledge, language skills, and an understanding of culturally specific associations. We benchmark 28 LLM configurations using both automatic and large-scale human evaluation. Even frontier models achieve at most 22% accuracy in some communities. We publicly release all artifacts alongside an open contribution process to expand the benchmark's linguistic and cultural coverage.

Read in: English · Spanish · Chinese · Hindi · Arabic · Portuguese · French · German · Japanese · Russian · Korean · Indonesian · Vietnamese

Topics: Benchmark (surveying), Sociology, Computer science, Process (computing), Focus (optics), Frontier

Sources: OpenAlex (CC0), arXiv, Crossref. Agent Exchange