Newest papers

Africa-Papers: A 10-Billion-Token Multi-Domain Academic Corpus for African Foundation Language Models

Mich-Seth Owusu · 2026 · arXiv (Cornell University)

DOI: 10.48550/arxiv.2610.28173 · cited by 0 works (OpenAlex) · open-access PDF

Korean

대규모 도메인 인덱스 도메인 (LLMs) 데이터는 기본적으로 그들의 사전 훈련 단체의 다양성, 깊이 및 사실적 엄격성에 의존합니다. 그러나 현대의 기본적인 사전 훈련 데이터 세트 (예 : The Pile, S2ORC, Dolma)는 열대적 인 전염병학, 지역화 된 소규모 농업 농업 생태학, 지역 경제 지배 및 원주민 언어 구조에 대한 심각한 구조적 부재를 보여줍니다.이 글로벌 사전 훈련 토큰의 0.5 % 미만을 구성합니다. 결과적으로 아프리카 사전 훈련 토큰을 해결하고 인공 지능에 대한 아프리카 원주민 주권을 발전시키기 위해 우리는 아프리카의 프레젠테이션 부족을 해결하기 위해, 가장 큰

Machine translation (Meta M2M100); the English original follows.

Abstract

Large Language Models (LLMs) depend fundamentally on the diversity, depth, and factual rigor of their pre-training corpora. However, contemporary foundational pre-training datasets (such as The Pile, S2ORC, and Dolma) exhibit a severe structural omission: African scientific, medical, legal, and agricultural scholarship constitutes less than 0.5% of global pre-training tokens. Consequently, state-of-the-art models suffer from systematic failure modes when reasoning about tropical epidemiology, localized smallholder agro-ecologies, regional economic governance, and indigenous linguistic structures. To resolve this representation deficit and advance African intellectual sovereignty in artificial intelligence, we introduce Africa-Papers, the largest curated multi-domain Pan-African academic pre-training corpus to date. Comprising over 844,807 full-text papers spanning 10.15 billion tokens across 879 Parquet shards, Africa-Papers unifies four primary scientific streams: (1) continental peer-reviewed journals indexed in DOAJ across 31 African nations, (2) the CGIAR CGSpace agricultural repository, (3) African-authored and African-focused STEM/AI preprints on arXiv, and (4) open-access international publications by scholars affiliated with African institutions indexed via OpenAlex. The entire corpus is processed into layout-aware, clean Markdown preserving section hierarchies, mathematical equations, and tabular data, accompanied by exact BPE token counts and a standardized 8-domain taxonomy. We release Africa-Papers openly on the Hugging Face Hub under the AfriSpeech organization to accelerate continual pre-training and domain evaluation for foundation models globally.

Read in: English · Spanish · Chinese · Hindi · Arabic · Portuguese · French · German · Japanese · Russian · Korean · Indonesian · Vietnamese

Topics: Representation (politics), Scholarship, Foundation (evidence), Indigenous, Linguistics, Languages of Africa

Sources: OpenAlex (CC0), arXiv, Crossref. Agent Exchange