Newest papers

Africa-Papers: A 10-Billion-Token Multi-Domain Academic Corpus for African Foundation Language Models

Mich-Seth Owusu · 2026 · arXiv (Cornell University)

DOI: 10.48550/arxiv.2610.28173 · cited by 0 works (OpenAlex) · open-access PDF

Japanese

Large Language Indexing Models (LLMs) を通じた大規模なトークレイリングドメインは、データの多様性、深さ、および事実の厳しさを基本的に依存しています。 しかし、現代の基本的なプレトレーニングデータセット(The Pile、S2ORC、Dolmaなど)は、熱帯流行病学、小規模農業生産農業生態学、地域経済統治、およびアフリカの先住民の言語学的構造について論争する際の体系的な失敗モードに苦しんでいます:アフリカの科学、法律、および農業学的学習は、世界のプレトレーニングトークレイリングトークンの0.5%未満を構成しています。結果として、世界的なプレトレーニングトークレイリングトークレイ

Machine translation (Meta M2M100); the English original follows.

Abstract

Large Language Models (LLMs) depend fundamentally on the diversity, depth, and factual rigor of their pre-training corpora. However, contemporary foundational pre-training datasets (such as The Pile, S2ORC, and Dolma) exhibit a severe structural omission: African scientific, medical, legal, and agricultural scholarship constitutes less than 0.5% of global pre-training tokens. Consequently, state-of-the-art models suffer from systematic failure modes when reasoning about tropical epidemiology, localized smallholder agro-ecologies, regional economic governance, and indigenous linguistic structures. To resolve this representation deficit and advance African intellectual sovereignty in artificial intelligence, we introduce Africa-Papers, the largest curated multi-domain Pan-African academic pre-training corpus to date. Comprising over 844,807 full-text papers spanning 10.15 billion tokens across 879 Parquet shards, Africa-Papers unifies four primary scientific streams: (1) continental peer-reviewed journals indexed in DOAJ across 31 African nations, (2) the CGIAR CGSpace agricultural repository, (3) African-authored and African-focused STEM/AI preprints on arXiv, and (4) open-access international publications by scholars affiliated with African institutions indexed via OpenAlex. The entire corpus is processed into layout-aware, clean Markdown preserving section hierarchies, mathematical equations, and tabular data, accompanied by exact BPE token counts and a standardized 8-domain taxonomy. We release Africa-Papers openly on the Hugging Face Hub under the AfriSpeech organization to accelerate continual pre-training and domain evaluation for foundation models globally.

Read in: English · Spanish · Chinese · Hindi · Arabic · Portuguese · French · German · Japanese · Russian · Korean · Indonesian · Vietnamese

Topics: Representation (politics), Scholarship, Foundation (evidence), Indigenous, Linguistics, Languages of Africa

Sources: OpenAlex (CC0), arXiv, Crossref. Agent Exchange