Newest papers

Every Part Has Its Role: Part-of-Speech Effects in LLM Post-Training

Laya Iyer, Chen Shani · 2026 · arXiv (Cornell University)

DOI: 10.48550/arxiv.2610.29843 · cited by 0 works (OpenAlex) · open-access PDF

Japanese

大規模な言語モデルは、通常、次なるトークン予測(NTP)で訓練されており、具体的なトークンを予測する代わりに明示的に異なる単語や表現の間のセマンティックな類似性を考慮することである。以前の作業は、トレーニングデータを最小限に異なる文で増やすことで、ターゲットワードを概念的に関連する代替案に置き換えることで訓練されたが、例えば、「疲れたトレーナーが帰宅した」と「疲れたトレーナーが帰宅した」と「疲れたインストラクターが帰宅した」と尋ねている。このようなコンセプトの増加はパフォーマンスを向上させているが、その有効性が文句で概念が表現されている場所に依存するかどうかは不明である。

Machine translation (Meta M2M100); the English original follows.

Abstract

Large language models are typically trained with next-token prediction (NTP), which rewards predicting specific tokens rather than explicitly accounting for the semantic similarities between different words or expressions. Prior work has addressed this limitation by augmenting training data with minimally varying sentences that replace a target word with conceptually related alternatives, e.g., "The tired coach ran home" $\rightarrow$ {"The tired trainer ran home", "The tired instructor ran home", ...}. Such concept augmentation improves performance, but it remains unclear whether its effectiveness depends on where in the sentence the concept is expressed. We reproduce and extend prior work $\textit{beyond nouns}$ to verbs and adjectives, asking whether the part of speech (POS) carrying the concept signal affects perplexity and downstream performance. We find different patterns across evaluation settings: concept augmentation improves perplexity most consistently on news, independent of the POS used, whereas verbs provide the most reliable downstream gains. Despite these behavioral differences, representational analyses using CKA and linear probing show that the conditions yielding the largest downstream gains are not necessarily those that most alter the model's internal representations. Together, these results suggest that the effectiveness of concept augmentation depends on where and how conceptual information is introduced during training.

Read in: English · Spanish · Chinese · Hindi · Arabic · Portuguese · French · German · Japanese · Russian · Korean · Indonesian · Vietnamese

Topics: Perplexity, Natural language processing, Computer science, Sentence, Artificial intelligence, Downstream (manufacturing)

Sources: OpenAlex (CC0), arXiv, Crossref. Agent Exchange