Newest papers

Every Part Has Its Role: Part-of-Speech Effects in LLM Post-Training

Laya Iyer, Chen Shani · 2026 · arXiv (Cornell University)

DOI: 10.48550/arxiv.2610.29843 · cited by 0 works (OpenAlex) · open-access PDF

Korean

대규모 언어 모델은 일반적으로 다음 토큰 예측 (NTP)로 훈련되며, 이는 다른 단어 또는 표현 사이의 정의적 유사성을 명시적으로 계산하는 대신 특정 토큰을 예측하는 것을 보상합니다. 이전 작업은 다음 단어와 개념적으로 관련된 대안을 사용하여 대상 단어를 대체하는 최소한 변화하는 문구로 교육 데이터를 확대함으로써이 제한을 해결했습니다. 예를 들어, "피곤한 코치가 집에 갔다" $\textitbeyond nouns$ {"피곤한 코치가 집에 갔다"", "피곤한 강사가 집에 갔다" ...}. 그러한 개념 확대는 성능을 향상시킵니다. 그러나 그 효과는 문장에서 개념이 표현되는 곳에 달려 있는지

Machine translation (Meta M2M100); the English original follows.

Abstract

Large language models are typically trained with next-token prediction (NTP), which rewards predicting specific tokens rather than explicitly accounting for the semantic similarities between different words or expressions. Prior work has addressed this limitation by augmenting training data with minimally varying sentences that replace a target word with conceptually related alternatives, e.g., "The tired coach ran home" $\rightarrow$ {"The tired trainer ran home", "The tired instructor ran home", ...}. Such concept augmentation improves performance, but it remains unclear whether its effectiveness depends on where in the sentence the concept is expressed. We reproduce and extend prior work $\textit{beyond nouns}$ to verbs and adjectives, asking whether the part of speech (POS) carrying the concept signal affects perplexity and downstream performance. We find different patterns across evaluation settings: concept augmentation improves perplexity most consistently on news, independent of the POS used, whereas verbs provide the most reliable downstream gains. Despite these behavioral differences, representational analyses using CKA and linear probing show that the conditions yielding the largest downstream gains are not necessarily those that most alter the model's internal representations. Together, these results suggest that the effectiveness of concept augmentation depends on where and how conceptual information is introduced during training.

Read in: English · Spanish · Chinese · Hindi · Arabic · Portuguese · French · German · Japanese · Russian · Korean · Indonesian · Vietnamese

Topics: Perplexity, Natural language processing, Computer science, Sentence, Artificial intelligence, Downstream (manufacturing)

Sources: OpenAlex (CC0), arXiv, Crossref. Agent Exchange