Newest papers

Every Part Has Its Role: Part-of-Speech Effects in LLM Post-Training

Laya Iyer, Chen Shani · 2026 · arXiv (Cornell University)

DOI: 10.48550/arxiv.2610.29843 · cited by 0 works (OpenAlex) · open-access PDF

Spanish

Los modelos de lenguaje de confianza generalmente se entrenan con predicción de token siguiente (NTP), que recompensa la predicción de tokens específicos en lugar de contabilizar explícitamente las similitudes semánticas entre diferentes palabras o expresiones. Trabajo anterior ha abordado esta limitación aumentando los datos de formación con frases mínimamente variables que sustituyen una palabra objetivo con alternativas conceptualmente relacionadas, por ejemplo, "El entrenador cansado corrió a casa" $\textitbey nouns$ {"El entrenador cansado corrió a casa", "El instructor cansado corrió a casa", ...}. Tal aumento del concepto afecta al rendimiento, pero permanece incierto si su eficacia depende de dónde en la frase se expresa el concepto. Reproducimos y extendemos el trabajo anterior $\textitbey

Machine translation (Meta M2M100); the English original follows.

Abstract

Large language models are typically trained with next-token prediction (NTP), which rewards predicting specific tokens rather than explicitly accounting for the semantic similarities between different words or expressions. Prior work has addressed this limitation by augmenting training data with minimally varying sentences that replace a target word with conceptually related alternatives, e.g., "The tired coach ran home" $\rightarrow$ {"The tired trainer ran home", "The tired instructor ran home", ...}. Such concept augmentation improves performance, but it remains unclear whether its effectiveness depends on where in the sentence the concept is expressed. We reproduce and extend prior work $\textit{beyond nouns}$ to verbs and adjectives, asking whether the part of speech (POS) carrying the concept signal affects perplexity and downstream performance. We find different patterns across evaluation settings: concept augmentation improves perplexity most consistently on news, independent of the POS used, whereas verbs provide the most reliable downstream gains. Despite these behavioral differences, representational analyses using CKA and linear probing show that the conditions yielding the largest downstream gains are not necessarily those that most alter the model's internal representations. Together, these results suggest that the effectiveness of concept augmentation depends on where and how conceptual information is introduced during training.

Read in: English · Spanish · Chinese · Hindi · Arabic · Portuguese · French · German · Japanese · Russian · Korean · Indonesian · Vietnamese

Topics: Perplexity, Natural language processing, Computer science, Sentence, Artificial intelligence, Downstream (manufacturing)

Sources: OpenAlex (CC0), arXiv, Crossref. Agent Exchange