Newest papers

On the Token Collapse Phenomenon in Textual Graph Soft Prompting

Andrea Giuseppe Di Francesco, Rishabh Jain, Franco Maria Nardini, Nicola Tonellotto, Pietro Liò, Fabrizio Silvestri · 2026 · arXiv (Cornell University)

DOI: 10.48550/arxiv.2610.31572 · cited by 0 works (OpenAlex) · open-access PDF

Spanish

Sin embargo, los métodos prácticos más recientes de tokenidad para inyectar la estructura de grafo en los prompts de LLM incluyen el grafo suave prompting, en el que una red neural de grafo (GNN) aprende tokens gráficos específicos que capturan información relacionada entre entidades en prompts textuales. G-Retriever combina tres benchmarks, tres codificadores de grafo, dos familias de prompts LLM, y una descripción verbalizada de las mismas condiciones de grafo y LoRA para fortalecer el mensaje de prompto textual. Realizamos un análisis en profundidad de G-Retriever y descubrimos un comportamiento patológico que es heredado por los métodos de desempeño subsiguientes, incluso con el acoplamiento de multi-token. Bajo el estándar, el

Machine translation (Meta M2M100); the English original follows.

Abstract

Recent methods for injecting graph structure into LLM prompts include graph soft prompting, in which a graph neural network (GNN) learns graph-specific tokens that capture relational information among entities in textual prompts. G-Retriever combines graph soft prompts with a verbalized description of the same graph to strengthen the textual prompt signal. We conduct an in-depth analysis of G-Retriever and uncover a pathological behavior we term token collapse. Across three benchmarks, three graph encoders, two LLM families, and both frozen and LoRA-adapted models, G-Retriever's graph tokens become almost indistinguishable across inputs. We provide evidence that the same failure is inherited by subsequent methods, even with multi-token pooling. Under the standard setup, swapping a token with the dataset mean, a token from another example, or one derived from a corrupted graph produces no resolved answer-level effect in accuracy. Moreover, a learned constant prompt that never accesses the graph matches or surpasses G-Retriever on all three benchmarks. These findings indicate that, rather than conveying graph content as intended, the collapsed token functions primarily as a generic soft prompt. We also examine conditions linked to this collapse and propose a two-stage procedure that yields non-collapsed tokens while maintaining competitive performance in most cases. However, token variability alone does not prove that the model uses graph information: via interventions and decoding probes, we show that depending on the configuration, a token can reflect the question, an answer prior, or graph-derived content that the LLM may or may not use. Overall, our results offer a practical approach to evaluating graph-token alignment rather than inferring it from task accuracy alone. Code to reproduce our experiments is available at https://github.com/difra100/Token_Collapse_KGQA.

Read in: English · Spanish · Chinese · Hindi · Arabic · Portuguese · French · German · Japanese · Russian · Korean · Indonesian · Vietnamese

Topics: Security token, Computer science, Graph, Theoretical computer science, Decoding methods, Token passing

Sources: OpenAlex (CC0), arXiv, Crossref. Agent Exchange