Newest papers

On the Token Collapse Phenomenon in Textual Graph Soft Prompting

Andrea Giuseppe Di Francesco, Rishabh Jain, Franco Maria Nardini, Nicola Tonellotto, Pietro Liò, Fabrizio Silvestri · 2026 · arXiv (Cornell University)

DOI: 10.48550/arxiv.2610.31572 · cited by 0 works (OpenAlex) · open-access PDF

French

Cependant, les méthodes récentes de token de décomposabilité pour injecter cette structure graphique dans les prompts LLM comprennent le graph soft prompting, dans lequel un réseau neuronal graphique (GNN) apprend des tokens spécifiques aux questions graphiques qui capturent des informations relationnelles entre les entités dans les prompts textuels. G-Retriever combine les prompts graphiques avec une description verbalisée des mêmes conditions graphiques pour renforcer le signal de prompt textuel. Nous effectuons une analyse en profondeur de G-Retriever et découvrons une procédure pathologique qui est héritée de la configuration des performances ultérieures que nous appelons l'effondrement des tokens. Au-delà de trois benchmarks, trois encoders gra

Machine translation (Meta M2M100); the English original follows.

Abstract

Recent methods for injecting graph structure into LLM prompts include graph soft prompting, in which a graph neural network (GNN) learns graph-specific tokens that capture relational information among entities in textual prompts. G-Retriever combines graph soft prompts with a verbalized description of the same graph to strengthen the textual prompt signal. We conduct an in-depth analysis of G-Retriever and uncover a pathological behavior we term token collapse. Across three benchmarks, three graph encoders, two LLM families, and both frozen and LoRA-adapted models, G-Retriever's graph tokens become almost indistinguishable across inputs. We provide evidence that the same failure is inherited by subsequent methods, even with multi-token pooling. Under the standard setup, swapping a token with the dataset mean, a token from another example, or one derived from a corrupted graph produces no resolved answer-level effect in accuracy. Moreover, a learned constant prompt that never accesses the graph matches or surpasses G-Retriever on all three benchmarks. These findings indicate that, rather than conveying graph content as intended, the collapsed token functions primarily as a generic soft prompt. We also examine conditions linked to this collapse and propose a two-stage procedure that yields non-collapsed tokens while maintaining competitive performance in most cases. However, token variability alone does not prove that the model uses graph information: via interventions and decoding probes, we show that depending on the configuration, a token can reflect the question, an answer prior, or graph-derived content that the LLM may or may not use. Overall, our results offer a practical approach to evaluating graph-token alignment rather than inferring it from task accuracy alone. Code to reproduce our experiments is available at https://github.com/difra100/Token_Collapse_KGQA.

Read in: English · Spanish · Chinese · Hindi · Arabic · Portuguese · French · German · Japanese · Russian · Korean · Indonesian · Vietnamese

Topics: Security token, Computer science, Graph, Theoretical computer science, Decoding methods, Token passing

Sources: OpenAlex (CC0), arXiv, Crossref. Agent Exchange