Newest papers

The Development of Sycophancy in Language Models: Tracing Vulnerabilities from Pretraining to Post-Training

Haonan Huang, Chengkun Cai · 2026 · arXiv (Cornell University)

DOI: 10.48550/arxiv.2610.26655 · cited by 0 works (OpenAlex) · open-access PDF

Chinese

语言模型可以改变一个承诺的答案,以适应用户的意见或反馈,而没有新的证据。然而,当这种行为在训练过程中发展,一个发布模型从早期阶段继承的东西仍然不清楚。我们跟踪了30个开放模型线的741个检查点,从早期预训练到发布模型。我们在道德场景上制定初步判断,并测量了模型在训练期间跟随用户的观点或放弃其判断的程度。在大多数线中,行为模式在训练后存在,其平均强度在两个方向发生变化,在各种线路之间不同阶段。一个更清晰的模式出现在个别问题的层面上。在没有压力的答案之外,预测行为预测了一个发布模型在压力下产生的问题很少。在训练后形成的模式:在沿着受监督

Machine translation (Meta M2M100); the English original follows.

Abstract

Language models can change a committed answer to accommodate a user's opinion or pushback without new evidence. However, when this behaviour develops during training and what a released model inherits from earlier stages remain unclear. We follow 741 checkpoints of 30 open model lineages, from early pretraining to released models. We plant initial verdicts on moral scenarios and measure how far a model follows a user's stated opinion or abandons its verdict under pressure. In most lineages the behaviour is present before post-training, and its average strength changes in either direction, at different stages across lineages. A clearer pattern emerges at the level of individual questions. Beyond responses without pressure, pretraining behaviour predicts little of which questions a released model yields on. That pattern forms during post-training: on routes with supervised fine-tuning, the fine-tuning checkpoint predicts part of it and the preference-optimization checkpoint most of it. A sibling trained from the same base along another route predicts far less: for Tülu-3 8B, its own preference-stage checkpoint removes 85% of the remaining prediction error, a sibling 19%. Internal states add predictive information without changing this ranking. Similar average scores conceal different vulnerabilities, and training history matters for auditing a released model.

Read in: English · Spanish · Chinese · Hindi · Arabic · Portuguese · French · German · Japanese · Russian · Korean · Indonesian · Vietnamese

Topics: Computer science, Measure (data warehouse), Psychology, Audit, Artificial intelligence, Cognitive psychology

Sources: OpenAlex (CC0), arXiv, Crossref. Agent Exchange