Newest papers

recAMPLIFY: Large-Scale Pretraining of Recursive Protein Language Models

Saurav Jha, Qianli Liao, Robert Vernon, Sarath Chandar, Quentin Fournier · 2026 · arXiv (Cornell University)

DOI: 10.48550/arxiv.2610.26747 · cited by 0 works (OpenAlex) · open-access PDF

Chinese

这种单一的基因层生物学控制模型(PLMs)是对蛋白质注释和突变效应预测的强大工具,但进展在很大程度上取决于扩展模型大小和数据,使它们昂贵,容易记忆。测试时间扩展提供了补充途径,但用于自动退缩的LLMs所开发的方法不会直接转移:隐藏的编码器缺乏生成路径,而氨基酸不会用推理语言进行语法化。相反,我们将回归预训练适应于掩盖的PLMs,其中重量绑定的Transformer核心在可调节的周期内改进了剩余排放的隐藏状态。我们将此配方作为PLAMPLIFY实例化,而没有使用155M独特参数的PLM进行语法化

Machine translation (Meta M2M100); the English original follows.

Abstract

Protein language models (pLMs) are powerful tools for protein annotation and mutation-effect prediction, but progress has largely depended on scaling model size and data, making them expensive and prone to memorization. Test-time scaling offers a complementary path, but methods developed for autoregressive LLMs do not transfer directly: masked encoders lack a generation loop, and amino acids do not verbalize in a reasoning language. Instead, we adapt recursive pretraining to masked pLMs, where a weight-tied Transformer core refines residue-aligned hidden states over an adjustable number of cycles. We instantiate this recipe as recAMPLIFY, a pLM with 155M unique parameters trained on the same 2T tokens as AMPLIFY-350M. Despite 56% fewer parameters, it improves CASP contact recovery and three out of four supervised tasks, notably raising fold accuracy by 22.5%. Because optimal depth for recAMPLIFY varies across tasks and assays, we then introduce Gated Mixture Recurrence (GMR), a framework that decodes each protein from a probability-weighted mixture of hidden states visited during recursion. GMR employs a learned residual gate at scalar, low-rank, or per-dimension resolution to control how much of each cycle's proposed update is accepted into the state. This unifies halting and gating into a single adaptive computation objective while recovering PonderNet-style halting as a special case without gates. Layered on top of recAMPLIFY, GMR further reduces expected layer depth by up to 37%, although the accuracy trade-off varies by task. Mechanistic analyses show that residual gating controls recurrence stability beyond the trained depth and that halt depth tracks biological properties.

Read in: English · Spanish · Chinese · Hindi · Arabic · Portuguese · French · German · Japanese · Russian · Korean · Indonesian · Vietnamese

Topics: Computer science, Artificial intelligence, Discriminator, Language model, CASP, Residual

Sources: OpenAlex (CC0), arXiv, Crossref. Agent Exchange