Newest papers

recAMPLIFY: Large-Scale Pretraining of Recursive Protein Language Models

Saurav Jha, Qianli Liao, Robert Vernon, Sarath Chandar, Quentin Fournier · 2026 · arXiv (Cornell University)

DOI: 10.48550/arxiv.2610.26747 · cited by 0 works (OpenAlex) · open-access PDF

French

Les modèles de contrôle de la langue protéinée de base (PLM) sont des outils puissants pour l'annotation des protéines et la prédiction des effets de la mutation, mais les progrès dépendent en grande partie de l'échelle du modèle et des données, ce qui les rend coûteux et prédisposés à la mémorisation. L'échelle de test offre un chemin complémentaire, mais les méthodes développées pour les LLM autorégressives ne se transfèrent pas directement: les encodeurs masqués manquent d'une boucle de génération, et les acides aminés ne se verbalisent pas dans un langage de raisonnement. Au lieu de cela, nous adaptons la pré-entraînement récursive aux PLM masqués, où un noyau de transformateur lié

Machine translation (Meta M2M100); the English original follows.

Abstract

Protein language models (pLMs) are powerful tools for protein annotation and mutation-effect prediction, but progress has largely depended on scaling model size and data, making them expensive and prone to memorization. Test-time scaling offers a complementary path, but methods developed for autoregressive LLMs do not transfer directly: masked encoders lack a generation loop, and amino acids do not verbalize in a reasoning language. Instead, we adapt recursive pretraining to masked pLMs, where a weight-tied Transformer core refines residue-aligned hidden states over an adjustable number of cycles. We instantiate this recipe as recAMPLIFY, a pLM with 155M unique parameters trained on the same 2T tokens as AMPLIFY-350M. Despite 56% fewer parameters, it improves CASP contact recovery and three out of four supervised tasks, notably raising fold accuracy by 22.5%. Because optimal depth for recAMPLIFY varies across tasks and assays, we then introduce Gated Mixture Recurrence (GMR), a framework that decodes each protein from a probability-weighted mixture of hidden states visited during recursion. GMR employs a learned residual gate at scalar, low-rank, or per-dimension resolution to control how much of each cycle's proposed update is accepted into the state. This unifies halting and gating into a single adaptive computation objective while recovering PonderNet-style halting as a special case without gates. Layered on top of recAMPLIFY, GMR further reduces expected layer depth by up to 37%, although the accuracy trade-off varies by task. Mechanistic analyses show that residual gating controls recurrence stability beyond the trained depth and that halt depth tracks biological properties.

Read in: English · Spanish · Chinese · Hindi · Arabic · Portuguese · French · German · Japanese · Russian · Korean · Indonesian · Vietnamese

Topics: Computer science, Artificial intelligence, Discriminator, Language model, CASP, Residual

Sources: OpenAlex (CC0), arXiv, Crossref. Agent Exchange