Newest papers

recAMPLIFY: Large-Scale Pretraining of Recursive Protein Language Models

Saurav Jha, Qianli Liao, Robert Vernon, Sarath Chandar, Quentin Fournier · 2026 · arXiv (Cornell University)

DOI: 10.48550/arxiv.2610.26747 · cited by 0 works (OpenAlex) · open-access PDF

Spanish

Los modelos de control de la lenguaje biológico de la prueba (PLMs) son herramientas poderosas para la anotación de proteínas y la predicción del efecto de la mutación, pero el progreso ha dependido en gran medida de la escalación del tamaño del modelo y de los datos, lo que los ha hecho caros y propensos a la memorización. El escalado de la prueba ofrece un camino complementario, pero los métodos desarrollados para los LLMs autoregresivos no se transfieren directamente: los codificadores enmascarados carecen de un loop de generación, y los aminoácidos no se verbalizan en un lenguaje de razonamiento. En cambio, adaptamos el pretraining recursivo a los PLMs enmascarados, donde un núcleo de Transformer ligado a un peso refina los estados ocultos de

Machine translation (Meta M2M100); the English original follows.

Abstract

Protein language models (pLMs) are powerful tools for protein annotation and mutation-effect prediction, but progress has largely depended on scaling model size and data, making them expensive and prone to memorization. Test-time scaling offers a complementary path, but methods developed for autoregressive LLMs do not transfer directly: masked encoders lack a generation loop, and amino acids do not verbalize in a reasoning language. Instead, we adapt recursive pretraining to masked pLMs, where a weight-tied Transformer core refines residue-aligned hidden states over an adjustable number of cycles. We instantiate this recipe as recAMPLIFY, a pLM with 155M unique parameters trained on the same 2T tokens as AMPLIFY-350M. Despite 56% fewer parameters, it improves CASP contact recovery and three out of four supervised tasks, notably raising fold accuracy by 22.5%. Because optimal depth for recAMPLIFY varies across tasks and assays, we then introduce Gated Mixture Recurrence (GMR), a framework that decodes each protein from a probability-weighted mixture of hidden states visited during recursion. GMR employs a learned residual gate at scalar, low-rank, or per-dimension resolution to control how much of each cycle's proposed update is accepted into the state. This unifies halting and gating into a single adaptive computation objective while recovering PonderNet-style halting as a special case without gates. Layered on top of recAMPLIFY, GMR further reduces expected layer depth by up to 37%, although the accuracy trade-off varies by task. Mechanistic analyses show that residual gating controls recurrence stability beyond the trained depth and that halt depth tracks biological properties.

Read in: English · Spanish · Chinese · Hindi · Arabic · Portuguese · French · German · Japanese · Russian · Korean · Indonesian · Vietnamese

Topics: Computer science, Artificial intelligence, Discriminator, Language model, CASP, Residual

Sources: OpenAlex (CC0), arXiv, Crossref. Agent Exchange