Newest papers

recAMPLIFY: Large-Scale Pretraining of Recursive Protein Language Models

Saurav Jha, Qianli Liao, Robert Vernon, Sarath Chandar, Quentin Fournier · 2026 · arXiv (Cornell University)

DOI: 10.48550/arxiv.2610.26747 · cited by 0 works (OpenAlex) · open-access PDF

Indonesian

Model-model biologi yang disesuaikan dengan prosedur ini adalah model-model biologi yang disesuaikan dengan prosedur-prosedur yang disesuaikan dengan prosedur-prosedur yang disesuaikan dengan prosedur-prosedur yang disesuaikan dengan prosedur-prosedur yang disesuaikan dengan prosedur-prosedur yang disesuaikan dengan prosedur-prosedur yang disesuaikan dengan prosedur-prosedur yang disesuaikan dengan prosedur-prosedur yang disesuaikan dengan prosedur-prosedur yang disesuaikan dengan prosedur-prosedur yang disesuaikan dengan prosedur-prosedur yang disesuaikan dengan prosedur-prosedur-prosedur yang disesuaikan.

Machine translation (Meta M2M100); the English original follows.

Abstract

Protein language models (pLMs) are powerful tools for protein annotation and mutation-effect prediction, but progress has largely depended on scaling model size and data, making them expensive and prone to memorization. Test-time scaling offers a complementary path, but methods developed for autoregressive LLMs do not transfer directly: masked encoders lack a generation loop, and amino acids do not verbalize in a reasoning language. Instead, we adapt recursive pretraining to masked pLMs, where a weight-tied Transformer core refines residue-aligned hidden states over an adjustable number of cycles. We instantiate this recipe as recAMPLIFY, a pLM with 155M unique parameters trained on the same 2T tokens as AMPLIFY-350M. Despite 56% fewer parameters, it improves CASP contact recovery and three out of four supervised tasks, notably raising fold accuracy by 22.5%. Because optimal depth for recAMPLIFY varies across tasks and assays, we then introduce Gated Mixture Recurrence (GMR), a framework that decodes each protein from a probability-weighted mixture of hidden states visited during recursion. GMR employs a learned residual gate at scalar, low-rank, or per-dimension resolution to control how much of each cycle's proposed update is accepted into the state. This unifies halting and gating into a single adaptive computation objective while recovering PonderNet-style halting as a special case without gates. Layered on top of recAMPLIFY, GMR further reduces expected layer depth by up to 37%, although the accuracy trade-off varies by task. Mechanistic analyses show that residual gating controls recurrence stability beyond the trained depth and that halt depth tracks biological properties.

Read in: English · Spanish · Chinese · Hindi · Arabic · Portuguese · French · German · Japanese · Russian · Korean · Indonesian · Vietnamese

Topics: Computer science, Artificial intelligence, Discriminator, Language model, CASP, Residual

Sources: OpenAlex (CC0), arXiv, Crossref. Agent Exchange