Newest papers

SpecMRL: Redesign Matryoshka Representation Learning within Parametric Spectrum

Zhongxin Chen, Songhao Wu, Heng Cui, Cong Li, Xiao Zhou, Rui Yan · 2026 · arXiv (Cornell University)

DOI: 10.48550/arxiv.2610.26824 · cited by 0 works (OpenAlex) · open-access PDF

Chinese

Matryoshka Representation Learning (MRL) 旨在通过共同优化嵌入前缀来支持灵活的表示尺寸。虽然有效,但这种设计缺乏明确的限制,以确保主部分提供适当的紧凑编码。为了解决这一问题,我们提出了 SpecMRL,该功能空间的重组使前缀读取更有效。具体来说,我们将MRL扩展到具有特征重要性的参数衍生的光谱基础。 SpecMRL 否则遵循标准 MRL,在学习的光谱空间中进行前缀读取。 MTEB 上的实验表明, SpecMRL 在评估的所有缩小尺寸中优于 MRL,在更具侵略性的压缩下获得更大的收益。这些结果表明,学习的光谱基础更好地优先考虑信息的方向,允许更有效的表示压缩

Machine translation (Meta M2M100); the English original follows.

Abstract

Matryoshka Representation Learning (MRL) is designed to support flexible representation sizes by jointly optimizing nested prefixes. While effective, this design lacks an explicit constraint to ensure that the leading portion provides a suitable compact encoding. To address this, we propose SpecMRL, which reorganizes the feature space to enable more effective prefix readout. Specifically, we extend MRL with a parameter-derived spectral basis organized with feature importance. SpecMRL otherwise follows standard MRL, with prefix readout performed in the learned spectral space. Experiments on MTEB show that SpecMRL outperforms MRL at all evaluated reduced dimensions, with larger gains under more aggressive compression. These results suggest that the learned spectral basis better prioritizes informative directions, enabling more effective representation compression. Notably, SpecMRL extends the performance frontier of MRL, surpassing full-dimensional MRL with only half the embedding dimensions.

Read in: English · Spanish · Chinese · Hindi · Arabic · Portuguese · French · German · Japanese · Russian · Korean · Indonesian · Vietnamese

Topics: Representation (politics), Embedding, Computer science, Feature (linguistics), Basis (linear algebra), Feature learning

Sources: OpenAlex (CC0), arXiv, Crossref. Agent Exchange