Newest papers

Phase Transition from Random Features to Fully Trainable Networks

Xin He, Huan Lao, Xiliang Lu, Shijun Zhang · 2026 · arXiv (Cornell University)

DOI: 10.48550/arxiv.2610.31663 · cited by 0 works (OpenAlex) · open-access PDF

Japanese

この論文は、接近範囲内のフェイスプレッシャー変動パターンを定量化し、接近範囲内の接近範囲内で同様の行動を観察します:急速かつスムーズな初期の改善は、減少した回収に伴い、より大きな初期化不一致の下でより鋭い移行を可能にします。 ポリノミアスプレッシャー変動の下で、Exponent $α>1$とハイブリッジ互換性をどのように利用するかを定量化し、L_2$の接近率 $\mathcal_O_$0_$0_$1を示す(P_log_(m/δm)/n=$1を示す(P1dd=1P1d=2d=2d

Machine translation (Meta M2M100); the English original follows.

Abstract

Constructing accurate and computationally tractable approximation spaces is a fundamental task in scientific computing. Partially trainable neural networks bridge fixed random features and fully adaptive representations. This paper quantifies a phase transition pattern in approximation bounds and observes analogous behavior in training errors: a rapid but smooth initial improvement followed by diminishing returns, with a sharper transition under greater initialization mismatch. Under polynomial spectral decay with exponent $α>1$ and leverage compatibility, we establish a squared $L_2$ approximation rate $\mathcal{O}((κ_P\log(m/δ)/m)^α)$ for two-layer networks of width $m$ and trainable proportion $P$, where $κ_P=(P/κ_1^*+(1-P)/κ_0^*)^{-1}$. The bound holds uniformly over the unit ball of a reproducing kernel Hilbert space (RKHS) with probability at least $1-δ$. The constants $κ_0^*$ and $κ_1^*$ quantify the fixed and trainable groups' sampling mismatches relative to the target RKHS. For spherical biased-ReLU features in dimension $d$, we verify these assumptions with $α=(d+2)/(d-1)$ and $κ_0^*=4\sqrt{2}τ(d+1)/d$ for initialization scale $τ\ge1$, and $κ_1^*=1$ under ridge leverage sampling. For $d=5$ and $τ=10$, a trainable proportion of approximately $4.1\%$ captures $90\%$ of the reduction in the approximation bound from $P=0$ to $P=1$. A structured three-layer example illustrates how a few trainable first-layer neurons exploit fixed downstream blocks through nonlinear composition. Numerical experiments across diverse settings, including networks with up to 15 layers, exhibit similar phase transitions in training errors.

Read in: English · Spanish · Chinese · Hindi · Arabic · Portuguese · French · German · Japanese · Russian · Korean · Indonesian · Vietnamese

Topics: Initialization, Mathematics, Leverage (statistics), Nonlinear system, Upper and lower bounds, Estimator

Sources: OpenAlex (CC0), arXiv, Crossref. Agent Exchange