Newest papers

The role of locality and weight sharing in convolutional neural networks

Francesco Mori, Melanie Weber · 2026 · arXiv (Cornell University)

DOI: 10.48550/arxiv.2610.30118 · cited by 0 works (OpenAlex) · open-access PDF

Chinese

连接神经网络的成功率往往归因于本地性和重量共享。然而,这些两个属性如何分开形成本地学习的速度,以及它们的效果如何取决于相同自然图像的较晚的空间相关性,仍然不太了解。我们在教师和学生模型中解决了这些问题,一个连接的老师在 $P$的输入位置上运作,与位置之间的相关性 $c$。我们比较了由在线斯托加斯特降级降级进行训练的连接和本地连接的学生,并得出常见的差异方程式,这些方程式描述了他们在大特征尺寸的极限内的动态。对于无关联的入口位置($c$ = 0$),两名学生遵循相同的学习轨迹,而连接的学生的学习速度

Machine translation (Meta M2M100); the English original follows.

Abstract

The success of convolutional neural networks is often attributed to locality and weight sharing. However, how these two properties separately shape learning, and how their effects depend on the spatial correlations of natural images, remain poorly understood. We address these questions in a teacher-student model with a convolutional teacher acting on $P$ input positions with correlation $c$ between positions. We compare convolutional and locally connected students trained by online stochastic gradient descent, and derive ordinary differential equations that describe their dynamics exactly in the limit of large feature dimension $D$. For uncorrelated positions ($c = 0$), the two students follow the same learning trajectory, with the convolutional student learning exactly $P$ times faster. We also find that locality affects the sample complexity: a student that matches the kernel size of the teacher recovers it from $\mathcal{O}_{D}(D)$ samples, whereas $\mathcal{O}_{D}(D\log(D))$ samples are needed with a larger kernel. For $c > 0$, spatial correlations can reduce the advantage of weight sharing. The asymptotic error of the convolutional student grows with $c$, whereas that of the locally connected student is independent of $c$ and overall lower when evaluated at the same learning rate. When the learning rate is instead tuned to the training horizon, however, the locally connected student wins at early times and the convolutional student at late times, with a transition that we predict analytically.

Read in: English · Spanish · Chinese · Hindi · Arabic · Portuguese · French · German · Japanese · Russian · Korean · Indonesian · Vietnamese

Topics: Convolutional neural network, Locality, Limit (mathematics), Computer science, Uncorrelated, Dimension (graph theory)

Sources: OpenAlex (CC0), arXiv, Crossref. Agent Exchange