Newest papers

Fine-Grained Analysis of Catastrophic Forgetting in Continual Learning via Algorithmic Stability

Guodong Zheng, Yan Sun, Peng Wang, Enneng Yang, Jiafei Lyu, Xiaochun Cao, Dacheng Tao, Li Shen · 2026 · arXiv (Cornell University)

DOI: 10.48550/arxiv.2610.31551 · cited by 0 works (OpenAlex) · open-access PDF

Chinese

理解FinNet的灾难性忘记,其中学习新任务降低了以前学到的任务的性能,仍然是持续学习(CL)的根本挑战。现有理论分析往往依赖于特定的模型或任务结构,并提供有限的洞察力,以了解迭代优化如何影响忘记。为了解决这一差距,我们开发了一种模拟稳定框架,用于预期人口忘记在SGD下,而不会规定特定的模型架构。具体而言,我们将忘记分解为任务内部和跨任务的通用化差距和经验性忘记。为了弥合通用化差距,我们跟踪模拟的颠覆如何影响任务中的预期人口忘记,并通过后续任务进行传播。我们认为,不断的任务分解,并不断地分解学习时间表。比较两个分解时间表,我们可以描述

Machine translation (Meta M2M100); the English original follows.

Abstract

Understanding catastrophic forgetting, in which learning new tasks degrades performance on previously learned tasks, remains a fundamental challenge in continual learning (CL). Existing theoretical analyses often rely on specific model or task structures and offer limited insight into how iterative optimization affects forgetting. To address this gap, we develop an algorithmic stability framework for expected population forgetting under SGD without prescribing a particular model architecture. Specifically, we decompose forgetting into intra-task and cross-task generalization gaps and empirical forgetting. To bound the generalization gaps, we track how a sample perturbation affects updates within a task and propagates through subsequent tasks. We consider constant, intra-task decaying, and continuously decaying learning-rate schedules. Comparing the two decaying schedules allows us to characterize how restarting learning-rate decay at task boundaries affects the propagation of historical sample perturbations. For empirical forgetting, we derive bounds for all three schedules by retaining cumulative gradient alignment along the optimization trajectory, without requiring convexity. Combining these results yields upper bounds on expected forgetting that capture the effects of training set size, the number of updates per task, the total number of tasks, learning-rate schedules, and cumulative gradient alignment. Finally, we conduct experiments on Split CIFAR-100 and Split MiniImageNet to examine the empirical implications of our theoretical findings.

Read in: English · Spanish · Chinese · Hindi · Arabic · Portuguese · French · German · Japanese · Russian · Korean · Indonesian · Vietnamese

Topics: Forgetting, Generalization, Computer science, Stability (learning theory), Task (project management), Set (abstract data type)

Sources: OpenAlex (CC0), arXiv, Crossref. Agent Exchange