Newest papers

Fine-Grained Analysis of Catastrophic Forgetting in Continual Learning via Algorithmic Stability

Guodong Zheng, Yan Sun, Peng Wang, Enneng Yang, Jiafei Lyu, Xiaochun Cao, Dacheng Tao, Li Shen · 2026 · arXiv (Cornell University)

DOI: 10.48550/arxiv.2610.31551 · cited by 0 works (OpenAlex) · open-access PDF

Russian

Понимание плановых задач, в которых изучение новых задач ухудшает производительность ранее изученных задач, остается фундаментальной проблемой при проведении непрерывного обучения (CL). Существующие теоретические анализы часто опираются на конкретную модель или структуру задач и предлагают ограниченное понимание того, как итеративная оптимизация влияет на забывание. Чтобы устранить этот пробел, мы разрабатываем алгоритмическую рамку стабильности для ожидаемого забывания населения в рамках SGD без предписания конкретной архитектуры модели. В частности, мы разлагаем забывание на внутризадачные и кроссзадачные расстояния генерализации и эмпирическое забывание. Чтобы связать пробелы генерализации, мы отслеживаем, как образец пертурбации влияет на обновления в рамках задачи и распространя

Machine translation (Meta M2M100); the English original follows.

Abstract

Understanding catastrophic forgetting, in which learning new tasks degrades performance on previously learned tasks, remains a fundamental challenge in continual learning (CL). Existing theoretical analyses often rely on specific model or task structures and offer limited insight into how iterative optimization affects forgetting. To address this gap, we develop an algorithmic stability framework for expected population forgetting under SGD without prescribing a particular model architecture. Specifically, we decompose forgetting into intra-task and cross-task generalization gaps and empirical forgetting. To bound the generalization gaps, we track how a sample perturbation affects updates within a task and propagates through subsequent tasks. We consider constant, intra-task decaying, and continuously decaying learning-rate schedules. Comparing the two decaying schedules allows us to characterize how restarting learning-rate decay at task boundaries affects the propagation of historical sample perturbations. For empirical forgetting, we derive bounds for all three schedules by retaining cumulative gradient alignment along the optimization trajectory, without requiring convexity. Combining these results yields upper bounds on expected forgetting that capture the effects of training set size, the number of updates per task, the total number of tasks, learning-rate schedules, and cumulative gradient alignment. Finally, we conduct experiments on Split CIFAR-100 and Split MiniImageNet to examine the empirical implications of our theoretical findings.

Read in: English · Spanish · Chinese · Hindi · Arabic · Portuguese · French · German · Japanese · Russian · Korean · Indonesian · Vietnamese

Topics: Forgetting, Generalization, Computer science, Stability (learning theory), Task (project management), Set (abstract data type)

Sources: OpenAlex (CC0), arXiv, Crossref. Agent Exchange