A Fix for Softmax and Cross-Entropy

Jonathan T. Barron, Dan Gnanapragasam, Michael Tschannen · 2026 · arXiv

DOI: 10.48550/arXiv.2610.31642 · open-access PDF

Abstract

Standard implementations of softmax, log-softmax, and cross-entropy have two numerical bugs. The first bug, Softmax Collapse, rounds the target logit's gradient to zero if that one logit is much larger than the others, resulting in gradients that no longer sum to zero. The second bug, Softmax Cancellation, occurs when student logits are close to teacher logits, and produces non-zero gradients even when the student and teacher logits are equal. We present simple implementations of log-softmax, softmax, and cross-entropy that fix both bugs. Our fix eliminates the previously-observed "Slingshot"

Read in: English · Spanish · Chinese · Hindi · Arabic · Portuguese · French · German · Japanese · Russian · Korean · Indonesian · Vietnamese

Is this paper real? This page was built from OpenAlex and arXiv records, not generated. Software can verify any reference in one call: GET https://store.agentexchange.work/gw/papers/verify?title=...&year=... (0.003 USD), fetch this record as JSON with /gw/papers/paper?id=2610.31642, or list what it cites with /gw/papers/references?id=.... Pay per call in USDC (x402) or with a 19 USD card key for 1,000 calls. Catalogue · llms.txt

Sources: OpenAlex (CC0), arXiv, Crossref. Agent Exchange