Optimizer-Induced Low-Dimensional Drift and Transverse Dynamics in Transformer Training (1 minute read)
Researchers discover a dominant low-dimensional drift in transformer training that captures 60-80% of long-term parameter changes, emerging from optimizer dynamics rather than gradients.
