Optimization Flaws and Non-Learned Matrix Components

Velikanov et al., arXiv:2601.04890 | LLM Training Diagnostics

8
Components Tracked
3
Matrix Structures
2-5x
Spectral vs Norm Error
Partial
Multiplier Fix

Component Learning Errors by Dimension

Standard vs Multiplier Training

Errors by Matrix Structure

Key Findings

ComponentLearning QualityMultiplier Helps?Flaw Category
Row NormsModerateYesScale (known)
Column NormsModerateYesScale (known)
Singular ValuesPoorPartiallySpectral (new)
Condition NumberVery PoorNoSpectral (new)
Spectral GapVery PoorNoSpectral (new)
Effective RankPoorNoStructural (new)