Optimisers and kernels
Outer optimiser: Muon for rank-two hidden matrices, Adan for everything else (roadmap working rule; model J: learning rates 0.01 and 0.0005).
Muon (Jordan, 2024): replace the momentum-averaged gradient G = U S Vᵀ by U Vᵀ using five Newton-Schulz steps, (a, b, c) = (3.4445, −4.7750, 2.0315). Claims in the post: CIFAR-10 94% in 2.6 instead of 3.3 A100-seconds; NanoGPT 1.35 times faster; 1.5B model to GPT-2 XL level on HellaSwag in 10 against 13.3 hours with AdamW. Evidence is language-model and speedrun based; no Muon-versus-AdamW ablation recorded here.
Adan (Xie et al., 2022): Nesterov-style momentum; code keeps averages of gradient, gradient change and squared corrected gradient (betas 0.98, 0.92, 0.99). Authors claim half the epochs and batch sizes 1k to 32k. Not in torch.optim, so written in Triton.
Strict guard (native_dispatch.py): allows allocation, copies and views; replaces add and mul with Triton; raises on any other GPU ATen operator. Proves no stray PyTorch arithmetic; says nothing about correctness or speed.
Ridge solve (lstsq_coefficients.py, lstsq_blocked.py): (AᵀA + λI) Z = AᵀT + λZ₀, λ = 0.01, Cholesky in one register tile for K ≤ 128, blocked in 64 × 64 blocks up to 256. Backward: G = M⁻¹dZ, dZ₀ = λG, dA = (T − AZ)Gᵀ − (AG)Zᵀ.
Defence answer. “The kernels are an enabling implementation: known techniques, tested against FP64, in the released code. The contribution is the representation and the amortised placement. I have not benchmarked against torch.linalg, so I make no speed claim.”