Part VI · Doing it properly · Chapter 18

Optimisers and kernels

About 20 minutes. Tap Continue to move on.

18.1 Why this chapter matters

Two things in your roadmap look odd from outside. The optimisers are Muon for hidden matrices and Adan for everything else, which most readers know little about. And every operation in training and inference is a hand-written Triton kernel under a guard that rejects anything else. A reviewer will want to know why, and whether the kernel work is science or plumbing.

By the end you can explain what Muon and Adan do in two sentences each, say what the strict dispatch guard does and does not prove, describe the blocked Cholesky and its hand-derived backward pass, and give an honest answer about whether any of it is a research contribution.

Before reading: what do you think “optimiser” means for the weights of a network, apart from plain gradient descent?

18.2 Optimisers in a few lines

An optimiser turns the gradient g of the loss into a change of the weights p. Three steps of increasing cleverness:

  • Gradient descent: p ← p − η g, where η is the learning rate.
  • Momentum: keep a running average of recent gradients, m ← β m + (1 − β) g, and step along m. It smooths noise and carries speed through flat stretches.
  • Adam also keeps a running average of the squared gradient, v ← β₂ v + (1 − β₂) g², and divides the step by √v. Each weight gets its own effective step size.

Adam is the default of deep learning[Kingma and Ba, 2014]. Its weakness is that it treats every weight as independent, even though a weight matrix acts as a whole on its input.

Two different optimisers in your system. The inner Adam of Chapter 11 moves Gaussian centres and shapes for one image, sixteen steps inside the forward pass. The outer optimiser trains the network's weights over thousands of steps across many images. This chapter is about the outer one: Muon plus Adan (the configuration for model J sets learning rates of 0.01 and 0.0005).

18.3 Muon: orthogonalised updates for matrices

Take the momentum-averaged gradient of one weight matrix, G. Its singular value decomposition is G = U S Vᵀ: a set of input directions, output directions and strengths S. Gradients of large matrices are often dominated by a few strong directions. Muon replaces G by U Vᵀ, the nearest semi-orthogonal matrix, which keeps the directions and sets every strength to one[Jordan, 2024].

Example. Singular values (10, 1, 0.1) become (1, 1, 1). A plain step would move almost only along the first direction; the Muon step moves equally along all three.

Computing an SVD each step would be slow, so Muon uses a Newton-Schulz iteration: repeated matrix products that push all singular values towards one. The post gives the coefficients (a, b, c) = (3.4445, −4.7750, 2.0315) and five steps. With A = X Xᵀ each step is X ← aX + (bA + cA²)X, and G is first divided by its norm[Jordan, 2024]. Your kernel file runtime_muon_adan_dynamic.py uses exactly these numbers, with momentum 0.95, Nesterov-style momentum and five iterations as defaults, and runs the products with the project's Triton matmul.

The rules from the source for where Muon applies: two-dimensional parameters of hidden layers only. Embeddings, output layers, scalars and vectors use a standard optimiser such as AdamW[Jordan, 2024]. Your code enforces this: every parameter group must name its algorithm, its source and its parameter names, and a Muon group rejects anything that is not rank two.

The evidence, as the post states it. Faster records: CIFAR-10 to 94% accuracy from 3.3 to 2.6 A100-seconds; a 1.35 times speed-up to 3.28 validation loss on FineWeb (NanoGPT speedrun); a 1.5B-parameter model reaching GPT-2 XL level on HellaSwag in 10 8×H100-hours against 13.3 with AdamW; overhead estimated at no more than about 0.7%. A later paper reports roughly 2 times computational efficiency over AdamW in scaling-law experiments and trains a 3B/16B mixture-of-experts model on 5.7 trillion tokens[Liu et al., 2025]. The post's own argument for credibility is that optimiser claims should be won against well-tuned baselines on a competitive task. For theory, see Bernstein's derivation[Bernstein, 2025] and the anthology on steepest descent under norms[Bernstein and Newhouse, 2024].

Honest scope: that evidence is language models and speedrun settings. Your heads are small (hidden width 64 in the configuration I read). The roadmap records no ablation of Muon against AdamW on this model, so “Muon is better here” is not a claim you can make; “I use the recipe the community validated at scale” is. The rectangular-matrix scale √max(1, rows/cols) in your code follows the upstream repository at the pinned commit; I could not confirm it from the post's text.

18.4 Adan, and why two optimisers

Adan is Adam with Nesterov-style acceleration built into the moment estimates. Its abstract describes a Nesterov momentum estimation that avoids the extra cost of computing the gradient at an extrapolated point. It states an O(ε−3.5) stochastic gradient complexity for reaching an approximate stationary point, a claim of matching or beating state-of-the-art optimisers with half the epochs on ViT, GPT-2 and MAE, and tolerance to batch sizes from 1k to 32k[Xie et al., 2022]. Those are the authors' claims, not yours.

What your code does, which is more useful to know: Adan keeps three running averages, of the gradient (m), of the change in gradient between steps (d) and of a squared, change-corrected gradient (v), with default retention coefficients 0.98, 0.92 and 0.99. The step uses m plus a multiple of d, divided by √v. The change term is the acceleration; it is why each weight needs the previous gradient, which the code stores.

Why both? Muon cannot handle vectors and scalars, so something else must. The standard companion is AdamW; the roadmap's working rule says “Optimisers stay as they are: Muon for hidden matrices, Adan for the rest.” Be honest about what supports that. It is a decision frozen early so that runs are comparable, not a result. In the code the split is by name, not shape: Muon gets readout_weight, slot_queries, key_weight, value_weight and query_weight, while the two 2-D output heads geometry_weight and readout_output_weight go to Adan with the vectors. A practical reason is also on record: torch.optim.Muon exists in PyTorch but Adan does not (lab book, 10 August 2026), and your rule is that every operation is a Triton kernel, so both had to be written.

18.5 Triton kernels and the strict guard

A GPU kernel is a small program that many GPU threads run at once on different pieces of data. Triton is a Python-embedded language for writing kernels at the level of tiles: you write what one program does to one block of numbers, and the compiler handles threads and memory details[Tillet et al., 2019]. The official tutorials are the quickest way in[Triton tutorials].

Your optimiser file is a typical example. Each operation is a few lines on a block of 256 elements: update the momentum, compute a norm, combine two matrices as αA + βB, apply the update. Details worth noticing, because they show the care behind the design:

  • FP32 and no fused multiply-add (enable_fp_fusion=False), so rounding is predictable and the same across runs and tests.
  • Learning rates are runtime arguments excluded from specialisation (do_not_specialize), so the schedule does not trigger a new compilation every step.
  • Finiteness checks on every input and state before the update and on every output after it, raising an error instead of continuing with NaN or infinity.
  • Provenance. The upstream commits are recorded as constants and checked when a checkpoint is loaded; run configurations pin the SHA-256 hash of the source file.

The general pattern, fusing steps into one kernel and recomputing instead of storing, is the one behind FlashAttention[Dao et al., 2022]. Your kernels are much simpler, because the matrices are small.

The strict dispatch guard

PyTorch lets a program intercept every low-level operation (an ATen operator) a tensor operation turns into. native_dispatch.py uses this to build a context that wraps a whole training or inference step.

Strict dispatch guard. While active, every ATen operator on a GPU tensor is sorted. Allocation, initialisation, copies and views are allowed. Addition and multiplication are replaced by Triton kernels. Anything else raises an error.

Why intercept addition at all? Autograd itself accumulates gradients with aten.add, which you cannot avoid, so the guard redirects it. The recorded GPU test (25 October 2026, PyTorch ROCm build with Triton 3.7.1) counted six such additions over two native updates, all replaced, with zero unexpected arithmetic operators. A diagnosis mode, record_only, lets unexpected operations run but marks the report as not passed.

What it proves. No stray PyTorch arithmetic kernel (a hidden matmul, softmax or division) executed inside the guarded step.

What it does not prove. That the Triton kernels are correct (the tests do that), that they are fast, or anything about libraries called from inside a kernel path: the file's own header says it audits ATen dispatch, not external library internals, and CPU calls pass through.

So when a run says “strict Triton clean”, the sentence means: the guard saw no disallowed operator. It is an audit trail for the claim “this is a native Triton pipeline”, not evidence about accuracy.

18.6 Cholesky and its hand-written backward

Chapter 4 gave you the equations. Per image, the features solve M Z = R with M = AᵀA + λI (a K × K matrix, K the number of Gaussians), R = AᵀT + λZ₀ and λ = 0.01. Here A holds the responsibilities, T the teacher field and Z₀ the weighted means.

Forward

lstsq_coefficients.py factors M = L Lᵀ by Cholesky in one pass: one GPU program per image holds the whole K × K tile in registers and, column by column, takes the pivot, divides the column below it by its square root and subtracts the outer product from what remains. Two triangular substitutions give Z. Pivots are floored at min(λ, 1), a safety net that never triggers in exact arithmetic because every pivot of M is at least λ.

Backward, by hand

Autograd could in principle differentiate the factorisation step by step, but that is K sequential steps of stored state. The better route is implicit: Z = M⁻¹R, so one extra solve with the same factor gives the gradient. For an incoming gradient dZ:

G = M⁻¹ dZ, dZ₀ = λ G, dA = (T − A Z) Gᵀ − (A G) Zᵀ

The two terms of dA come from the two places A appears: T Gᵀ from the right-hand side R = AᵀT, and −A(G Zᵀ + Z Gᵀ) from M = AᵀA; add them and regroup. The teacher T gets no gradient because it is frozen. This is the variable-projection idea of Chapter 4[Golub and Pereyra, 1973]: geometry outside, features solved inside, derivative by the implicit function theorem. The code checks itself against an FP64 torch.linalg oracle, and the roadmap records 37 new tests.

Blocked, for more than 128 Gaussians

One register tile stops working above K = 128. lstsq_blocked.py keeps the old path bit for bit for K ≤ 128 (its source hash pins every earlier configuration) and adds a right-looking blocked Cholesky up to K = 256: split the matrix into 64 × 64 blocks and, for each diagonal block in turn, do three things.

1232332333
  1. Factor the diagonal block (the same register tile code).
  2. Solve for the panel of blocks below it, Lij = Sij Ljj−T.
  3. Update everything still to come (the trailing blocks), Sii′ ← Sii′ − Li Li′T, with an IEEE FP32 tile product.

Each stage is its own launch, so every value a kernel reads was written by an earlier launch, which avoids hazards inside a kernel. The forward solve and the whole backward pass are reused unchanged. The blocked path is what lets one model be trained and scored at 256 Gaussians (the any-count results).

The numbers 1, 2, 3 in the picture are the three steps for the first diagonal block; the same pattern repeats on the shrinking lower-right part. Dashed cells are the upper triangle, which is not computed.

18.7 Research contribution or engineering?

You will be asked, and a short honest answer is best. Take the pieces one at a time.

  • Muon and Adan: re-implemented from pinned upstream code. A faithful port, not a contribution.
  • Cholesky solve: textbook. The blocked right-looking form is textbook numerical linear algebra, done here in a GPU kernel language.
  • Implicit backward pass: textbook calculus (variable projection and the implicit function theorem). Writing it by hand is engineering, and necessary here.
  • Strict dispatch guard: a useful audit tool and a self-imposed rule. It supports a claim of a native pipeline, not an accuracy claim.

What the kernels do give the science. They make the differentiable closed-form feature solve a practical part of training; the measured 22% lower dense error from least-squares features, and the any-count results up to 256, rest on them. The contribution is that this idea works, not the kernel that runs it.

The weak point. A reviewer can ask: why not torch.linalg.cholesky and autograd? That would be simpler and might well be fine. The files I read record no timing or accuracy comparison against it. Native-Triton-everywhere is a rule you chose, with a real cost in development time. If you wanted to claim a kernel contribution you would need such a comparison: speed and memory against a standard batched solver across K and batch size, and accuracy against FP64.

Suggested framing (my judgement): present the kernels as an enabling implementation with tests, in an appendix or supplement and in the released code; keep the paper's contribution to the representation and the amortised placement. Say what you did and did not measure.

18.8 Check yourself

Answer from memory; each question gives its reason once you choose.

1. What does Muon do to a weight matrix’s averaged gradient?

Singular directions are kept and all singular values are set to one, approximated with five Newton-Schulz steps. Dividing by a running second moment is what Adam does.

2. Which parameters does your code route to Muon?

Muon groups reject anything that is not rank two, and the source recommends keeping embeddings, heads, vectors and scalars on another optimiser. Here that is Adan.

3. What does a “strict Triton clean” report prove?

The guard only checks which operators were dispatched. Correctness comes from tests against FP64 references, and speed has to be measured separately.

4. Why is the backward pass of the ridge solve done implicitly?

Differentiating Z = M⁻¹R needs G = M⁻¹dZ, one more pair of substitutions using the stored factor. The formulas hold for any positive lambda.

5. What is the honest answer on whether the kernels are research?

Every piece is a known technique, and no timing against a standard solver is recorded. They enable the method and belong in the code and an appendix, not in the contribution list.

18.9 Close the loop

Without scrolling up, say in plain words what Muon and Adan do, what the strict guard proves and does not prove, and how you would answer “is the kernel work a contribution?”.

A model answer

“Outer training uses Muon for rank-two hidden matrices (orthogonalised momentum by five Newton-Schulz steps) and Adan for the rest, both as FP32 Triton kernels under a guard that rejects any other GPU arithmetic. The ridge solve is a Cholesky in a register tile (blocked above 128 Gaussians) with an implicit backward pass. These are known techniques that enable the method; I have not benchmarked them against standard solvers, so I present them as engineering, not as a contribution.”

Read next Keller Jordan, “Muon: An optimizer for hidden layers in neural networks” (2024). Short, and it states its own evidence standard, which is useful to borrow for your own claims.

Something unclear, or want to rehearse how you would say this to a reviewer? Ask your teacher in the chat.

One-page summary of this chapter ·