Appendix A · Rebuilding the pipeline in NumPy

Rebuilding the pipeline in NumPy

About 40 minutes, longer than a chapter: it walks the whole of model J, equation by equation, with the shape of every tensor. Keep learning/rebuild/numpy_pipeline.py open beside it; every step names the NumPy function and the original file:function it mirrors.

A.1 Why rebuild it

An examiner can ask you to put any part of the model on the board: what goes in, what shape it has, what the gradient is, and why it is computed that way. The production code is Triton kernels under a strict guard, which is hard to read at the board. So this appendix rebuilds model J, the current best model on the 32×32 input512 target, in about 900 lines of NumPy, loads the real trained weights, and checks the result against the GPU run.

By the end you can write every stage of J from input tokens to training loss with shapes, derive the gradient of the dense error with respect to the Gaussians through the least-squares solve, state exactly what is detached in training and why, write the Muon and Adan updates, and explain why the split and merge hierarchies lost.
SymbolMeaningValue in JConfig key
Bimages in a batch32 in training (4 here)batch
G, Pgrid side, tokens32, 1024grid.grid
Qquadrature points per token2×2 = 4grid.quadrature
Dteacher feature width384model.features
KGaussians8×8 = 64model.rows, cols
Hslot width64model.hidden
Iclustering iterations8model.refinement_steps
Srefinement Adam steps16 (32 at evaluation)refine.steps, eval_steps

Before reading on: write the pipeline of J from teacher tokens to the loss as a list of stages, with one shape per stage if you can.

A.2 Inputs, the Gaussian, and the matrix A

The target. The cache holds DINOv3 ViT-S/16 tokens of each image rendered at 512 px: a 32×32 grid, FP16, shape (B, 1024, 384). Each token is divided by its length: T = x / max(‖x‖, 10⁻⁸). So T has unit rows. NumPy teacher_target; original free_gaussian_fine.py:_cluster_geometry, first line.

Coordinates. The image is [−1, 1]². Token J (row-major, J = 32·row + column) has centre ((col + ½)/16 − 1, (row + ½)/16 − 1). Its four quadrature points sit at the centre ± 1/64 in x and y, shape (P, Q, 2) = (1024, 4, 2). NumPy quadrature_points; original free_gaussian_fine.py:FineGrid.points → geometry.py:patch_quadrature. The rebuild reproduces the checkpoint's points buffer exactly.

A Gaussian is a centre μ, a Cholesky factor L = [[a, 0], [b, d]] with Σ = LLᵀ, and a mass logit. The code uses two parameterisations of the same object:

WhereStored as (per Gaussian, 6 numbers)DecodedFloor on a, d
Clustering, raw(u₁, u₂, u₃, u₄, u₅, m)μ = tanh(u₁, u₂); a = .008 + .05 softplus(u₃); b = .05 u₄; d = .008 + .05 softplus(u₅)0.008
Refinement, q(cx, cy, log(a − F), b, log(d − F), m)a = F + exp(·), F = 0.0040.004
Moment fitafter width scalingmax(s·a₀, .009)0.009

A positive diagonal makes Σ positive definite (Chapter 2). The mass is π = softmax(m) over the 64 Gaussians. NumPy free_decode, q_encode, q_decode; original free_gaussian_spatial.py:_decode_free, free_gaussian_refine.py:_encode, _qdecode.

Responsibilities. With u = (x − μx)/a and v = (y − μy − b u)/d, the whitened offset is (u, v) = L⁻¹(x − μ), and

log ρk(x) = log πk − log ak − log dk − ½(u² + v²)

ℓpk = log Σq exp log ρk(xpq), Apk = softmaxk ℓpk

Shapes: the offsets are (B, P, Q, K) = (B, 1024, 4, 64); ℓ and A are (B, 1024, 64). The constants log 2π and log Q are dropped because they cancel in the softmax. The density is averaged over the four points before the softmax, as Chapter 3 explained. NumPy responsibilities; original kernels.py:gaussian_responsibilities (Triton), ground truth kernels.py:gaussian_responsibilities_reference.

A.3 Features by ridge least squares, and its gradient

Given A, the features Z (B, 64, 384) solve, per image,

(AᵀA + λI) Z = AᵀT + λZ₀, λ = 0.01, Z₀ = Nᵀ T, Npk = Apk / max(ΣpApk, 10⁻⁸)

M = AᵀA + λI is 64×64 and positive definite, so it is factored once, M = CCᵀ, and one forward and one back substitution solve all 384 channels. The field is F = AZ, shape (B, 1024, 384). NumPy ridge_coefficients, cholesky_solve; original lstsq_coefficients.py:RidgeCoefficients.forward (via lstsq_blocked.py:final_coefficients_any, the same call for K ≤ 128), ground truth ridge_coefficients_reference.

The adjoint. For a loss l(Z) with gradient dZ:

G = M⁻¹ dZ, dA = (T − AZ)Gᵀ − (AG)Zᵀ, dZ₀ = λG

and dZ₀ continues into A through the column normalisation: dN = T dZ₀ᵀ, dA += (dN − ΣpN⊙dN)/colsum. Derivation in one line: differentiate MZ = AᵀT + λZ₀ to get dM·Z + M·dZ = dAᵀT + λdZ₀, with dM = dAᵀA + AᵀdA; pair with G and collect the dA terms. This is implicit differentiation through a linear solve, the unconstrained case of differentiating through an optimisation layer[Amos and Kolter, 2017][Blondel et al., 2021]. It costs one extra solve with the stored factor; the Cholesky factorisation itself is never differentiated. NumPy ridge_coefficients_backward; original RidgeCoefficients.backward.

Exactly what is differentiated. Z is not detached anywhere in J. Training and the inner refinement gradient both pass through the solve, the adjoint above, and the Z₀ path. Only the teacher T is a constant (target.detach()).

Variable projection. Golub and Pereyra eliminate the linear parameters by least squares and optimise only the nonlinear ones[Golub and Pereyra, 1973]. Here the nonlinear parameters are the geometry and the linear ones are Z. With λ = 0 the normal equations give Aᵀ(AZ − T) = 0. Then the implicit part of the gradient vanishes and

dE/dA = (∂E/∂F) Zᵀ with Z held fixed

This is the envelope theorem. With λ > 0 and Z₀ depending on A it is no longer exact, so the code keeps the full adjoint. Measured on two holdout images at the clustering output, the fixed-Z gradient differs from the exact one by 3.6% (relative norm). That is small but not negligible. NumPy dense_objective_gradient(..., explicit_only=True).

A.4 The clustering stage (8 unrolled iterations)

Start. The centres follow a Halton sequence in bases 2 and 3: μi = 1.8·(h₂(i+1), h₃(i+1)) − 0.9. Both diagonals start at 0.12, b = 0, and all mass logits are 0. This is a fixed buffer, not learned (initial_geometry; free_gaussian_spatial.py:FreeGaussianModel.__init__). Slot states start at the learned slot_queries (64, 64).

Loop-invariant inputs (cluster_context): keys = unit([T, xy] Wkᵀ) and values = [T, xy] Wvᵀ, each (B, 1024, 64), with Wk, Wv of shape (64, 386). The moment basis is (x, y, x², xy, y²) at token centres, (1024, 5).

Iteration i (cluster_iteration; free_gaussian_fine.py:_cluster_geometry). The learned per-iteration scalars are blend βi = σ(·), sharpness wf,i, wg,i = exp(·) and width si = exp(·):

  1. Decode, render A, and form weighted-mean features Z̄ = NᵀT (not least squares inside the loop). Compute the per-token error ep = meand(AZ̄ − T)², (B, 1024), which is detached.
  2. Importance ωp = 0.5 + 0.5 min(ep/ē, 4). Geometry score gkp = −log(1 + ‖Lk⁻¹(xp − μk)‖²) + log ωp, (B, 64, 1024). This is a heavy-tailed score, not a Gaussian log density.
  3. E-step: softpk = softmaxk(wf,i⟨unit Z̄k, tp⟩ + wg,i gkp), (B, 1024, 64). Assignment = soft ⊙ ω.
  4. M-step: moments = normcols(assignment)ᵀ · basis, (B, 64, 5). Then cluster_fit: a₀ = √(E x² − x̄² + 10⁻⁴), c₀ = covxy/a₀, d₀ = √max(vary − c₀², 10⁻⁴); scale by si with floor 0.009; invert the decode; mass logit = log(Σpsoftpk + 0.01).
  5. Blend: raw ← (1 − βi) raw + βi fitted.
  6. Slot update: q = unit(state Wqᵀ) and scores = 8 q·keysᵀ, (B, 64, 1024). Attention = 0.75 softmaxp(scores + g) + 0.25 softmaxp(scores); context = attention · values. Coordination = softmax(4 · unit(state)unit(state)ᵀ) · state. Then state ← unit(state + context + 0.1·coordination).
  7. Residual head: raw ← raw + ([state, raw] Wgᵀ + bg)/8, with Wg of shape (6, 70).

Steps 1 to 4 are soft k-means, i.e. EM for a Gaussian mixture in feature-plus-position space[Bishop, 2006][Dempster et al., 1977]. Unrolling EM as a trainable network is Neural EM[Greff et al., 2017]. Step 6 is slot attention, with two differences: softmax runs over tokens rather than over slots, and a geometric bias is added[Locatello et al., 2020]. The trained J has blend 0.66 → 0.42 and width 0.56 → 1.21 across the eight iterations, so the early iterations fit narrower Gaussians than the moments say.

A.5 Refinement: 16 Adam steps on the true error

From the clustering output, q₀ = QEncode(FreeDecode(raw)) is (B, 64, 6). Then, for t = 0 … 15 (refine; free_gaussian_refine.py:refine_geometry, _adam, _step):

S(q) = 10⁴ Σb mseb(q), mseb = meanp,d(A Z(A) − T)²

g = ∂S/∂q; m ← m + 0.1(g − m); v ← 0.999v + 0.001g²

dir = (m/(1 − 0.9t+1t+1)) + 10⁻⁸; q ← q − st,group·dir

The step sizes are 0.03 for all three groups (centre; covariance (log a, b, log d); mass), fixed in J (learn_step_size: false). The moments carry across the steps. This is exactly torch.optim.Adam with learning rate 0.03[Kingma and Ba, 2014], and compare_with_torch.py confirms it to 10⁻¹⁴.

The gradient by hand (dense_objective_gradient, responsibilities_backward):

  1. dF = 10⁴ · 2(F − T)/(PD).
  2. dA = dF Zᵀ + ridge adjoint(AᵀdF) (A.3).
  3. Softmax over Gaussians: dℓ = A ⊙ (dA − ΣkA⊙dA).
  4. Within the token, wpqk = exp(log ρpqk − ℓpk) sums to one over q, and γ = dℓ·w is the gradient on log ρ.
  5. ∂log ρ/∂μx = u/a − bv/(ad), ∂/∂μy = v/d, ∂/∂a = (u² − 1)/a − buv/(ad), ∂/∂b = uv/d, ∂/∂d = (v² − 1)/d, ∂/∂log π = 1. Sum γ × these over p and q.
  6. Log-softmax for the mass: dm = gπ − π Σ gπ. The decode: ∂a/∂log(a − F) = a − F.

Against central differences in float64 the worst relative error is 5×10⁻⁸; against torch autograd of the repository's reference_objective it agrees to 5×10⁻¹⁴.

Research behind it. A fixed number of optimiser steps run as network layers is algorithm unrolling. LISTA unrolled ISTA and learned its matrices[Gregor and LeCun, 2010]; the survey covers the design space[Monga et al., 2019]. J learns neither the operator nor the update rule, only the start. Learned primal-dual is the precedent for keeping the true forward model inside each layer[Adler and Öktem, 2017].

A.6 Loss, backward, and the optimisers

The loss (training_loss; run_free_gaussian_fine_20261113.py:main), with weights [384, .05, 96, .1, .01] plus 1 on the deep-supervision sum:

L = 384·densew + 0.05·homog + 96·global + 0.1·gram + 0.01·cover + Σt=015 384·1.016−t·mse(qt)

  • densew = Σ wp(F − T)²/(BPD), where w = 0.5 + 0.5 min(P softmax(10⟨tp, g⟩/‖g‖), 4) is renormalised to mean 1 and g is the teacher's unit CLS. Only this term is relevance-weighted. The 16 deep-supervision terms, the refinement objective S and the reported holdout metric are all unweighted.
  • homog = Σ Apk‖tp − zk‖²/(BP) with z the A-weighted means. Its gradient holds z fixed, which is exact by the envelope argument (z minimises the sum).
  • global: MSE(unit(meanpF + 0.1·MLP(blockpool4×4F)), g). The MLP is 6144 → 64 (exact GELU) → 384 (read_field).
  • gram: MSE between unit(F)unit(F)ᵀ and TTᵀ, (B, 1024, 1024).
  • cover: mean over a 32×32 grid of max(mink‖Lk⁻¹(x − μk)‖² − 9, 0). It was 0 on the images checked.

What is cut in training (torch autograd through everything else):

  1. The clustering error ep is computed from field.detach().
  2. The Adam direction is detached: no second derivatives, and the inner gradient is recomputed on a fresh leaf at each step.
  3. T is constant.

So the loss reaches the step sizes through qt+1 = qt − s·dir, giving ∂L/∂st,k = −Σgroup k gqt+1·dirt. It reaches the clustering stage through the identity path ∂qt+1/∂qt = I. Deep supervision adds 384·∂mset/∂qt = 384·gt/(10⁴B) at each step, reusing the inner gradient (KnownGradient)[Lee et al., 2014]. NumPy reduced_backward writes this chain out to the step sizes, to q₀, to raw, and through the last clustering iteration to β₇, s₇, wf,7, wg,7. It is checked against finite differences of the frozen-direction loss (to 10⁻⁸) and against torch autograd (to 10⁻¹²). With directions recomputed, the finite difference is a different number: for example the centre step at t = 0 gives 1.80 against 0.377. That is the price of detaching.

Optimisers (muon_update, adan_update; runtime_muon_adan_dynamic.py). Muon (lr 0.01, weight decay 0.01) updates the 2-D matrices readout_weight, slot_queries, key_weight, value_weight, query_weight:

buf ← .95buf + .05g; u = .05g + .95buf; X = NS₅(u/(‖u‖F + 10⁻⁷)); p ← p(1 − lr·wd) − lr√max(1, rows/cols)·X

Here NS₅ is five steps of X ← 3.4445X + (−4.775A + 2.0315A²)X with A = XXᵀ[Jordan, 2024]. Adan (lr 0.0005, weight decay 0, ε 10⁻⁶, retentions .98/.92/.99) updates all 1-D parameters and two 2-D ones, geometry_weight and readout_output_weight, which optimizer_groups excludes from Muon by name[Xie et al., 2022]:

Δ = g − gprev; m ← .98m + .02g; d ← .92d + .08Δ; v ← .99v + .01(g + .92Δ)²

p ← p − lr·m/((1 − .98t)·den) − lr·.92·d/((1 − .92t)·den), den = √v/√(1 − .99t) + ε

Schedule: batch 32, 6,000 updates, global gradient norm clipped to 1. The learning-rate factor is min(1, (n + 1)/200)·(0.1 + 0.45(1 + cos(πn/6000))).

A.7 Does the rebuild agree?

numpy_pipeline.py (float64) runs the first four holdout images and all its checks in about 2 minutes. compare_with_torch.py runs the repository's PyTorch references on CPU. The Triton model cannot run on a CPU because its guard rejects CPU tensors.

Holdout image0123
Dense MSE after 0 steps (NumPy)1.773e-41.863e-41.749e-41.189e-4
After 16 steps (NumPy)1.24724e-41.36947e-41.34403e-48.12717e-5
GPU run, summary.json1.24716e-41.36949e-41.34404e-48.12711e-5
  • NumPy against torch float64, every stage: about 10⁻¹⁴ (curve, A, Z, q, loss, gradients).
  • NumPy against the real FP32 Triton GPU run: 3.6×10⁻⁵ relative; NumPy in float32: 2.1×10⁻⁵. The gap is float32 rounding amplified through 16 Adam steps, because Adam normalises tiny gradients to unit-sized moves. It is not a difference in the method.
  • The task's target of 10⁻⁶ is met against the torch path, not against the GPU, for that reason.

Over these four images refinement lowers the error by 27%. On all 800 holdout images the GPU went from 0.000153 to 0.000114.

A.8 Split and merge: the equations, and why they lost

These are quick experiments in plain PyTorch, not the trained model. Note their conventions: a relative ridge λ = 10⁻³ × mean diag(AᵀA) instead of J's fixed 0.01, and log-masses instead of mass logits (equivalent up to a per-image constant). They also use per-image k-means starts and Adam at 0.03.

Moment-preserving split (scripts/research/quick/s6_split_hierarchy.py:children). Eigen-decompose Σ = λ₁uuᵀ + λ₂vvᵀ. With offset factor α (0.5, 0.7 or 0.9 tried), the two children are

μ± = μ ± α√λ₁ u, Σ' = (1 − α²)λ₁uuᵀ + λ₂vvᵀ, log m' = log m − log 2

The pair keeps the parent's mass, mean and covariance: Σ' + α²λ₁uuᵀ = Σ. Candidates are ranked by residual energy ΣpApkep, information gain on 256 teacher-feature codes, mass, random, or by an oracle that tries each split.

Moment-matched merge (s7_fine_budget.py:merge_params), with wa = ma/(ma + mb):

m = ma + mb, μ = waμa + wbμb, Σ = Σi wi(Σi + (μi − μ)(μi − μ)ᵀ)

Pairs are ranked by the Ward-like cost (mamb/(ma + mb))(‖za − zb‖² + ¼‖μa − μb‖²).

input512, 100 images128643216
Merge down from 256, +4 steps per level0.00008090.0001350.0002320.000365
Direct k-means, equal total steps0.00006260.0001110.0001980.000328

Merging loses by 11 to 29%. Its cheap scores rank the true error increase poorly (mean Spearman 0.57 at best). On the 16×16 target, splitting 16 → 64 by residual energy reached 0.0000866 against 0.0000622 for direct k-means at the same count and steps (200 images). Information gain (0.0000937) did not beat mass (0.0000925) or random (0.0000955).

Why. Preserving moments preserves the mixture's shape, but under least squares the Gaussians act as a basis, and the error depends on how the basis spans the field, not on the mixture's moments. A child inherits a placement that was optimal for its parent's role. A few Adam steps cannot undo that, whereas clustering at the target count starts every Gaussian where tokens actually group. This is the inheritance effect on Chapter 14's list. One seed, settings tuned on the same images.

A.9 Check yourself

1. Why does the code keep the full ridge adjoint instead of holding Z fixed?

With λ = 0 the normal equations zero the implicit term. The ridge pull towards Z₀, which itself depends on A, breaks that; the difference measured here is 3.6%. The factor is reused, never differentiated.

2. How does the training loss reach the clustering stage?

Directions are detached, so ∂qt+1/∂qt = I. Deep supervision adds 384·∂mset/∂qt at every step on the way back.

3. Which of these terms is weighted by relevance to the CLS token?

semantic_importance weights only the 384 × dense term. The inner objective S, the deep-supervision terms and the reported holdout dense MSE are unweighted.

4. Which optimiser updates geometry_weight, a 6 × 70 matrix?

optimizer_groups sends 2-D parameters to Muon except geometry_weight and readout_output_weight, the output heads; Adan takes those and all vectors.

5. Why does the NumPy result differ from the GPU by 3.6 × 10⁻⁵?

Torch and NumPy in float64 agree to 10⁻¹⁴. Adam divides by √v, so tiny rounding differences in small gradients become order-one changes in direction; float32 NumPy shows the same size of gap.

A.10 Close the loop

Without scrolling up, list the stages of J with their shapes, and say what is detached in training.

A model answer

“Unit teacher tokens T (B, 1024, 384) on a 32×32 grid, four quadrature points per token. 64 Gaussians start at a Halton layout. Eight clustering iterations each render A (B, 1024, 64) with weighted-mean features, soft-assign tokens by feature and geometry, fit moments with a learned width, blend, and add a slot-attention residual. Then 16 Adam steps at 0.03 run on the true dense error, with Z solved by ridge least squares, (AᵀA + 0.01I)Z = AᵀT + 0.01Z₀, and the gradient passing through the solve by the implicit adjoint. The output is F = AZ (B, 1024, 384). The loss is 384 × relevance-weighted dense error plus small homogeneity, global, Gram and coverage terms, plus 384 × the unweighted error after each step. Training detaches the clustering error and the Adam directions only; Muon updates five matrices and Adan the rest.”

Read next Golub and Pereyra, “Separable nonlinear least squares: the variable projection method and its applications” (2003), sections 1 to 3. It derives the gradient of the projected functional that A.3 used, and when dropping the implicit term is exact.

Something unclear, or want to rehearse how you would say this to a reviewer? Ask your teacher in the chat.

One-page summary of this appendix ·