Rebuilding the pipeline in NumPy
Code: learning/rebuild/numpy_pipeline.py (NumPy only), load_checkpoint.py (torch export to .npz), compare_with_torch.py (repository references on CPU).
Shapes: T (B, 1024, 384) unit rows; points (1024, 4, 2) at token centre ± 1/64; geometry (B, 64, 6); A (B, 1024, 64); Z (B, 64, 384); F = AZ (B, 1024, 384).
Responsibilities: log ρ = log π − log a − log d − ½‖L⁻¹(x − μ)‖²; logsumexp over the 4 points, then softmax over Gaussians.
Features: (AᵀA + 0.01I)Z = AᵀT + 0.01Z₀, Z₀ weighted means. Adjoint: G = M⁻¹dZ, dA = (T − AZ)Gᵀ − (AG)Zᵀ, dZ₀ = λG → A. Z is never detached; the Cholesky factor is reused, not differentiated. Fixed-Z (variable projection) gradient is 3.6% off at λ = 0.01.
Clustering (8 iterations): weighted-mean render → detached error → soft assignment by wf·cosine + wg·(−log(1 + m) + log ω) → moment fit with width s and floor 0.009, mass log(count + 0.01) → blend β → slot attention (8× scores, 0.75 local / 0.25 global) → residual head /8. Start: Halton, a = d = 0.12.
Refinement: 16 Adam steps (lr 0.03 for all groups, fixed, 0.9 / 0.999 / 10⁻⁸) on S = 10⁴ Σ mse; q = (μ, log(a − .004), b, log(d − .004), m).
Loss: 384·dense (relevance-weighted) + .05·homogeneity + 96·global + .1·gram + .01·coverage + Σt 384·mset (unweighted). Detached: clustering error, Adam directions, T. ∂L/∂st = −Σ gqt+1·dirt.
Optimisers: Muon (0.01, wd 0.01; NS5 3.4445/−4.775/2.0315; momentum .95 Nesterov) on readout_weight, slot_queries, key/value/query weights. Adan (0.0005, wd 0, ε 10⁻⁶, .98/.92/.99) on the rest, including the 2-D geometry_weight and readout_output_weight. Batch 32, 6,000 updates, 200 warm-up, cosine to 0.1, clip 1.
Agreement: NumPy vs torch float64 about 10⁻¹⁴ everywhere; vs the FP32 GPU run 3.6×10⁻⁵ (float32 rounding through Adam). Four images: 1.64e-4 → 1.19e-4 after 16 steps; GPU, 800 images: 0.000153 → 0.000114.
Split and merge: split keeps mass, mean and covariance (μ ± α√λ₁u, (1 − α²)λ₁ along u); merge is moment-matched. Both lose to direct clustering (merge 11 to 29% on input512; split 0.0000866 vs 0.0000622 at 64 on 16×16) because children inherit a placement made for their parent; quick experiments use relative λ = 10⁻³·mean diag(AᵀA).
Defence: “The features are solved in closed form and differentiated by the implicit adjoint; the network only places Gaussians; refinement is plain Adam on the true error with detached directions, so only the start is learned.”