Summary · Chapter 4

Giving each Gaussian a feature: least squares

Field: F̂ = AZ: A is P × K responsibilities, Z is K × D features (D = 384). Target T is P × D, each row scaled to length one.

Weighted mean (the old rule, and the prior Z₀): zk = Σp Apktp / Σp Apk. Ignores overlap.

Least squares: minimise ‖AZ − T‖²; normal equations AᵀA Z = AᵀT. Features act jointly as a basis and may leave the range of the data.

Ridge towards the prior (this project, lstsq_coefficients.py):

(AᵀA + λI) Z = AᵀT + λZ₀, λ = 0.01

Prevents huge cancelling features when Gaussians nearly coincide (example: exact z = (25.5, −24.5), ridge (1.47, −0.47)); negligible otherwise (average diagonal of AᵀA is at most P/K).

Solve: AᵀA + λI is symmetric positive definite, so factor LLᵀ (_cholesky) and do forward then back substitution (_cholesky_solve). One K × K factor per image serves all 384 channels. Up to 128 Gaussians in one tile; 256 via lstsq_blocked.py.

Variable projection (Golub and Pereyra, 1973): features solved inside, geometry optimised outside; the error is a function of geometry alone.

Implicit differentiation (RidgeCoefficients.backward): G = M⁻¹dZ with the saved factor, dA = (T − AZ)Gᵀ − (AG)Zᵀ, dZ₀ = λG.

Gridless64, 16×16 targetHoldout dense MSENotes
Weighted means0.0001343seed 20261112, 800 images
Least squares, trained0.0001041 (−22.5%)three-seed mean 0.00010442 (−22.2%); probe unchanged
Least squares swapped in, placement frozen−6.7%seed 20261108

Holdout reused throughout development; not a sealed test.

Defence answer: “Features are not predicted; given the placement they are the ridge least-squares optimum, solved by Cholesky. The network only places Gaussians, and gradients pass through the solve by implicit differentiation. It cut dense error by about 22% with the probe unchanged.”

Sources: Hastie, Tibshirani, Friedman, Elements of Statistical Learning, section 3.4.1; Golub and Pereyra (1973).