Giving each Gaussian a feature: least squares
Field: F̂ = AZ: A is P × K responsibilities, Z is K × D features (D = 384). Target T is P × D, each row scaled to length one.
Weighted mean (the old rule, and the prior Z₀): zk = Σp Apktp / Σp Apk. Ignores overlap.
Least squares: minimise ‖AZ − T‖²; normal equations AᵀA Z = AᵀT. Features act jointly as a basis and may leave the range of the data.
Ridge towards the prior (this project, lstsq_coefficients.py):
(AᵀA + λI) Z = AᵀT + λZ₀, λ = 0.01
Prevents huge cancelling features when Gaussians nearly coincide (example: exact z = (25.5, −24.5), ridge (1.47, −0.47)); negligible otherwise (average diagonal of AᵀA is at most P/K).
Solve: AᵀA + λI is symmetric positive definite, so factor LLᵀ (_cholesky) and do forward then back substitution (_cholesky_solve). One K × K factor per image serves all 384 channels. Up to 128 Gaussians in one tile; 256 via lstsq_blocked.py.
Variable projection (Golub and Pereyra, 1973): features solved inside, geometry optimised outside; the error is a function of geometry alone.
Implicit differentiation (RidgeCoefficients.backward): G = M⁻¹dZ with the saved factor, dA = (T − AZ)Gᵀ − (AG)Zᵀ, dZ₀ = λG.
| Gridless64, 16×16 target | Holdout dense MSE | Notes |
|---|---|---|
| Weighted means | 0.0001343 | seed 20261112, 800 images |
| Least squares, trained | 0.0001041 (−22.5%) | three-seed mean 0.00010442 (−22.2%); probe unchanged |
| Least squares swapped in, placement frozen | −6.7% | seed 20261108 |
Holdout reused throughout development; not a sealed test.
Defence answer: “Features are not predicted; given the placement they are the ridge least-squares optimum, solved by Cholesky. The network only places Gaussians, and gradients pass through the solve by implicit differentiation. It cut dense error by about 22% with the probe unchanged.”
Sources: Hastie, Tibshirani, Friedman, Elements of Statistical Learning, section 3.4.1; Golub and Pereyra (1973).