Optimisation inside the network
Unrolled optimisation: layers are steps of an optimiser on a known objective, with some parts trained end to end. LISTA learns operators; learned primal-dual keeps the true forward model in every layer; learning to optimise learns the update rule (risk: poor generalisation, Chen et al.).
Adam: q ← q − s · m̂/(√v̂ + ε), per-parameter scaling. s5 gate (100 images, from k-means): Adam closes 38 / 52 / 69% of the gap to 200 steps after 4 / 8 / 16 steps; normalised gradient descent 21 / 32 / 41%. Resetting Adam state or birth/death hurt.
Module (free_gaussian_refine.py): q = (cx, cy, log(a − F), b, log(d − F), mass logit), F = 0.004; objective 104 × Σ dense MSE with ridge least-squares features re-solved inside; gradient detached; step size per step and group (centre, covariance, mass), 0.03. Training gradient reaches step sizes through q − s · dir and the clustering stage through the identity path q0 → qK. No second derivatives.
Deep supervision: dense MSE after each step, weight w · δK−t. D, E (δ = 0.5) degraded: the start's weight was 1.5 or 0.006. Uniform δ = 1 fixed it.
| Arm | Setting | 16 steps | 32 steps |
|---|---|---|---|
| Plain Adam on B | untrained | 0.0000630 | 0.0000567 |
| G / H | 16, weight 48, fixed / learned steps | 0.0000609 / 0.0000619 | 0.0000547 / 0.0000560 |
| I | 32 trained, weight 24 | 0.0000642 | 0.0000571 |
| J | 16, weight 384, fixed steps | 0.0000587 | 0.0000531 |
| K / L | weight 1536 / 32 trained | 0.0000588 / 0.0000592 | 0.0000532 / 0.0000535 |
16×16 target, 64 Gaussians, 800 holdout images, one seed. J over three seeds: 0.0000588 / 0.0000532. J's start (0.0000885) is worse than clustering B (0.0000804) but it ends better.
Defence: “J is a learned clustering initialiser trained end to end through 16 fixed Adam steps on the true least-squares error, with uniform deep supervision. The update rule is not learned; learned step sizes shrank and hurt. Supervision strength was the lever and saturates.”